检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
ModifyObjectMetaData", "obs:object:DeleteObject", "obs:object:ListMultipartUploadParts", "obs:bucket:HeadBucket", "obs:object:AbortMultipartUpload", "obs:bucket:ListBucket", "obs:object:PutObject", "obs:bucket:ListAllMyBuckets" ],
Delete the OBS certificate.
ModifyObjectMetaData", "obs:object:DeleteObject", "obs:object:ListMultipartUploadParts", "obs:bucket:HeadBucket", "obs:object:AbortMultipartUpload", "obs:bucket:ListBucket", "obs:object:PutObject", "obs:object:GetAccessLabel", "obs:object:DeleteAccessLabel"
Create a table. create table test(name string) location "obs://OBS parallel file system name/user/hive/warehouse/Database name/Table name"; Check the table location. desc formatted test; It is found that the Location of the table is the OBS path.
Interconnecting Sqoop with OBS Using an IAM Agency After connecting the Sqoop client to an OBS file system by referring to Interconnecting an MRS Cluster with OBS Using an IAM Agency, you can import tables from a relational database to OBS or export tables from OBS to a relational
-put /tmp/test/ obs://OBS parallel file system name/tmp/flinkjob/ Run WordCount.jar. flink run Client installation directory/Flink/flink/examples/batch/WordCount.jar -input obs://OBS parallel file system name/tmp/flinkjob/test -output obs://OBS parallel file system name/tmp/flinkjob
If the file list is returned, OBS is successfully accessed. hadoop fs -ls obs://mrs-word001/ Figure 1 Returned file list Run the following command to upload the /opt/test.txt file from the client node to the OBS file system path: hdfs dfs -put /opt/test.txt obs://OBS parallel file
Accessing OBS Using MapReduce Through Guardian After Guardian is interconnected with OBS by referring to Disabling Ranger OBS Path Authentication for Guardian or Enabling Ranger OBS Path Authentication for Guardian, you need to add custom configurations for the MapReduce component
Interconnecting Hive with OBS Using an IAM Agency After configuring decoupled storage and compute for a cluster by referring to Interconnecting an MRS Cluster with OBS Using an IAM Agency, you can create tables with OBS paths as their location on the Hive client.
Interconnecting Flume with OBS Using an IAM Agency After configuring decoupled storage and compute for a cluster by referring to Interconnecting an MRS Cluster with OBS Using an IAM Agency, you can run OBS jobs using Flume.
The table data is stored in the OBS file system.
Accessing OBS Using HetuEngine Through Guardian After Guardian is interconnected with OBS by referring to Disabling Ranger OBS Path Authentication for Guardian or Enabling Ranger OBS Path Authentication for Guardian, you can create a table with a specified location as the OBS file
Interconnecting Spark with OBS Using an IAM Agency After configuring decoupled storage and compute for a cluster by referring to Interconnecting an MRS Cluster with OBS Using an IAM Agency, you can create tables with OBS paths as their location on the Spark client.
Accessing OBS Using Flink Through Guardian After Guardian is interconnected with OBS by referring to Disabling Ranger OBS Path Authentication for Guardian or Enabling Ranger OBS Path Authentication for Guardian, you can access the OBS parallel file system and run jobs in a Flink job
Access the OBS file system. hdfs dfs -ls obs://OBS parallel file system name/path Create a directory in the OBS file system. hdfs dfs -mkdir obs://OBS parallel file system name/hadoop1 Execute the YARN task to access OBS. yarn jar Client installation directory/HDFS/hadoop/share/hadoop
q Parent Topic: Example for Interconnecting an MRS Cluster with OBS
Accessing OBS Using Spark Through Guardian After Guardian is interconnected with OBS by referring to Disabling Ranger OBS Path Authentication for Guardian or Enabling Ranger OBS Path Authentication for Guardian, you can create a table whose location is the OBS path on the Spark client
Accessing OBS Using Hive Through Guardian After Guardian is interconnected with OBS by referring to Disabling Ranger OBS Path Authentication for Guardian or Enabling Ranger OBS Path Authentication for Guardian, you can create tables stored in the OBS parallel file system on the Hive
Access the OBS file system. hdfs dfs -ls obs://OBS parallel file system name/path Create a directory in the OBS file system. hdfs dfs -mkdir obs://OBS parallel file system name/hadoop Upload the /opt/test.txt file on the client node to the obs://OBS parallel file system name/hadoop
Accessing OBS Using Hudi Through Guardian After Guardian is interconnected with OBS by referring to Disabling Ranger OBS Path Authentication for Guardian or Enabling Ranger OBS Path Authentication for Guardian, you can create a Hudi COW table in spark-shell and store it to OBS.
