检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Request body: { "job_name":"MapReduceTest", "job_type":"MapReduce", "arguments":[ "obs://obs-test/program/hadoop-mapreduce-examples-x.x.x.jar", "wordcount", "obs://obs-test/input/", "obs://obs-test/job/mapreduce/output" ], "properties
data from an SFTP server to HDFS or OBS Importing data from an SFTP server to HBase Importing data from an SFTP server to Phoenix tables Importing data from an SFTP server to Hive tables Importing data from an FTP server to HDFS or OBS Importing data from an FTP server to HBase Importing
and use the OBS cross-region replication function to copy the data to OBS in the region where the target cluster resides.
Exports all types of files from HDFS/OBS to an SFTP server. The open source component Sqoop can export only text files and SequenceFile files. Supports file coding format conversion during file import and export.
Supported OBS monitoring. Upgraded the OBS packages. Resolved the issue that some data is not inserted when 10 data records are concurrently inserted into hive-jdbc. Resolved the issue that Hive occasionally reports a Kryo deserialization failure.
Supported OBS monitoring. Upgraded the OBS packages. Resolved the issue that some data is not inserted when 10 data records are concurrently inserted into hive-jdbc. Resolved the issue that Hive occasionally reports a Kryo deserialization failure.
Supported OBS monitoring. Upgraded the OBS packages. Resolved the issue that some data is not inserted when 10 data records are concurrently inserted into hive-jdbc. Resolved the issue that Hive occasionally reports a Kryo deserialization failure.
Bucket mrs-bucket (Mandatory) Name of the OBS file system to which audit logs are exported. OBS Path opt/omm/oms/auditLog (Mandatory) OBS path to which audit logs are exported. AK - (Mandatory) User's access key ID. SK - (Mandatory) User's secret access key.
What Should I Do If Error 408 Is Reported When an MRS Node Accesses OBS? How Do I Enable Different Service Programs to Use Different Yarn Queues? What Should I Do If a Job Fails to Be Submitted and the Error Is Related to OBS?
Does OBS Support the ListObjectsV2 Protocol? Can a Crawler Service Be Deployed on Nodes in an MRS Cluster? Does MRS Support Secure Deletion? How Do I Use PySpark to Connect MRS Spark? Why Mapped Fields Do not Exist in the Database After HBase Synchronizes Data to CSS?
Figure 1 MemArtsCC structure Table 1 Structure Name Description MemArtsCC SDK SDK used by OBSA, a Hadoop client plug-in on the FS client, to access OBS server objects.
Impact on the System The system cannot access OBS after the temporary agency credential expires. Operations such as reading and writing cold data in OBS cannot be performed on tables configured with cold and hot separation policies.
Supported OBS monitoring. Upgraded the OBS packages. Resolved the issue that some data is not inserted when 10 data records are concurrently inserted into hive-jdbc. Resolved the issue that Hive occasionally reports a Kryo deserialization failure.
Managing Data Connections Enabling Access to OBS by Mapping HDFS Addresses to OBS Addresses By mapping HDFS addresses to OBS addresses, you can access data migrated from HDFS to OBS without changing the service logic.
Figure 1 Hot and cold data separation OBS provides secure, reliable, and cost-effective distributed storage service that supports large-scale data. Doris uses OBS to store data separately. Figure 2 shows the principle.
For example, create the table test in the obs://mrs-word001/table/ directory. create table test(id int) location 'obs://mrs-word001/table/'; Query all tables. show tables; If the table test is displayed in the command output, OBS is successfully accessed.
Set this parameter to the OBS storage path of the hive Catalog, which is obtained in Creating a LakeFormation Instance. Figure 1 Configuring hive.metastore.warehouse.dir Click Save. Interconnecting Spark with OBS If your cluster does not have the Spark component, skip this step.
data from an SFTP server to HDFS or OBS Importing data from an SFTP server to HBase Importing data from an SFTP server to Phoenix tables Importing data from an SFTP server to Hive tables Importing data from an FTP server to HDFS or OBS Importing data from an FTP server to HBase Importing
Figure 1 MRS Job Submission Process Data processed by MRS jobs is usually stored in OBS or HDFS. Before creating a job, you need to upload the data to be analyzed to an OBS file system or the HDFS in the MRS cluster.
OBS Permission Control Click Manage and modify the mapping between MRS users and OBS permissions. For details, see Configuring Fine-Grained OBS Access Permissions for MRS Cluster Users. Logging Used to collect logs about cluster creation and scaling failures.
