检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Creating a VPC and Subnet Object Storage Service (OBS) OBS stores the following user data: MRS job input data, such as user programs and data files MRS job output data, such as result files and log files of jobs In MRS clusters, HDFS, Hive, MapReduce, YARN, Spark, Flume, and Loader
Question: When data is exported from a DistCP job, if some files already exist in OBS, how will the job process the files? Answer: DistCP jobs will overwrite the files in OBS. Parent topic: Job Management
You can enter the path or click HDFS or OBS to select a file. The path can contain a maximum of 1,023 characters. It cannot contain special characters (;|&>,<'$) and cannot be left blank or all spaces. The OBS program path starts with obs://.
You can enter the path or click HDFS or OBS to select a file. The path can contain a maximum of 1,023 characters. It cannot contain special characters (;|&>,<'$) and cannot be left blank or all spaces. The OBS program path starts with obs://.
The submission command example is as follows (the topology name is obs-test): storm jar /opt/jartarget/storm-examples-1.0.jar com.huawei.storm.example.obs.SimpleOBSTopology obs://my-bucket obs-test After the topology is submitted successfully, log in to OBS Browser to view the topology
OBS Indicates that backup files are stored in an OBS directory. You need to set the Target Path to the OBS directory for storing backup data. Set Maximum Number of Recovery Points to the number of snapshots that can be retained in the cluster.
Interconnection with Other Cloud Services Using MRS Spark SQL to Access GaussDB(DWS) Connecting to the OBS File System with an MRS Hive Table Interconnecting Hive with CSS
Importing Doris Data Importing Data to Doris with Broker Load Importing OBS Data to Doris with Broker Load Importing Data to Doris with Stream Load Parent topic: Using Doris
Prerequisites You have uploaded the program packages and data files required by jobs to OBS or HDFS. If the job program needs to read and analyze data in the OBS file system, you need to configure storage-compute decoupling for the MRS cluster.
ClickHouse Data Import Interconnecting ClickHouse with RDS for MySQL Interconnecting ClickHouse with OBS Synchronizing Kafka Data to ClickHouse Importing DWS Table Data to ClickHouse Using ClickHouse to Import and Export Data Parent topic: Using ClickHouse
Set Program Path to the path where programs are stored on OBS, for example, obs://sparkpi/program/spark-examples_2.11-2.1.0.jar. In Program Parameter, select --class for Parameter and set Value to org.apache.spark.examples.SparkPi. Set Parameters to 10.
Loader supports the following data export modes: Exporting data from HDFS or OBS to an SFTP server Exporting data from HDFS or OBS to a relational database Exporting data from HBase to an SFTP server Exporting data from HBase to a relational database Exporting data from Phoenix tables
The OBS program path should start with obs://, for example, obs://wordcount/program/XXX.jar. The HDFS program path should start with hdfs://, for example, hdfs://hacluster/user/XXX.jar.
Developing an HDFS Application HDFS Development Plan Initializing HDFS Writing Data to an HDFS File Appending HDFS File Content Reading an HDFS File Deleting an HDFS File HDFS Colocation Setting HDFS Storage Policies Using HDFS to Access OBS Parent topic: HDFS Development Guide
to MRS Preparing the Network Before Data Migration to MRS Migrating Data from Hadoop to MRS Migrating Data from HBase to MRS Migrating Data from Hive to MRS Using BulkLoad to Import Data to HBase in Batches Migrating MySQL Data to MRS Hive with CDM Migrating Data from MRS HDFS to OBS
ClickHouse Data Import Interconnecting ClickHouse with RDS for MySQL Interconnecting ClickHouse with OBS Interconnecting ClickHouse with HDFS (MRS 3.2.0-LTS) Interconnecting ClickHouse with HDFS (MRS 3.3.0-LTS and later) Configuring Interconnection Between ClickHouse and Kafka Synchronizing
ECS BMS VPC EVS Image Management Service (IMS) OBS EIP SMN IAM For details about how to view and modify quotas, see Quotas.
What Are the Differences Between OBS and HDFS in Data Storage? What Are the Solutions for Processing 1 Billion Data Records? What are the advantages of the compression ratio of zstd?
In addition, MRS supports fine-grained permission management of OBS for multiple tenants. Permissions to access OBS buckets and objects in the buckets are differentiated based on user roles, so that MRS users can each control a different directory in OBS buckets.
If you want to restore data from OBS, you have connected the Doris cluster to OBS and have the permission to access OBS.
