检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
OBS Permission Control Mapping between MRS users and OBS permissions. Data Connection Type of the data connection associated with the cluster. Agency Agency bound to or modified by the cluster. Key Pair Name of a key pair, which is set during cluster creation.
What Should I Do If Data Failed to Be Synchronized to a Hive Table on the OBS Using hive-table? What Should I Do If Data Failed to Be Synchronized to an ORC or Parquet Table Using hive-table? What Should I Do If Data Failed to Be Synchronized Using hive-table?
OBS Indicates that backup files are stored in an OBS directory. This option is available for MRS 3.3.0-LTS.1 or later only. Target Path: indicates the OBS directory for storing backup data.
In the environmental protection industry, climate data is stored on OBS and periodically dumped into HDFS for batch analysis. 10 TB of climate data can be analyzed in 1 hour.
If a low-permission user lacks OBS path access permission for the default database, Spark will display a permission error message but will still successfully create the database.
Set this parameter to a correct OBS path. The OBS path does not support files or programs encrypted by KMS. The value can contain a maximum of 1,023 characters, excluding special characters such as ;|&>'<$, and can be left blank.
It provides a data abstraction layer for computing frameworks including Apache Spark, Presto, MapReduce, and Apache Hive, so that upper-layer computing applications can access persistent storage systems including HDFS and OBS through unified client APIs and a global namespace.
You can obtain the AK and SK by logging in to the OBS console and go to the My Credentials page.
OBS: indicates that backup files are stored in OBS. If you select this option, set the following parameters: Target Path: indicates the OBS directory for storing backup data.
Table 1 HDFS parameters Parameter Description Default Value fs.obs.security.provider Implementation method of obtaining the key for accessing the OBS file system Value options are as follows: com.huawei.mrs.MrsObsCredentialsProvider: obtains a credential through an MRS agency. com.obs.services.EcsObsCredentialsProvider
Enterprise-Class Enhancements of Hive Storing Hive Table Partitions to OBS and HDFS Configuring Automatic Removal of Old Data from the Hive Directory to the Recycle Bin Configuring Hive to Insert Data to a Directory That Does Not Exist Configuring that Location Cannot Be Specified
File Format Loader supports the following file formats of data stored in OBS: CSV_FILE: Specifies a text file. When the destination link is a database link, only the text file is supported. BINARY_FILE: Specifies binary files excluding text files.
import command sqoop import --connect jdbc:mysql://10.100.231.134:3306/test --username root --password xxx --table component --target-dir obs://obs-file-bucket/xx --delete-target-dir --fields-terminated-by "," -m 1 --as-textfile Importing data from MySQL to OBS tables outside Hive
import command sqoop import --connect jdbc:mysql://10.100.231.134:3306/test --username root --password xxx --table component --target-dir obs://obs-file-bucket/xx --delete-target-dir --fields-terminated-by "," -m 1 --as-textfile Importing data from MySQL to OBS tables outside Hive
Parameter Default Value New Value --conf spark.shuffle.readHostLocalDisk false true --conf spark.io.encryption.enabled true false --conf spark.shuffle.service.enabled false true Adjusting Spark Scheduling Parameters to Shorten the Spark Scheduling Delay in the OBS Enabling OBS storage
Data is stored in OBS, which features low cost and unlimited storage capacity. The clusters can be deleted at any time in OBS. The computing performance is determined by OBS access performance and is lower than that of HDFS.
If you want to back up data to OBS, you have connected the current cluster to OBS and have the permission to access OBS. Backing Up CDL Service Data On FusionInsight Manager, choose O&M > Backup and Restoration > Backup Management. Click Create.
OBS: indicates that backup files are stored in OBS. If you select this option, set the following parameters: Target Path: indicates the OBS directory for storing backup data.
OBS: indicates that backup files are stored in OBS. If you select this option, set the following parameters: Target Path: indicates the OBS directory for storing backup data.
OBS: indicates that backup files are stored in OBS. If you select this option, set the following parameters: Target Path: indicates the OBS directory for storing backup data.
