检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Constraints on OBS During file migration, the system automatically transfers the files concurrently. In this case, Concurrent Extractors in the task configuration is invalid. Resumable transmission is not supported. If CDM fails to transfer files, OBS fragments are generated.
Commercial use None 2 File migration You can migrate files from OBS to HDFS. Commercial use Table/File Migration Jobs Earlier Feature Releases View PDF of earlier feature releases.
If data is migrated between file-related data sources, such as FTP, SFTP, OBS, and HDFS, the value of File Format must the same as the source file format.
Cdm.0800 400 Failed to connect to the OBS server. The OBS server fails to be connected.
The migration path is as follows: CDH → DES → CDM (HUAWEI CLOUD) → OBS → CDM (HUAWEI CLOUD) → MRS DES is well suited to the scenario where a large amount of data is to be transmitted, no private line is set up between the private cloud and HUAWEI CLOUD, and the bandwidth from the
Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS for future viewing. Before writing dirty data, create an OBS link on the CDM console.
Currently, CDM uses connectors to connect to OBS, MRS, and databases. New connectors can be added to CDM as well. Link A link is an object set up based on a connector and used to connect to a specific data source.
Write Dirty Data: Specify this parameter if data that fails to be processed or filtered out during job execution needs to be written to OBS for future viewing. Before writing dirty data, create an OBS link on the CDM console.
After KMS encryption is performed, the encryption status of the objects on OBS cannot be changed. A key in use cannot be deleted. Otherwise, the object encrypted with this key cannot be downloaded. Parent topic: Advanced Data Migration Guidance
Public services, such as Elastic Cloud Server (ECS), Elastic Volume Service (EVS), Object Storage Service (OBS), Virtual Private Cloud (VPC), Elastic IP (EIP), and Image Management Service (IMS), are shared within the same region.
After KMS encryption is performed, the encryption status of the objects on OBS cannot be changed. A key in use cannot be deleted. Otherwise, the object encrypted with this key cannot be downloaded. Parent topic: Key Operation Guide
Object storage Object Storage Service (OBS) Jobs can be split based on files. File system FTP Jobs can be split based on files. SFTP Jobs can be split based on files. HTTP Jobs can be split based on files.
Object Storage Service (OBS) For details about the parameters, see OBS Link Parameters. MRS HDFS FusionInsight HDFS Apache HDFS If the data source is HDFS of MRS, Apache Hadoop, or FusionInsight HD, see HDFS Link Parameters.
storage support The server must support OBS storage.
The field name cannot be the same as that of any source field (including custom fields). cdc_last_update_date Hudi table creation parameters Location OBS or HDFS path where database table files are stored - Hudi Table Type Storage type of the Hudi table MOR: Data is written to a log
The corresponding link parameters are as follows: generic-jdbc-connector: link to relational database obs-connector: link to OBS hdfs-connector: link to HDFS hbase-connector: link to HBase and link to CloudTable hive-connector: link to Hive ftp-connector/sftp-connector: link to an
NOTE: If the destination is OBS, only the binary format is supported. CSV JSON Type This parameter is displayed only when File Format is set to JSON. Type of a JSON object stored in a JSON file. The options are JSON object and JSON array.
If files are migrated between FTP, SFTP, OBS, and HDFS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping. You can create a field converter on the Map Field page when creating a table/file migration job.
Column names are displayed when the source of the migration job is OBS, CSV files are to be migrated, and parameter Extract first row as columns is set to Yes. Field mapping is not involved when the binary format is used to migrate files to files.
If files are migrated between FTP, SFTP, OBS, and HDFS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping. You can create a field converter on the Map Field page when creating a table/file migration job.
