检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Archive to OBS: Dirty data is stored in OBS and printed in job logs. Do not archive Write Dirty Data Link This parameter is displayed when Dirty Data Policy is set to Archive to OBS.
Archive to OBS: Dirty data is stored in OBS and printed in job logs. Do not archive Write Dirty Data Link This parameter is displayed when Dirty Data Policy is set to Archive to OBS.
The following versions are recommended: 1.2.X 3.1.X Object storage OBS Supported Supported - File system FTP and SFTP Supported Not supported - Relational database RDS for MySQL, RDS for PostgreSQL, RDS for SQL Server, Oracle, RDS for SAP HANA, and GBase 8a Supported Supported You
In earlier versions, the dlg_agency automatically created by DataArts Studio has the administrator permissions of the following services by default: DLI, GaussDB(DWS), MRS, RDS, OBS, SMN, and KMS.
None None Importing a Connection (to Be Taken Offline) This API is used to import one or more connection files from OBS to the Data Development module.
For details about how to create a bucket on the OBS console, see Creating a Bucket in Object Storage Service Console Operation Guide. Upload data to OBS bucket fast-demo.
EXTERNAL: Data is stored in an OBS table. When Table Type is set to EXTERNAL, you must set OBS Path. The OBS path format is /bucket_name/filepath. DWS models support the following table types: DWS_ROW: row-store table. Tables are stored to disk partitions by row.
The field name cannot be the same as that of any source field (including custom fields). cdc_last_update_date Hudi table creation parameters Location OBS or HDFS path where database table files are stored - Hudi Table Type Storage type of the Hudi table MOR: Data is written to a log
Importing scripts, jobs, environment variables, and resources depends on the OBS service.
Case study Case: Trade Data Statistics and Analysis Consulting company H uses CDM to import local trade statistics to OBS, and uses Data Lake Insight (DLI) to analyze the trade statistics.
If you select Sample library for Random Mode, the OBS sample file can only be used for static DLI data masking tasks and the HDFS sample file can only be used for static MRS data masking tasks.
Archive to OBS: Dirty data is stored in OBS and printed in job logs. Do not archive Write Dirty Data Link This parameter is displayed when Dirty Data Policy is set to Archive to OBS.
EXTERNAL: Data is stored in an OBS table. When Table Type is set to EXTERNAL, you must set OBS Path. The OBS path format is /bucket_name/filepath. DWS models support the following table types: DWS_ROW: row-store table. Tables are stored to disk partitions by row.
MRS Impala Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS, HBase, or the Object Storage Service (OBS).
Before selecting a JAR package, upload the JAR package to the OBS bucket, create a resource on the Manage Resource page, and add the JAR package to the resource management list. For details, see Creating a Resource.
EXTERNAL: Data is stored in an OBS table. When Table Type is set to EXTERNAL, you must set OBS Path. The OBS path format is /bucket_name/filepath. DWS models support the following table types: DWS_ROW: row-store table. Tables are stored to disk partitions by row.
You can enter an HDFS or OBS path. The path format is as follows: OBS path: obs://bucket/warehouse HDFS path: /tmp/warehouse Global Configuration of Hudi Table Attributes N/A N/A Some advanced functions can be configured using parameters.
Figure 7 Orchestrating a job The node configurations are as follows: source_sdi: a CDM Job node, which is used to import data from OBS to the original table in MRS Hive.
Set this parameter when a DLI or SQL script is created. none bzip2 deflate gzip Storage Path Yes OBS path where the result file is stored. After selecting an OBS path, customize a folder. Then, the system will create it automatically for storing the result file.
You can select an HDFS or OBS path. Output Data Path Set the output data path. You can select an HDFS or OBS path.
