检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Figure 7 Orchestrating a job The node configurations are as follows: source_sdi: a CDM Job node, which is used to import data from OBS to the original table in MRS Hive.
Set this parameter when a DLI or SQL script is created. none bzip2 deflate gzip Storage Path Yes OBS path where the result file is stored. After selecting an OBS path, customize a folder. Then, the system will create it automatically for storing the result file.
You can select an HDFS or OBS path. Output Data Path Set the output data path. You can select an HDFS or OBS path.
SET SEARCH_PATH TO dgc; SELECT * FROM top_active_movie Figure 6 Viewing the data in the top_active_movie table Developing and Scheduling a Job Assume that the movie and rating tables in the OBS bucket are changing in real time.
current day exceeds the specifications of this version, a message is displayed indicating that the number of job node scheduling times/day exceeds the quota when a batch processing job is scheduled or a real-time job is started. [3] Number of technical assets: number of tables and OBS
On the Details tab page, view the basic attributes of the technical metadata, add or delete classifications, tags, and security levels for the table, table columns, or OBS objects, and edit the description.
EXTERNAL: Data is stored in an OBS table. When Table Type is set to EXTERNAL, you must set OBS Path. The OBS path format is /bucket_name/filepath. DWS models support the following table types: DWS_ROW: Tables are stored to disk partitions by row.
This quota is calculated based on the total number of tables and OBS files in DataArts Catalog. You can locate a DataArts Studio instance, click More, and select Quota Usage to view this quota.
Select the created DMS for Kafka and OBS data connections and the migration resource group for which the network connection has been configured.
For example, if you want to enable communication between DataArts Studio (containing modules such as Management Center and CDM) and services in other regions (such as MRS and OBS), use a public network or Direct Connect.
The corresponding link parameters are as follows: generic-jdbc-connector: link to relational database obs-connector: link to OBS hdfs-connector: link to HDFS hbase-connector: link to HBase and link to CloudTable hive-connector: link to Hive ftp-connector/sftp-connector: link to an
If files are migrated between FTP, SFTP, HDFS, and OBS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping. You can create a field converter on the Map Field page when creating a table/file migration job.
NOTE: If the destination is OBS, only the binary format is supported. CSV JSON Type This parameter is displayed only when File Format is set to JSON. Type of a JSON object stored in a JSON file. The options are JSON object and JSON array.
If OBS is unavailable in the same region as DataArts Studio, RDS data connections are not supported. For host connections, only Linux hosts are supported.
If files are migrated between FTP, SFTP, OBS, and HDFS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping. You can create a field converter on the Map Field page when creating a table/file migration job.
If files are migrated between FTP, SFTP, OBS, and HDFS and the migration source's File Format is set to Binary, files will be directly transferred, free from field mapping. You can create a field converter on the Map Field page when creating a table/file migration job.
Column names are displayed when the source of the migration job is OBS, CSV files are to be migrated, and parameter Extract first row as columns is set to Yes. Field mapping is not involved when the binary format is used to migrate files to files.
When you select HIVE for Data Source Type, you can change Database to URL to authorize an OBS path in the storage-compute decoupling scenario.
The corresponding link parameter is generic-jdbc-connector, which indicates a relational database link. obs-connector: link to OBS hdfs-connector: link to HDFS hbase-connector: link to HBase and link to CloudTable hive-connector: link to Hive ftp-connector/sftp-connector: link to
When you select HIVE for Data Source Type, you can change Database to URL to authorize an OBS path in the storage-compute decoupling scenario.
