检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
So, you must set LOCATION to an OBS path.
The options are as follows: jar: JAR file Pyfile: User Python file file: User file modelfile: User AI model file obs_jar_paths: OBS path of the resource package. The parameter format is {bucketName}.{obs domain name}/{jarPath}/{jarName}.
Configure an OBS bucket. Enable checkpointing. Checkpointing must be enabled when using Hudi. Submit the job and check the FlinkUI and logs. Click Submit in the upper right corner of the page.
Supported data sources Cloud: OBS, RDS, GaussDB(DWS), CSS, MongoDB, and Redis On-premises: self-built databases, MongoDB, and Redis Cloud: OBS On-premises: HDFS Ecosystem compatibility DLV, Yonghong BI, and Fanruan BI Big data ecosystem tool Custom image Supported.
View Log: Redirects to the OBS page where you can see the complete log archive addresses of the job, including commit logs, driver logs, and executor logs. You can download the logs here.
How Do I Set Up AK/SK So That a General Queue Can Access Tables Stored in OBS? How Do I View the Resource Usage of DLI Spark Jobs? How Do I Use Python Scripts to Access the MySQL Database If the pymysql Module Is Missing from the Spark Job Results Stored in MySQL?
You can configure the spark.sql.shuffle.partitions parameter to set the number of files to be inserted into the OBS bucket in the non-DLI table.
You can retain the default value. .load("obs://bucket/to_your_table"); // Specify the path of the Hudi table to read. DLI supports only OBS paths. dataFrame.show(100); // 2.
Select Save Job Log, and specify the OBS bucket for saving job logs.
Possible values are as follows: JAR: JAR file PyFile: User Python file File: User file ModelFile: User AI model file JAR OBS Path Select the OBS path of the corresponding package. NOTE: The program package must be uploaded to OBS in advance. Only files can be selected.
When this statement is used to drop a foreign table, the data in the OBS directory is not automatically deleted. When deleting an MOR table, the tables with the _rt and _ro suffixes are not automatically deleted. To delete them, you need to execute a DROP statement separately.
OBS Path: Specify the OBS path of the KafkaToKafka.properties file. Group Name: Enter a name for a new group or select an existing group name. Figure 6 Creating a DLI package Create a Flink Jar job and run it.
Select Save Job Log and select an OBS bucket. If the bucket is not authorized, click Authorize. This allows job logs be saved to your OBS bucket after a job fails for fault locating.
Upstream and downstream data connection Open-source connectors and out-of-the-box connectors for data sources including databases (RDS and GaussDB), message queues (DMS), data warehouses (GuassDB DWS), and object storage (OBS).
For details, see Creating an OBS Table or Creating a DLI Table.
How Do I Resolve an Unauthorized OBS Bucket Error?
For details about the OBS table permissions, see Table 2. Figure 4 Granting OBS table permissions to a user Figure 5 Granting OBS table permissions to a project Table 2 Parameter description Parameter Description Authorization Object Select User or Project.
You can use it to create a table for storing OBS data.
Only OBS tables can be created, which means the table path must be configured through the LOCATION parameter. When using the metadata service provided by LakeFormation, both internal and external tables are supported.
Improving the performance of OBS Committer when writing small files Improved the performance of Object Storage Service (OBS) when writing small files, improving data transfer efficiency.
