检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
Figure 1 Viewing logs Obtain the folder of the archived logs in the OBS directory. The details are as follows: Spark SQL jobs: Locate the log folder whose name contains driver or container_ xxx _000001.
Notes Flink jobs can directly access DIS, OBS, and SMN data sources without using datasource connections. You are advised to use enhanced datasource connections to connect DLI to data sources.
Creating a datasource connection: VPC ReadOnlyAccess Creating yearly/monthly resources: BSS Administrator Creating a tag: TMS FullAccess and EPS EPS FullAccess Using OBS for storage: OBS OperateAccess Creating an agency: Security Administrator DLI ReadOnlyAccess Read-only permissions
Step 3: Create an OBS Bucket to Store Output Data Create an OBS bucket to store checkpoints, job logs, and debugging test data for the Flink OpenSource SQL job.
Figure 2 Selecting a data catalog on the SQL editor page When connecting DLI to a LakeFormation instance, you need to specify the OBS path for storing the database when creating it.
Insert the following data into the source Kafka topic: 202103251505050001,appshop,2021-03-25 15:05:05,500.00,400.00,2021-03-25 15:10:00,0003,Cindy,330108 202103241606060001,appShop,2021-03-24 16:06:06,200.00,180.00,2021-03-24 16:10:06,0001,Alice,330106 Read the ORC file in the OBS
Method 2: If you allow DLI to save job logs in OBS, view the output in the taskmanager.out file. +I(47.29.201.179 - - [28/Feb/2019:13:17:10 +0000] "GET /?p=1 HTTP/2.0"2005316"https://domain.com/?
Only the exclusive_cluster mode is supported. .withLogEnabled(true) // Enable the function of uploading job logs to OBS buckets. .withObsBucket("YourObsBucketName") // OBS bucket name, which is used to store logs and checkpoint data.
log_enabled=True, # Enable the function of uploading job logs to OBS buckets. obs_bucket="your_obs_bucket_name", # OBS bucket name, which is used to store logs and checkpoints. job_type="flink_opensource_sql_job", # Job type.
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
Insert the following data into the source Kafka topic: 202103251505050001,appShop,2021-03-25 15:05:05,500.00,400.00,2021-03-25 15:10:00,0003,Cindy,330108 202103241606060001,appShop,2021-03-24 16:06:06,200.00,180.00,2021-03-24 16:10:06,0001,Alice,330106 Read the Parquet file in the OBS
Range None result_path No String Definition OBS path of job results Range None execution_details_path No String Definition Path used to store the job execution plan.
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
Create a Kafka cluster for DMS, enable SASL_SSL, download the SSL certificate, and upload the downloaded certificate client.jks to an OBS bucket.
Range None config_url No String Definition OBS path of the config package uploaded by the user. Range None tm_cus No int Definition Number of CUs per TaskManager node. Range None tm_slot_num No int Definition Number of slots per TaskManager node.
The column name contains letters, digits, and underscores (_). using Uses hudi to define and create a Hudi table. table_comment Description of the table. location_path OBS path.
