检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
检测到您已登录华为云国际站账号，为了您更好的体验，建议您访问国际站服务网站 https://www.huaweicloud.com/intl/zh-cn
不再显示此消息
Only OBS tables can be created, which means the table path must be configured through the LOCATION parameter. When using the metadata service provided by LakeFormation, both internal and external tables are supported.
You can directly use SQL to read and write data from cloud services, such as DIS, OBS, CloudTable, MRS, RDS, SMN and DCS.
Mapping Between DLI SDK V1 and APIs OBS Authorization Table 1 Mapping between OBS authorization APIs and SDKs Class Method Java Method Python Method API Authorize OBS authorization authorizeBucket - POST /v1.0/{project_id}/dli/obs-authorize Queue-related SDKs Table 2 Mapping between
Submitting a Spark job Upload the Python code file to the OBS bucket. In the Spark job editor, select the corresponding dependency module and execute the Spark job. For Spark 2.3.2 (soon to be take offline) or 2.4.5, set Module to sys.datasource.css when submitting a job.
`obs://tablePath` LIMIT 1; System Response Returns the table's historical operations, with the meaning of the result indicators shown in the following table.
When using the metadata service provided by DLI, this command does not support OBS paths. System Response You can check if the job status is successful, view the job result, and review the job logs to confirm if there are any exceptions. Parent topic: Hudi DML Syntax
option("es.net.http.auth.pass", "***") .load() dataFrameR.show() Before data is inserted: Response: Submitting a Spark job Generate a JAR file based on the code file and upload the JAR file to the OBS bucket.
Flink Jar Jobs Using DEW to Acquire Access Credentials for Reading and Writing Data from and to OBS Guideline for Flink Jar jobs to acquire an AK/SK to read and write data from and to OBS.
The options are as follows: jar: JAR file Pyfile: User Python file file: User file modelfile: User AI model file paths: OBS path of the resource package. The parameter format is {bucketName}.
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
Select Save Job Log, and specify the OBS bucket for saving job logs. Change the values of the parameters in bold as needed in the following script.
OBS Table, DLI Table, and CloudTable Table The table type indicates the storage location of data. OBS table indicates that data is stored in the OBS bucket. DLI table indicates that data is stored in the internal table of DLI.
Export Result: Export the job running result to the created OBS bucket. In addition to the preceding operations, the EXPORT job also includes the following operations: Download View Log: Save job logs to the temporary OBS bucket created by DLI.
DLI supports SQL statements and Spark applications for heterogeneous data sources, including CloudTable, RDS, DWS, CSS, OBS, custom databases on ECSs, and offline databases.
You can directly use SQL statements to read and write data from various cloud services, such as Data Ingestion Service (DIS), Object Storage Service (OBS), CloudTable Service (CloudTable), MapReduce Service (MRS), Relational Database Service (RDS), Simple Message Notification (SMN
Step 3: Create an OBS Bucket to Store Output Data Create an OBS bucket to store checkpoints, job logs, and debugging test data for the Flink OpenSource SQL job.
OBS Bucket: Select an OBS bucket for storing job logs and grant access permissions of the OBS bucket as prompted. Enable Checkpointing: Enable this function. Enter the SQL statement in the editing pane. The following is an example. Modify the parameters in bold as you need.
The default value is 10. obs_bucket No String OBS bucket where users are authorized to save the snapshot. This parameter is valid only when checkpoint_enabled is set to true. OBS bucket where users are authorized to save the snapshot.
`obs://bucket_name0/db0/delta_table0` add columns(new_col string); Response You can run the DESCRIBE command to view the new column. Parent topic: Schema Evolution Syntax
When data is stored on OBS, any charges for storage resource usage will be billed by OBS, not DLI. Billing for scanned data You are billed based on the scanned data volume in each job, in GB.
