数据湖探索 DLI-使用Spark作业访问DLI元数据:Java样例代码

时间：2024-05-29 17:18:44

数据湖探索 DLI Spark Jar作业开发指南

Java样例代码

本示例操作步骤采用Java进行编码，具体完整的样例代码参考如下：

package com.huawei.dli.demo;

import org.apache.spark.sql.SparkSession;

public class DliCatalogTest {
    public static void main(String[] args) {

        SparkSession spark = SparkSession
                .builder()
                .config("spark.sql.session.state.builder", "org.apache.spark.sql.hive.UQueryHiveACLSessionStateBuilder")
                .config("spark.sql.catalog.class", "org.apache.spark.sql.hive.UQueryHiveACLExternalCatalog")
                .config("spark.sql.extensions","org.apache.spark.sql.DliSparkExtension")
                .appName("java_spark_demo")
                .getOrCreate();

        spark.sql("create database if not exists test_sparkapp").collect();
        spark.sql("drop table if exists test_sparkapp.dli_testtable").collect();
        spark.sql("create table test_sparkapp.dli_testtable(id INT, name STRING)").collect();
        spark.sql("insert into test_sparkapp.dli_testtable VALUES (123,'jason')").collect();
        spark.sql("insert into test_sparkapp.dli_testtable VALUES (456,'merry')").collect();

        spark.sql("drop table if exists test_sparkapp.dli_testobstable").collect();
        spark.sql("create table test_sparkapp.dli_testobstable(age INT, name STRING) using csv options (path 'obs://dli-test-obs01/testdata.csv')").collect();


        spark.stop();

    }
}

上一篇：数据湖探索 DLI-使用Spark作业访问DLI元数据:步骤8：查看作业运行结果

下一篇：数据湖探索 DLI-使用Spark作业访问DLI元数据:步骤8：查看作业运行结果