云服务器内容精选

华为云首页用户手册

Sqoop常见问题

MAPREDUCE服务 MRS-连接postgresql或者gaussdb时报错:回答

回答场景一：（import场景）使用sqoop import命令抽取开源postgre到MRS hdfs或hive等。问题现象：使用sqoop命令查询postgre表可以，但是执行sqoop import命令倒数时报错： The authentication type 12 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it The authentication type 5 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it 问题根因：报错中type为5时：在执行sqoop import命令时，会启动MapReduce任务，由于MRS Hadoop安装目录（/opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib）下自带了postgre驱动包gsjdbc4-*.jar，与开源postgre服务不兼容导致报错。报错中type为12时：数据库的pg_hba.conf文件配置有误。解决方案：报错中type为5时：在每台MRS NodeManager实例所在节点上移动驱动包gsjdbc4-*.jar到tmp目录下。 mv /opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib/gsjdbc4-*.jar /tmp 报错中type为12时：调整数据库的pg_hba.conf文件，将address改成sqoop所在节点的ip。场景二：（export场景）使用sqoop export命令抽取开源postgre到MRS hdfs或hive等。问题现象：使用sqoop命令查询postgre表可以，但是执行sqoop export命令倒数时报错：The authentication type 5 is not supported. Check that you have configured the pg_hba.conf file to include the client's IP address or subnet, and that it 问题根因：在执行sqoop export命令时，会启动MapReduce任务，由于MRS Hadoop安装目录（/opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib）下自带了postgre驱动包gsjdbc4-*.jar，与开源postgre服务不兼容导致报错。解决方案：在每台MRS NodeManager实例所在节点上移动驱动包gsjdbc4-*.jar到tmp目录下。 mv /opt/Bigdata/FusionInsight_HD_*/1_*_NodeManager/install/hadoop/share/hadoop/common/lib/gsjdbc4-*.jar /tmp 将/opt/Bigdata/client/Hive/Beeline/lib/gsjdbc4-*.jar删除。

MAPREDUCE服务 MRS Sqoop常见问题
MapReduce服务 MRS-使用hcatalog方式同步数据，报错getHiveClient方法不存在:回答

回答将https://repo.huaweicloud.com/repository/maven/huaweicloudsdk/org/apache/hive/hcatalog/hive-hcatalog-core/源下的jar包替换到mrs客户端的hcatalog的目录下，并重命名之前的同名hcatalog的jar包。如图302002就是替换后的包，310001-SNAPSHOT.jar.bak就是加了.bak后缀的原包。图1 hcatalog目录图2 替Jar包

MapReduce服务 MRS Sqoop常见问题