精选文章 Failed to initialize compiler: object scala.runtime in compiler mirror-Spark Shell由于Scala编译器原因不能正常启动

Failed to initialize compiler: object scala.runtime in compiler mirror-Spark Shell由于Scala编译器原因不能正常启动

作者:merrily01 时间: 2019-11-06 10:50:32
merrily01 2019-11-06 10:50:32

有日子没用之前搭的测试集群,今天想用来测试下 Spark一个问题,发现spark-shell启动失败了,报错信息如下:

hadoop@master:/software/servers/spark$ spark-shell
Listening for transport dt_socket at address: 9001
19/11/06 09:34:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/06 09:34:26 INFO SignalUtils: Registered signal handler for INT
[init] error: error while loading Object, Missing dependency 'object scala in compiler mirror', required by /data0/jdk/jdk1.8.0_161/jre/lib/rt.jar(java/lang/Object.class)

Failed to initialize compiler: object scala in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.

Failed to initialize compiler: object scala in compiler mirror not found.
** Note that as of 2.8 scala does not assume use of the java classpath.
** For the old behavior pass -usejavacp to scala, or if using a Settings
** object programmatically, settings.usejavacp.value = true.
Interpreter encountered errors during initialization!
19/11/06 09:34:28 INFO ShutdownHookManager: Shutdown hook called
19/11/06 09:34:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-e02e85a6-32a2-432a-8e3f-322e669655aa/repl-af263f59-afaa-4b6c-af08-40fe71511dd5
19/11/06 09:34:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-e02e85a6-32a2-432a-8e3f-322e669655aa
19/11/06 09:34:28 INFO ShutdownHookManager: Deleting directory /tmp/spark-06593104-4d74-4765-9f7f-4779b874f406

奇怪的是spark-shell --version / run-example SparkPi / spark-submit 等命令都可用。

因为之前为了规范集群环境,修改过环境变量等一些设置,不方便清楚的定位是哪里出了问题,google了一下,发现有如下几种解答:

原因:
2.8版本以上的scala将不再默认使用java的classpath.要解决这个问题,很显然,就是要在配置文件中添加使用java classpath的命令。
解决:
   修改\bin\spark-class2.cmd文件下的内容,在JAVA_OPTS里添加:-Dscala.usejavacp=true

因为我用的是spark-2.4.3代码,看了下spark-class2.cmd的内容和commit信息,发现JAVA_OPTS这块的逻辑早就变了。

经查spark-shell代码:


# SPARK-4161: scala does not assume use of the java classpath,
# so we need to add the "-Dscala.usejavacp=true" flag manually. We
# do this specifically for the Spark shell because the scala REPL
# has its own class loader, and any additional classpath specified
# through spark.driver.extraClassPath is not automatically propagated.
SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -Dscala.usejavacp=true"

发现,spark-shell已经在SPARK_SUBMIT_OPTS中添加了-Dscala.usejavacp=true的配置。不过这里也提醒了我,可能跟SPARK_SUBMIT_OPTS有关;

之前为了debug调试方便,将 SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001

配置到了spark-env中,程序启动的时候会设置该环境变量,去掉spark-env中自行设置的该环境变量后,问题解决。


那么,导致上述问题的根本原因到底是什么呢?

我们进一步分析一下:

当时背景

1. 我在spark-env中配置了:SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 ;
2. spark-submit / run-example 等命令可用,spark-shell不可用;
3. spark-shell代码中已经在SPARK_SUBMIT_OPTS中添加了-Dscala.usejavacp=true的配置项;

分析过程:

1. 问题肯定出在spark-shell中我自行配置的 SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001;
2. 为什么spark-submit、run-example 可用?
3. 为什么spark-shell不可用?

其实如果你熟悉spark提交过程,到这里就应该已经明白具体是因为啥了。

没错,根本原因就是:

spark-shell执行后,export了带有配置项-Dscala.usejavacp=true 的 SPARK_SUBMIT_OPTS,spark-shell会调用spark-submit,继而调用spark-class,然后调用load-spark-env.sh,也就会执行我自行设置的 SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001 ,导致 SPARK_SUBMIT_OPTS 被重写覆盖,反而失去了 -Dscala.usejavacp=true这个配置项,进一步导致上述问题。

照这样理解,如果我仍需要自行设置的配置的话应改为: 

SPARK_SUBMIT_OPTS="$SPARK_SUBMIT_OPTS -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=9001"

验证通过,在根本上问题已解决。


为了思路更清晰、更好理解,我在spark-shell和spark-env中加了输出日志来验证,更详细的验证过程就不贴出了,如果有疑惑,可以在评论中提问,我会适时回复~

 

PS:

本来就想把报错日志记录一下,结果啰里啰嗦这么多,希望能帮助到有需要的人~

-。- 

 

勿删,copyright占位
分享文章到微博
分享文章到朋友圈

上一篇:ATMJaDi恶意软件分析

下一篇:android 邮箱分享(添加附件筛选邮箱应用)

您可能感兴趣

华为云40多款云服务产品0元试用活动

免费套餐,马上领取!
CSDN

CSDN

中国开发者社区CSDN (Chinese Software Developer Network) 创立于1999年,致力为中国开发者提供知识传播、在线学习、职业发展等全生命周期服务。