Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)

peterliu
Hello there,

I didn't get any response/help from the user list for the following question and thought people on the dev list might be able to help?:

I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload and got the error (java.lang.AbstractMethodError) never seen before; see the error exception stack attached in (a) bellow.

anyone knows if  spark 2.3.1 works well with kafka spark-streaming-kafka-0-10?

this link of spark kafka integration page doesn't say anything about any limitation:

but this discussion seems to say there is indeed an issue when upgrading to spark 2.3.1:

i also rebuilt the workload with some spark 2.3.1 jars (see (b) below). it doesn't seem to help.

Would be great if anyone could kindly share any insights here.

Thanks!

Peter

(a) the exception
Exception in thread "stream execution thread for [id = 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId = 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
        at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializeLogIfNecessary(KafkaSourceProvider.scala:369)
        at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(KafkaSourceProvider.scala:369)
        at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(KafkaSourceProvider.scala:369)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpdater.set(KafkaSourceProvider.scala:439)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaParamsForDriver(KafkaSourceProvider.scala:394)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:90)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:277)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
        at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
        at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:75)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:265)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)

(b) the build script update:

[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff  build.sbt spk211-build.sbt.original
10,11c10,11
< libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.3.1"
< libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.3.1"
---
> libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
> libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$

 


Reply | Threaded
Open this post in threaded view
|

Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)

Sean Owen-2
Somewhere, you have mismatched versions of Spark on your classpath. 

On Sun, Jul 1, 2018, 9:01 AM Peter Liu <[hidden email]> wrote:
Hello there,

I didn't get any response/help from the user list for the following question and thought people on the dev list might be able to help?:

I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload and got the error (java.lang.AbstractMethodError) never seen before; see the error exception stack attached in (a) bellow.

anyone knows if  spark 2.3.1 works well with kafka spark-streaming-kafka-0-10?

this link of spark kafka integration page doesn't say anything about any limitation:

but this discussion seems to say there is indeed an issue when upgrading to spark 2.3.1:

i also rebuilt the workload with some spark 2.3.1 jars (see (b) below). it doesn't seem to help.

Would be great if anyone could kindly share any insights here.

Thanks!

Peter

(a) the exception
Exception in thread "stream execution thread for [id = 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId = 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
        at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializeLogIfNecessary(KafkaSourceProvider.scala:369)
        at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(KafkaSourceProvider.scala:369)
        at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(KafkaSourceProvider.scala:369)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpdater.set(KafkaSourceProvider.scala:439)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaParamsForDriver(KafkaSourceProvider.scala:394)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:90)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:277)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
        at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
        at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:75)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:265)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)

(b) the build script update:

[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff  build.sbt spk211-build.sbt.original
10,11c10,11
< libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.3.1"
< libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.3.1"
---
> libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
> libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$

 


Reply | Threaded
Open this post in threaded view
|

Re: spark 2.3.1 with kafka spark-streaming-kafka-0-10 (java.lang.AbstractMethodError)

peterliu
Hi Sean,

thanks for your reply (sorry for the late response - it was a long weekend here).

do you happen to have any suggestion for any specific area or jars I should take a close look? (i have the spark environment page attached below).

This is very much appreciated!

Peter

Environment

Runtime Information

NameValue
Java Home/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
Java Version1.8.0-internal (Oracle Corporation)
Scala Versionversion 2.11.8

Spark Properties

NameValue
spark.serializerorg.apache.spark.serializer.KryoSerializer
spark.executor.extraJavaOptions-XX:ParallelGCThreads=4
spark.driver.hostisvhdp11lab.spark.ibm.com
spark.history.fs.logDirectoryhdfs://isvhdp7lab/sparkLogs
spark.eventLog.enabledtrue
spark.executor.heartbeatInterval60s
spark.ui.port0
spark.driver.port39641
spark.yarn.app.idapplication_1530116533665_0008
spark.app.nameConsumer Cfg: Stream_Dur300000_RecPerSec60000_camp100_recGenNumP10
spark.network.timeout1200
spark.scheduler.modeFIFO
spark.driver.memory4g
spark.executor.instances60
spark.yarn.secondary.jarsorg.apache.kafka_kafka-clients-0.10.0.1.jar,org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar,org.spark-project.spark_unused-1.0.0.jar,net.jpountz.lz4_lz4-1.3.0.jar,org.xerial.snappy_snappy-java-1.1.2.6.jar,org.slf4j_slf4j-api-1.7.16.jar
spark.executor.iddriver
spark.streaming.stopGracefullyOnShutdowntrue
spark.yarn.app.container.log.dir*********(redacted)
spark.submit.deployModecluster
spark.masteryarn
spark.ui.filtersorg.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
spark.executor.memory3g
spark.eventLog.dirhdfs://isvhdp7lab/sparkLogs
spark.executor.cores1
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_HOSTSisvhdp7.aus.stglabs.ibm.com
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASEShttp://isvhdp7.aus.stglabs.ibm.com:8088/proxy/application_1530116533665_0008
spark.yarn.dist.jarsfile:///home/pgl/SparkStreamingBenchmark-RemoteConsumer/org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///home/pgl/.ivy2/jars/org.apache.spark_spark-sql-kafka-0-10_2.11-2.2.0.jar,file:///home/pgl/.ivy2/jars/org.apache.kafka_kafka-clients-0.10.0.1.jar,file:///home/pgl/.ivy2/jars/org.spark-project.spark_unused-1.0.0.jar,file:///home/pgl/.ivy2/jars/net.jpountz.lz4_lz4-1.3.0.jar,file:///home/pgl/.ivy2/jars/org.xerial.snappy_snappy-java-1.1.2.6.jar,file:///home/pgl/.ivy2/jars/org.slf4j_slf4j-api-1.7.16.jar
spark.app.idapplication_1530116533665_0008
spark.sql.shuffle.partitions40

System Properties

NameValue
java.io.tmpdir/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/tmp
line.separator
path.separator:
sun.management.compilerHotSpot 64-Bit Tiered Compilers
sun.cpu.endianlittle
java.specification.version1.8
java.vm.specification.nameJava Virtual Machine Specification
java.vendorOracle Corporation
java.vm.specification.version1.8
user.home/home/pgl
file.encoding.pkgsun.io
sun.nio.ch.bugLevel
sun.arch.data.model64
sun.boot.library.path/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ppc64le
user.dir/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001
java.library.path/home/maha/hadoop-2.9.0/lib/native/:/home/maha/hadoop-2.9.0/lib/native/:/home/pgl/hadoop-2.9.0/lib/native::/usr/java/packages/lib/ppc64le:/usr/lib64:/lib64:/lib:/usr/lib
sun.cpu.isalist
os.arch
ppc64le
java.vm.version25.71-b00
java.endorsed.dirs/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/endorsed
java.runtime.version1.8.0-internal-root_2018_04_19_13_32-b00
java.vm.infomixed mode
java.ext.dirs/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/ext:/usr/java/packages/lib/ext
java.runtime.nameOpenJDK Runtime Environment
file.separator/
java.class.version52.0
java.specification.nameJava Platform API Specification
sun.boot.class.path/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/resources.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/rt.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/sunrsasign.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/jsse.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/jce.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/charsets.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/lib/jfr.jar:/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre/classes
file.encodingUTF-8
user.timezoneAmerica/Chicago
java.specification.vendorOracle Corporation
sun.java.launcherSUN_STANDARD
java.vm.compressedOopsModeZero based
os.version4.14.0-81.el7a.ppc64le
sun.os.patch.levelunknown
java.vm.specification.vendorOracle Corporation
user.countryUS
sun.jnu.encodingUTF-8
user.languageen
java.vendor.urlhttp://java.oracle.com/
java.awt.printerjobsun.print.PSPrinterJob
java.awt.graphicsenvsun.awt.X11GraphicsEnvironment
awt.toolkitsun.awt.X11.XToolkit
os.nameLinux
java.vm.vendorOracle Corporation
java.vendor.url.bughttp://bugreport.sun.com/bugreport/
user.namepgl
java.vm.nameOpenJDK 64-Bit Server VM
sun.java.commandorg.apache.spark.deploy.yarn.ApplicationMaster --class SparkStreamingTest --jar file:/home/pgl/SparkStreamingBenchmark-RemoteConsumer/target/scala-2.11/sparkstreamingbenchmark-remoteconsumer_2.11-0.1-SNAPSHOT.jar --arg 300000 --arg 60000 --arg 20 --arg 10 --arg 100 --arg 80 --arg 80 --arg 80 --arg Stream_Dur300000_RecPerSec60000_camp100_recGenNumP10 --properties-file /spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__/__spark_conf__.properties
java.home/home/pgl/root-dir/jdk8_trlopt_55cf01d/jre
java.version1.8.0-internal
sun.io.unicode.encodingUnicodeLittle

Classpath Entries

ResourceSource
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-core-1.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax-api-1.0-2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-beanutils-core-1.8.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guava-11.0.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/datanucleus-api-jdo-3.2.6.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/guava-11.0.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-util-1.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-hive_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javax.inject-1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-recipes-2.7.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/libthrift-0.9.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/okio-1.13.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-cli-1.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-3.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/leveldbjni-all-1.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-network-common_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/okhttp-3.8.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/junit-4.11.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jersey-server-1.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/json4s-ast_2.11-3.2.11.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-configuration-1.6.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/avro-1.7.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/generex-1.0.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arpack_combined_all-0.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-common-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-jaxrs-1.9.13.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-cli-1.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/kubernetes-client-3.0.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/eigenbase-properties-1.1.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/pyrolite-4.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang-2.6.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0-tests.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-guice-1.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-sslengine-6.1.26.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-kubernetes_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/log4j-1.2.17.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spire_2.11-0.13.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-encoding-1.8.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-common-2.22.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-util-1.0.0-M20.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-6.1.26.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-test-2.7.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-core-asl-1.9.13.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-net-2.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-server-1.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/avro-1.7.7.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/logging-interceptor-3.8.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-compress-1.4.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-auth-2.7.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-xc-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-net-3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-jvm-3.1.5.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/guice-servlet-3.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-registry-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/machinist_2.11-0.6.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/xz-1.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpclient-4.5.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jetty-util-6.1.26.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stax-api-1.0.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jsr305-1.3.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/netty-all-4.1.17.Final.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arrow-memory-0.8.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xml-apis-1.3.04.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/antlr-runtime-3.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.inject-1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/orc-core-1.4.4-nohive.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-io-2.5.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-core-3.1.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-graphx_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/chill_2.11-0.8.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-launcher_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/paranamer-2.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-compress-1.4.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-client-2.7.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpclient-4.5.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-hadoop-bundle-1.6.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/avro-1.7.7.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-cli-1.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/zjsonpatch-0.3.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/curator-framework-2.7.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-guava-2.22.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/gson-2.2.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-catalyst_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/snappy-0.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/log4j-1.2.17.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/compress-lzf-1.0.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-beeline-1.2.1.spark2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-digester-1.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/metrics-core-3.0.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/super-csv-2.2.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-dataformat-yaml-2.6.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-core-2.6.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/JavaEWAH-0.3.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.ws.rs-api-2.0.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-core-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/apache-log4j-extras-1.2.17.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/asm-3.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/ehcache-3.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-json-1.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/asm-3.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsp-api-2.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javolution-5.5.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-databind-2.6.7.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xmlenc-0.52.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-mesos_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/janino-3.0.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/htrace-core-3.1.0-incubating.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-common-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/breeze_2.11-0.13.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang-2.6.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-format-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/snappy-java-1.0.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/mesos-1.4.0-shaded-protobuf.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/woodstox-core-5.0.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-core-1.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-nfs-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/servlet-api-2.5.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/protobuf-java-2.5.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-dbcp-1.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-core-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-logging-1.1.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/asm-3.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jcl-over-slf4j-1.7.16.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math-2.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/javax.inject-1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jets3t-0.9.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/libfb303-0.9.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/core-1.1.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-mllib_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jpam-1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-util-6.1.26.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-client-2.7.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-core_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/api-asn1-api-1.0.0-M20.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jtransforms-2.4.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-network-shuffle_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-math3-3.4.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/snappy-java-1.0.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-column-1.8.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/mockito-all-1.8.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jaxb-api-2.2.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/breeze-macros_2.11-0.13.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-kvstore_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/base64-2.3.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-xml_2.11-1.0.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/oro-2.0.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-1.7.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jetty-6.1.26.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/aopalliance-repackaged-2.4.0-b34.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-logging-1.1.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-hdfs-2.7.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jets3t-0.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/guava-11.0.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/xmlenc-0.52.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-common-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-graphite-3.1.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-compiler-2.11.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-server-2.22.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/aopalliance-1.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/joda-time-2.9.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/netty-3.9.9.Final.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/activation-1.1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax2-api-3.1.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/gson-2.2.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsch-0.1.54.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-module-jaxb-annotations-2.6.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scalap-2.11.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/py4j-0.10.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xmlenc-0.52.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/curator-client-2.7.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javassist-3.18.1-GA.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-core-1.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/log4j-1.2.17.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/kryo-shaded-3.0.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/httpcore-4.4.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/calcite-core-1.2.0-incubating.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-compiler-3.0.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-recipes-2.7.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-servlet-3.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-server-common-2.7.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/zookeeper-3.4.6.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-configuration-1.6.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stringtemplate-3.2.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/curator-framework-2.7.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-io-2.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/antlr-2.7.7.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsr305-3.0.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-math3-3.1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-2.7.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-lang-2.6.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-jackson-1.8.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-mllib-local_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-parser-combinators_2.11-1.0.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.inject-2.4.0-b34.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-client-2.22.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsch-0.1.54.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/httpclient-4.5.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-httpclient-3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jsr305-3.0.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/lz4-java-1.4.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jta-1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/netty-3.6.2.Final.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/protobuf-java-2.5.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/snakeyaml-1.15.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-auth-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/datanucleus-core-3.2.10.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/avro-mapred-1.7.7-hadoop2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/aopalliance-1.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.servlet-api-3.1.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/etc/hadoopSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-media-jaxb-2.22.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-unsafe_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-xc-1.9.13.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/antlr4-runtime-4.7.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.0-tests.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-sketch_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-asn1-api-1.0.0-M20.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/objenesis-2.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/guava-14.0.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/slf4j-api-1.7.16.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/avro-1.7.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/ivy-2.4.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-client-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__/__hadoop_conf__System Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-6.1.26.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/slf4j-api-1.7.25.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-pool-1.5.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/curator-framework-2.7.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/xz-1.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hk2-api-2.4.0-b34.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jettison-1.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/calcite-linq4j-1.2.0-incubating.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hppc-0.7.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hadoop-annotations-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-collections-3.2.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-module-scala_2.11-2.6.7.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/guice-3.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/kubernetes-model-2.0.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-jdbc-1.2.1.spark2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-io-2.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-repl_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-hadoop-1.8.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/osgi-resource-locator-1.0.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-sslengine-6.1.26.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-api-2.2.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-annotations-2.6.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/opencsv-2.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/log4j-1.2.17.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/apacheds-i18n-2.0.0-M15.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-collections-3.2.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/guice-3.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/activation-1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-io-2.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-codec-1.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/aircompressor-0.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/nimbus-jose-jwt-3.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-router-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/zstd-jni-1.3.2-2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-yarn_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/json4s-jackson_2.11-3.2.11.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/validation-api-1.1.0.Final.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/parquet-common-1.8.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/javassist-3.18.1-GA.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-lang3-3.5.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okio-1.4.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/metrics-json-3.1.5.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jline-2.12.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/log4j-1.2.17.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/paranamer-2.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-codec-1.10.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/stax-api-1.0-2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/fst-2.50.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-configuration-1.6.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/json-smart-1.1.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-client-2.7.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xmlenc-0.52.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/derby-10.12.1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_conf__System Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/paranamer-2.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/okhttp-2.4.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-2.9.0-tests.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jcip-annotations-1.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/minlog-1.3.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-io-2.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/flatbuffers-1.2.0-3f79e055.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/stax2-api-3.1.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/json4s-core_2.11-3.2.11.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-cli-1.2.1.spark2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/httpcore-4.4.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/netty-3.6.2.Final.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hk2-utils-2.4.0-b34.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-api-2.7.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jettison-1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jackson-xc-1.9.13.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/curator-recipes-2.7.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-container-servlet-core-2.22.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/api-util-1.0.0-M20.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/hamcrest-core-1.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-math3-3.1.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hk2-locator-2.4.0-b34.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-crypto-1.0.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/ST4-4.0.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-core-1.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/httpcore-4.4.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/guice-servlet-3.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-lang-2.6.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/hadoop-common-2.9.0-tests.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-sql_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/json-smart-1.1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/snappy-java-1.0.5.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-native-client-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xercesImpl-2.9.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-library-2.11.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jetty-6.1.26.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-server-common-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-digester-1.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-beanutils-1.7.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jsp-api-2.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jersey-guice-1.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-yarn-common-2.7.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/api-util-1.0.0-M20.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/automaton-1.11-8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-net-3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/htrace-core4-4.1.0-incubating.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/woodstox-core-5.0.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-api-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jets3t-0.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/univocity-parsers-2.5.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-codec-1.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stream-2.7.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spire-macros_2.11-0.13.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/datanucleus-rdbms-3.2.9.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-lang3-3.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/hadoop-annotations-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/aopalliance-1.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-streaming_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-lang3-3.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jersey-server-1.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/zookeeper-3.4.6.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/snappy-java-1.1.2.6.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001System Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-io-2.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-tags_2.11-2.3.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jodd-core-3.5.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/RoaringBitmap-0.5.11.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jersey-container-servlet-2.22.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/slf4j-log4j12-1.7.16.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/java-xmlbuilder-0.4.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/activation-1.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arrow-vector-0.8.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/paranamer-2.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-annotations-2.7.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/asm-3.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/arrow-format-0.8.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/servlet-api-2.5.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/junit-4.11.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/shapeless_2.11-2.3.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-client-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-module-paranamer-2.7.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jsp-api-2.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/commons-beanutils-core-1.8.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/macro-compat_2.11-1.1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-client-1.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/spark-hive-thriftserver_2.11-2.3.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.9.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/stax-api-1.0-2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/commons-cli-1.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/xz-1.0.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/servlet-api-2.5.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-json-1.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/bonecp-0.8.0.RELEASE.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/hadoop-hdfs-client-2.9.0-tests.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/bcprov-jdk15on-1.58.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/jersey-server-1.9.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/chill-java-0.8.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-collections-3.2.2.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jackson-mapper-asl-1.9.13.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/leveldbjni-all-1.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/javax.annotation-api-1.2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/gson-2.2.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hadoop-mapreduce-client-app-2.7.3.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jdo-api-3.0.1.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/avro-ipc-1.7.7.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xz-1.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/jul-to-slf4j-1.7.16.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jsr305-3.0.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/calcite-avatica-1.2.0-incubating.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/java-xmlbuilder-1.1.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/commons-codec-1.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-metastore-1.2.1.spark2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/scala-reflect-2.11.8.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/orc-mapreduce-1.4.4-nohive.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/hive-exec-1.2.1.spark2.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/xbean-asm5-shaded-4.4.jarSystem Classpath
/spark2/pgl/yarn/usercache/pgl/appcache/application_1530116533665_0008/container_1530116533665_0008_02_000001/__spark_libs__/commons-digester-1.8.jarSystem Classpath
/home/pgl/hadoop-2.9.0/share/hadoop/common/lib/jcip-annotations-1.0.jar


On Sun, Jul 1, 2018 at 10:09 AM, Sean Owen <[hidden email]> wrote:
Somewhere, you have mismatched versions of Spark on your classpath. 


On Sun, Jul 1, 2018, 9:01 AM Peter Liu <[hidden email]> wrote:
Hello there,

I didn't get any response/help from the user list for the following question and thought people on the dev list might be able to help?:

I upgraded to spark 2.3.1 from spark 2.2.1, ran my streaming workload and got the error (java.lang.AbstractMethodError) never seen before; see the error exception stack attached in (a) bellow.

anyone knows if  spark 2.3.1 works well with kafka spark-streaming-kafka-0-10?

this link of spark kafka integration page doesn't say anything about any limitation:

but this discussion seems to say there is indeed an issue when upgrading to spark 2.3.1:

i also rebuilt the workload with some spark 2.3.1 jars (see (b) below). it doesn't seem to help.

Would be great if anyone could kindly share any insights here.

Thanks!

Peter

(a) the exception
Exception in thread "stream execution thread for [id = 5adae836-268a-4ebf-adc4-e3cc9fbe5acf, runId = 70e78d5c-665e-4c6f-a0cc-41a56e488e30]" java.lang.AbstractMethodError
        at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.initializeLogIfNecessary(KafkaSourceProvider.scala:369)
        at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.log(KafkaSourceProvider.scala:369)
        at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.logDebug(KafkaSourceProvider.scala:369)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$ConfigUpdater.set(KafkaSourceProvider.scala:439)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider$.kafkaParamsForDriver(KafkaSourceProvider.scala:394)
        at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:90)
        at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:277)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:80)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1$$anonfun$applyOrElse$1.apply(MicroBatchExecution.scala:77)
        at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
        at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:77)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution$$anonfun$1.applyOrElse(MicroBatchExecution.scala:75)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
        at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
        at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan$lzycompute(MicroBatchExecution.scala:75)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.logicalPlan(MicroBatchExecution.scala:61)
        at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:265)
        at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)

(b) the build script update:

[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$ diff  build.sbt spk211-build.sbt.original
10,11c10,11
< libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.3.1"
< libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.3.1"
---
> libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.2.1"
> libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.1"
[pgl@datanode20 SparkStreamingBenchmark-RemoteConsumer-Spk231]$