Spark Thirft Service not started in Hadoop with Azure Storage Blob configuration

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark Thirft Service not started in Hadoop with Azure Storage Blob configuration

Selva
This post has NOT been accepted by the mailing list yet.
We have Created a High availability hadoop cluster with default file system as azure blob storage instead of hdfs by following the link https://hadoop.apache.org/docs/stable/hadoop-azure/index.html

Hivethrift service where started successfully but spark thift service where not started. I can able to connect the spark-shell and connect with blob by referening the jar file hadoop-azure.jar but cannot start the thrift service.

Command used to start spark thrift server: spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --master yarn



Error details:



17/04/26 10:19:32 INFO metastore: Connected to metastore.
Exception in thread "main" java.lang.IllegalArgumentException: Error while insta
ntiating 'org.apache.spark.sql.hive.HiveSessionState':
        at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$
$reflect(SparkSession.scala:981)
        at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSessio
n.scala:110)
        at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109
)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.appl
y(SparkSession.scala:878)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.appl
y(SparkSession.scala:878)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.sca
la:99)
        at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.sca
la:99)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala
:230)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.sc
ala:878)
        at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.
scala:47)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveTh
riftServer2.scala:81)
        at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThr
iftServer2.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSub
mit$$runMain(SparkSubmit.scala:738)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:18
7)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
orAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
onstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$
$reflect(SparkSession.scala:978)
        ... 22 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.ap
ache.spark.sql.hive.HiveExternalCatalog':
        at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$inter
nal$SharedState$$reflect(SharedState.scala:169)
        at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86
)
        at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkS
ession.scala:101)
        at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkS
ession.scala:101)
        at scala.Option.getOrElse(Option.scala:121)
        at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession
.scala:101)
        at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)

        at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:
157)
        at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.sc
ala:32)
        ... 27 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
orAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
onstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$inter
nal$SharedState$$reflect(SharedState.scala:166)
        ... 35 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
orAccessorImpl.java:57)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
onstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(Is
olatedClientLoader.scala:264)
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.s
cala:366)
        at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.s
cala:270)
        at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCata
log.scala:65)
        ... 40 more
Caused by: java.lang.RuntimeException: org.apache.hadoop.fs.azure.AzureException
: java.util.NoSuchElementException: An error occurred while enumerating the resu
lt, check the original exception for details.
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
a:522)
        at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl
.scala:192)
        ... 48 more
Caused by: org.apache.hadoop.fs.azure.AzureException: java.util.NoSuchElementExc
eption: An error occurred while enumerating the result, check the original excep
tion for details.
        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadat
a(AzureNativeFileSystemStore.java:1930)
        at org.apache.hadoop.fs.azure.NativeAzureFileSystem.getFileStatus(Native
AzureFileSystem.java:1592)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
        at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(Sess
ionState.java:596)
        at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(Sess
ionState.java:554)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
a:508)
        ... 49 more
Caused by: java.util.NoSuchElementException: An error occurred while enumerating
 the result, check the original exception for details.
        at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySe
gmentedIterator.java:113)
        at org.apache.hadoop.fs.azure.StorageInterfaceImpl$WrappingIterator.hasN
ext(StorageInterfaceImpl.java:128)
        at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadat
a(AzureNativeFileSystemStore.java:1909)
        ... 54 more
Caused by: com.microsoft.azure.storage.StorageException: The server encountered
an unknown failure: OK
        at com.microsoft.azure.storage.StorageException.translateException(Stora
geException.java:178)
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(Exe
cutionEngine.java:273)
        at com.microsoft.azure.storage.core.LazySegmentedIterator.hasNext(LazySe
gmentedIterator.java:109)
        ... 56 more
Caused by: java.lang.ClassCastException: org.apache.xerces.parsers.XIncludeAware
ParserConfiguration cannot be cast to org.apache.xerces.xni.parser.XMLParserConf
iguration
        at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
        at org.apache.xerces.parsers.SAXParser.<init>(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.<init>(Unknown Sou
rce)
        at org.apache.xerces.jaxp.SAXParserImpl.<init>(Unknown Source)
        at org.apache.xerces.jaxp.SAXParserFactoryImpl.newSAXParser(Unknown Sour
ce)
        at com.microsoft.azure.storage.core.Utility.getSAXParser(Utility.java:54
6)
        at com.microsoft.azure.storage.blob.BlobListHandler.getBlobList(BlobList
Handler.java:72)
        at com.microsoft.azure.storage.blob.CloudBlobContainer$6.postProcessResp
onse(CloudBlobContainer.java:1253)
        at com.microsoft.azure.storage.blob.CloudBlobContainer$6.postProcessResp
onse(CloudBlobContainer.java:1217)
        at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(Exe
cutionEngine.java:148)
        ... 57 more
17/04/26 10:19:33 INFO SparkContext: Invoking stop() from shutdown hook
17/04/26 10:19:33 INFO SparkUI: Stopped Spark web UI at http://10.0.0.4:4040
17/04/26 10:19:33 INFO YarnClientSchedulerBackend: Interrupting monitor thread
17/04/26 10:19:33 INFO YarnClientSchedulerBackend: Shutting down all executors
17/04/26 10:19:33 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each exec
utor to shut down
17/04/26 10:19:33 INFO SchedulerExtensionServices: Stopping SchedulerExtensionSe
rvices
(serviceOption=None,
 services=List(),
 started=false)
17/04/26 10:19:33 INFO YarnClientSchedulerBackend: Stopped
17/04/26 10:19:33 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEnd
point stopped!
17/04/26 10:19:33 INFO MemoryStore: MemoryStore cleared
17/04/26 10:19:33 INFO BlockManager: BlockManager stopped
17/04/26 10:19:33 INFO BlockManagerMaster: BlockManagerMaster stopped
17/04/26 10:19:33 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
OutputCommitCoordinator stopped!
17/04/26 10:19:33 INFO SparkContext: Successfully stopped SparkContext
17/04/26 10:19:33 INFO ShutdownHookManager: Shutdown hook called
17/04/26 10:19:33 INFO ShutdownHookManager: Deleting directory C:\Users\labuser\
AppData\Local\Temp\2\spark-11c406ec-2c53-4042-b336-9d1164c3c6f9
17/04/26 10:19:33 INFO MetricsSystemImpl: Stopping azure-file-system metrics sys
tem...
17/04/26 10:19:33 INFO MetricsSystemImpl: azure-file-system metrics system stopp
ed.
17/04/26 10:19:33 INFO MetricsSystemImpl: azure-file-system metrics system shutd
own complete.

Please help me to resolve this issue. any help would be greatly appreciated.
Loading...