Quantcast

Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

StanZhai
Hi all,

We just upgraded our Spark from 1.6.2 to 2.1.0.

Our Spark application is started by spark-submit with config of `--executor-memory 35G` in standalone model, but the actual use of memory up to 65G after a full gc(jmap -histo:live $pid) as follow:

test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
test      181941  181 34.7 94665384 68836752 ?   Sl   09:25 711:21 /home/test/service/jdk/bin/java -cp /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/ -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@xxx.xxx.xxx.xxx:47781 --executor-id 1 --hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url spark://Worker@test-192:33890

Our Spark jobs are all sql.

The exceed memory looks like off-heap memory, but the default value of `spark.memory.offHeap.enabled` is `false`.

We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?

Any help is greatly appreicated!

Best,
Stan
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

rxin
Are you using G1 GC? G1 sometimes uses a lot more memory than the size allocated. 


On Sun, Jan 22, 2017 at 12:58 AM StanZhai <[hidden email]> wrote:
Hi all,



We just upgraded our Spark from 1.6.2 to 2.1.0.



Our Spark application is started by spark-submit with config of

`--executor-memory 35G` in standalone model, but the actual use of memory up

to 65G after a full gc(jmap -histo:live $pid) as follow:



test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend

test      181941  181 34.7 94665384 68836752 ?   Sl   09:25 711:21

/home/test/service/jdk/bin/java -cp

/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/

-Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails

-XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc

org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url

spark://[hidden email]:47781 --executor-id 1

--hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url

spark://Worker@test-192:33890



Our Spark jobs are all sql.



The exceed memory looks like off-heap memory, but the default value of

`spark.memory.offHeap.enabled` is `false`.



We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?



Any help is greatly appreicated!



Best,

Stan







--

View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html

Sent from the Apache Spark Developers List mailing list archive at Nabble.com.



---------------------------------------------------------------------

To unsubscribe e-mail: [hidden email]



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

Koert Kuipers
could this be related to SPARK-18787?

On Sun, Jan 22, 2017 at 1:45 PM, Reynold Xin <[hidden email]> wrote:
Are you using G1 GC? G1 sometimes uses a lot more memory than the size allocated. 


On Sun, Jan 22, 2017 at 12:58 AM StanZhai <[hidden email]> wrote:
Hi all,



We just upgraded our Spark from 1.6.2 to 2.1.0.



Our Spark application is started by spark-submit with config of

`--executor-memory 35G` in standalone model, but the actual use of memory up

to 65G after a full gc(jmap -histo:live $pid) as follow:



test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend

test      181941  181 34.7 94665384 68836752 ?   Sl   09:25 711:21

/home/test/service/jdk/bin/java -cp

/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/

-Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails

-XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc

org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url

spark://CoarseGrainedScheduler@xxx.xxx.xxx.xxx:47781 --executor-id 1

--hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url

spark://Worker@test-192:33890



Our Spark jobs are all sql.



The exceed memory looks like off-heap memory, but the default value of

`spark.memory.offHeap.enabled` is `false`.



We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?



Any help is greatly appreicated!



Best,

Stan







--

View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html

Sent from the Apache Spark Developers List mailing list archive at Nabble.com.



---------------------------------------------------------------------

To unsubscribe e-mail: [hidden email]




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

StanZhai
In reply to this post by rxin
I'm using Parallel GC.
rxin wrote
Are you using G1 GC? G1 sometimes uses a lot more memory than the size
allocated.


On Sun, Jan 22, 2017 at 12:58 AM StanZhai <[hidden email]> wrote:

> Hi all,
>
>
>
> We just upgraded our Spark from 1.6.2 to 2.1.0.
>
>
>
> Our Spark application is started by spark-submit with config of
>
> `--executor-memory 35G` in standalone model, but the actual use of memory
> up
>
> to 65G after a full gc(jmap -histo:live $pid) as follow:
>
>
>
> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>
> test      181941  181 34.7 94665384 68836752 ?   Sl   09:25 711:21
>
> /home/test/service/jdk/bin/java -cp
>
>
> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
>
> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>
> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>
> spark://[hidden email]:47781 --executor-id 1
>
> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
> --worker-url
>
> spark://Worker@test-192:33890
>
>
>
> Our Spark jobs are all sql.
>
>
>
> The exceed memory looks like off-heap memory, but the default value of
>
> `spark.memory.offHeap.enabled` is `false`.
>
>
>
> We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?
>
>
>
> Any help is greatly appreicated!
>
>
>
> Best,
>
> Stan
>
>
>
>
>
>
>
> --
>
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe e-mail: [hidden email]
>
>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

Michael Allman-2
Hi Stan,

What OS/version are you using?

Michael

On Jan 22, 2017, at 11:36 PM, StanZhai <[hidden email]> wrote:

I'm using Parallel GC.
rxin wrote
Are you using G1 GC? G1 sometimes uses a lot more memory than the size
allocated.


On Sun, Jan 22, 2017 at 12:58 AM StanZhai &lt;

mail@

&gt; wrote:

Hi all,



We just upgraded our Spark from 1.6.2 to 2.1.0.



Our Spark application is started by spark-submit with config of

`--executor-memory 35G` in standalone model, but the actual use of memory
up

to 65G after a full gc(jmap -histo:live $pid) as follow:



test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend

test      181941  181 34.7 94665384 68836752 ?   Sl   09:25 711:21

/home/test/service/jdk/bin/java -cp


/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/

-Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails

-XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc

org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url

spark://

CoarseGrainedScheduler@.xxx

:47781 --executor-id 1

--hostname test-192 --cores 36 --app-id app-20170122092509-0017
--worker-url

<a href="spark://Worker@test-192:33890" class="">spark://Worker@test-192:33890



Our Spark jobs are all sql.



The exceed memory looks like off-heap memory, but the default value of

`spark.memory.offHeap.enabled` is `false`.



We didn't find the problem in Spark 1.6.x, what causes this in Spark
2.1.0?



Any help is greatly appreicated!



Best,

Stan







--

View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html

Sent from the Apache Spark Developers List mailing list archive at
Nabble.com.



---------------------------------------------------------------------

To unsubscribe e-mail: 

dev-unsubscribe@.apache










--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

StanZhai
CentOS 7.1,
Linux version 3.10.0-229.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) #1 SMP Fri Mar 6 11:36:42 UTC 2015

Michael Allman-2 wrote
Hi Stan,

What OS/version are you using?

Michael

> On Jan 22, 2017, at 11:36 PM, StanZhai <[hidden email]> wrote:
>
> I'm using Parallel GC.
> rxin wrote
>> Are you using G1 GC? G1 sometimes uses a lot more memory than the size
>> allocated.
>>
>>
>> On Sun, Jan 22, 2017 at 12:58 AM StanZhai <
>
>> mail@
>
>> > wrote:
>>
>>> Hi all,
>>>
>>>
>>>
>>> We just upgraded our Spark from 1.6.2 to 2.1.0.
>>>
>>>
>>>
>>> Our Spark application is started by spark-submit with config of
>>>
>>> `--executor-memory 35G` in standalone model, but the actual use of memory
>>> up
>>>
>>> to 65G after a full gc(jmap -histo:live $pid) as follow:
>>>
>>>
>>>
>>> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
>>>
>>> test      181941  181 34.7 94665384 68836752 ?   Sl   09:25 711:21
>>>
>>> /home/test/service/jdk/bin/java -cp
>>>
>>>
>>> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
>>>
>>> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
>>>
>>> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
>>>
>>> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
>>>
>>> spark://
>
>> CoarseGrainedScheduler@.xxx
>
>> :47781 --executor-id 1
>>>
>>> --hostname test-192 --cores 36 --app-id app-20170122092509-0017
>>> --worker-url
>>>
>>> spark://Worker@test-192:33890
>>>
>>>
>>>
>>> Our Spark jobs are all sql.
>>>
>>>
>>>
>>> The exceed memory looks like off-heap memory, but the default value of
>>>
>>> `spark.memory.offHeap.enabled` is `false`.
>>>
>>>
>>>
>>> We didn't find the problem in Spark 1.6.x, what causes this in Spark
>>> 2.1.0?
>>>
>>>
>>>
>>> Any help is greatly appreicated!
>>>
>>>
>>>
>>> Best,
>>>
>>> Stan
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> View this message in context:
>>> http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
>>>
>>> Sent from the Apache Spark Developers List mailing list archive at
>>> Nabble.com <http://nabble.com/>.
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>> To unsubscribe e-mail:
>
>> dev-unsubscribe@.apache
>
>>>
>>>
>>>
>>>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html <http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697p20707.html>
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com <http://nabble.com/>.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email] <mailto:[hidden email]>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

Jacek Laskowski
In reply to this post by StanZhai
Hi,

Just to throw few zlotys to the conversation, I believe that Spark
Standalone does not enforce any memory checks to limit or even kill
executors beyond requested memory (like YARN). I also found that
memory does not have much of use while scheduling tasks and CPU
matters only.

My understanding of `spark.memory.offHeap.enabled` is `false` is that
it does not disable off heap memory used in Java NIO for buffers in
shuffling, RPC, etc. so the memory is always (?) more than you request
for mx using executor-memory.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sun, Jan 22, 2017 at 9:57 AM, StanZhai <[hidden email]> wrote:

> Hi all,
>
> We just upgraded our Spark from 1.6.2 to 2.1.0.
>
> Our Spark application is started by spark-submit with config of
> `--executor-memory 35G` in standalone model, but the actual use of memory up
> to 65G after a full gc(jmap -histo:live $pid) as follow:
>
> test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend
> test      181941 181 34.7 94665384 68836752 ?   Sl   09:25 711:21
> /home/test/service/jdk/bin/java -cp
> /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/
> -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc
> org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
> spark://[hidden email]:47781 --executor-id 1
> --hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url
> spark://Worker@test-192:33890
>
> Our Spark jobs are all sql.
>
> The exceed memory looks like off-heap memory, but the default value of
> `spark.memory.offHeap.enabled` is `false`.
>
> We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0?
>
> Any help is greatly appreicated!
>
> Best,
> Stan
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

StanZhai
In reply to this post by StanZhai
From thread dump page of Executor of WebUI, I found that there are about 1300 threads named  "DataStreamer for file /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_000069_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet" in TIMED_WAITING state like this:


The exceed off-heap memory may be caused by these abnormal threads.

This problem occurs only when writing data to the Hadoop(tasks may be killed by Executor during writing).

Could this be related to https://issues.apache.org/jira/browse/HDFS-9812?

It's may be a bug of Spark when killing tasks during writing data. What's the difference between Spark 1.6.x and 2.1.0 in killing tasks?

This is a critical issue, I've worked on this for days.

Any help?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

StanZhai
I've filed a JIRA about this problem. https://issues.apache.org/jira/browse/SPARK-19532

I've tried to set `spark.speculation` to `false`, but the off-heap also exceed about 10G after triggering a FullGC to the Executor process(--executor-memory 30G), as follow:

test@test Online ~ $ ps aux | grep CoarseGrainedExecutorBackend
test      105371  106 21.5 67325492 42621992 ?   Sl   15:20  55:14 /home/test/service/jdk/bin/java -cp /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/ -Xmx30720M -Dspark.driver.port=9835 -Dtag=spark_2_1_test -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@172.16.34.235:9835 --executor-id 4 --hostname test-192 --cores 36 --app-id app-20170213152037-0043 --worker-url spark://Worker@test-192:33890

So, I think these are also other reasons for this problem.

We have been trying to upgrade our Spark from the releasing of Spark 2.1.0.

This version is unstable and not available for us because of the memory problems, we should pay attention to this.

StanZhai wrote
From thread dump page of Executor of WebUI, I found that there are about 1300 threads named  "DataStreamer for file /test/data/test_temp/_temporary/0/_temporary/attempt_20170207172435_80750_m_000069_1/part-00069-690407af-0900-46b1-9590-a6d6c696fe68.snappy.parquet" in TIMED_WAITING state like this:


The exceed off-heap memory may be caused by these abnormal threads.

This problem occurs only when writing data to the Hadoop(tasks may be killed by Executor during writing).

Could this be related to https://issues.apache.org/jira/browse/HDFS-9812?

It's may be a bug of Spark when killing tasks during writing data. What's the difference between Spark 1.6.x and 2.1.0 in killing tasks?

This is a critical issue, I've worked on this for days.

Any help?
Vel
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Executors exceed maximum memory defined with `--executor-memory` in Spark 2.1.0

Vel
This post has NOT been accepted by the mailing list yet.
Hi guys, I am facing the same problem.
My Spark application gets slower by time and at the end it hangs.
I am running the application in AWS-EMR , 16 nodes ,Each node equipped with 236 Gb RAM with 64 cores.
The application is full of sql dataframe joins .

The exception i am getting is

1)
All datanodes DatanodeInfoWithStorage[10.253.194.172:50010,DS-e975f145-0e8c-418a-8597-5787112958d4,DISK] are bad. Aborting...
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1109)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:871)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:401)
17/04/05 12:03:10 ERROR LiveListenerBus: Listener EventLoggingListener threw an exception



2)
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval
        at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:48)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
        at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)
        at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$reportHeartBeat(Executor.scala:518)



I tried running with changing configurations but it dint help either.
        --conf spark.default.parallelism=240
        --num-executors 20 --executor-cores 4 --driver-cores 4 --executor-memory 45g --driver-memory 200g
        --conf spark.memory.useLegacyMode=true
        --conf spark.shuffle.memoryFraction=0.4
        --conf spark.storage.memoryFraction=0.4  
        --conf spark.executor.heartbeatInterval=30
        --conf spark.network.timeout=800s
        --conf spark.eventLog.enabled=false
        --conf spark.driver.extraJavaOptions="-XX:MaxPermSize=8192m -XX:PermSize=256m "
        --conf spark.dynamicAllocation.enabled=false  
        --conf spark.sql.broadcastTimeout=1200
        --conf spark.yarn.executor.memoryOverhead=4096
        --conf spark.yarn.driver.memoryOverhead=8192

I have been working on this for 3 days yet could not able to get this through.
Any help provided will be greatly useful.
Thanks in advance.

Loading...