Futures timeout exception in executor logs

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Futures timeout exception in executor logs

Simon Scott

Hi,

 

Just wondering if anybody has any insights on this SPARK-14140: Futures timeout exception in executor logs ?

 

We are seeing the exact same exception during a long-running iterative application on a Spark Standalone cluster, v 2.1.

At the same time as the exception appears on an executor, the driver dies without any logged message.

 

Any advice much appreciated

 

Thanks

Simon Scott

Viavi Solutions

Reply | Threaded
Open this post in threaded view
|

Re: Futures timeout exception in executor logs

Xuefu Zhang-2
I saw this quite often in our clusters. we have increased spark.executor.heartbeatInterval to 60s from the default value, which should help. The problem seems due to poor Spark driver performance and/or locking issues when it cannot process incoming events quickly enough.

Thanks,
Xuefu

On Thu, Sep 14, 2017 at 1:00 AM, Simon Scott <[hidden email]> wrote:

Hi,

 

Just wondering if anybody has any insights on this SPARK-14140: Futures timeout exception in executor logs ?

 

We are seeing the exact same exception during a long-running iterative application on a Spark Standalone cluster, v 2.1.

At the same time as the exception appears on an executor, the driver dies without any logged message.

 

Any advice much appreciated

 

Thanks

Simon Scott

Viavi Solutions