Quantcast

Need Help: getting java.lang.OutOfMemory Error : GC overhead limit exceeded (TransportChannelHandler)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Need Help: getting java.lang.OutOfMemory Error : GC overhead limit exceeded (TransportChannelHandler)

naresh gundla
Hi ,

I am running a spark application and getting out of memory errors in yarn nodemanager logs and container get killed. Please find below for the errors details.
Has anyone faced with this issue?

Enabled spark dynamic allocation and yarn shuffle

 2017-02-15 14:50:48,047 WARN io.netty.util.concurrent.DefaultPromise: An exception was thrown by org.apache.spark.network.server.TransportRequestHandler$2.operationComplete()
java.lang.OutOfMemoryError: GC overhead limit exceeded
2017-02-15 15:21:09,506 ERROR org.apache.spark.network.server.TransportRequestHandler: Error opening block StreamChunkId{streamId=1374579274227, chunkIndex=241} for request from /10.154.16.83:50042
java.lang.IllegalStateException: Received out-of-order chunk index 241 (expected 114)
        at org.apache.spark.network.server.OneForOneStreamManager.getChunk(OneForOneStreamManager.java:81)
        at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:121)
        at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:100)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:104)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at java.lang.Thread.run(Thread.java:745)

2017-02-15 14:50:14,692 WARN org.apache.spark.network.server.TransportChannelHandler: Exception in connection from /10.154.16.74:58547
java.lang.OutOfMemoryError: GC overhead limit exceeded

Thanks
Naresh

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Need Help: getting java.lang.OutOfMemory Error : GC overhead limit exceeded (TransportChannelHandler)

Ryan Blue
Naresh,

We've configured our Spark JVMs to shut down if there is an OutOfMemoryError. Otherwise, the error will bring down a random thread an cause trouble like the IllegalStateException you hit. It is best to let Spark recover by replacing the executor or failing the job.

rb

On Wed, Feb 15, 2017 at 1:58 PM, naresh gundla <[hidden email]> wrote:
Hi ,

I am running a spark application and getting out of memory errors in yarn nodemanager logs and container get killed. Please find below for the errors details.
Has anyone faced with this issue?

Enabled spark dynamic allocation and yarn shuffle

 2017-02-15 14:50:48,047 WARN io.netty.util.concurrent.DefaultPromise: An exception was thrown by org.apache.spark.network.server.TransportRequestHandler$2.operationComplete()
java.lang.OutOfMemoryError: GC overhead limit exceeded
2017-02-15 15:21:09,506 ERROR org.apache.spark.network.server.TransportRequestHandler: Error opening block StreamChunkId{streamId=1374579274227, chunkIndex=241} for request from /10.154.16.83:50042
java.lang.IllegalStateException: Received out-of-order chunk index 241 (expected 114)
        at org.apache.spark.network.server.OneForOneStreamManager.getChunk(OneForOneStreamManager.java:81)
        at org.apache.spark.network.server.TransportRequestHandler.processFetchRequest(TransportRequestHandler.java:121)
        at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:100)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:104)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at java.lang.Thread.run(Thread.java:745)

2017-02-15 14:50:14,692 WARN org.apache.spark.network.server.TransportChannelHandler: Exception in connection from /10.154.16.74:58547
java.lang.OutOfMemoryError: GC overhead limit exceeded

Thanks
Naresh




--
Ryan Blue
Software Engineer
Netflix
Loading...