回复:Driver hung and happend out of memory while writing to console progress bar

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

回复:Driver hung and happend out of memory while writing to console progress bar

John Fang
the spark version is 2.1.0

------------------------------------------------------------------
发件人:方孝健(玄弟) <[hidden email]>
发送时间:2017年2月10日(星期五) 12:35
收件人:spark-dev <[hidden email]>; spark-user <[hidden email]>
主 题:Driver hung and happend out of memory while writing to console progress bar

[Stage 172:==============================>                 (10328 + 93) / 16144][Stage 172:==============================>                 (10329 + 93) / 16144][Stage 172:==============================>                 (10330 + 93) / 16144][Stage 172:==============================>                 (10331 + 93) / 16144][Stage 172:==============================>                 (10333 + 92) / 16144][Stage 172:==============================>                 (10333 + 93) / 16144][Stage 172:==============================>                 (10333 + 94) / 16144][Stage 172:==============================>                 (10334 + 94) / 16144][Stage 172:==============================>                 (10338 + 93) / 16144][Stage 172:==============================>                 (10339 + 92) / 16144][Stage 172:==============================>                 (10340 + 93) / 16144][Stage 172:==============================>                 (10341 + 92) / 16144][Stage 172:==============================>                 (10341 + 93) / 16144][Stage 172:==============================>                 (10342 + 93) / 16144][Stage 172:==============================>                 (10343 + 93) / 16144][Stage 172:==============================>                 (10344 + 92) / 16144][Stage 172:==============================>                 (10345 + 92) / 16144][Stage 172:==============================>                 (10345 + 93) / 16144][Stage 172:==============================>                 (10346 + 93) / 16144][Stage 172:==============================>                 (10348 + 92) / 16144][Stage 172:==============================>                 (10348 + 93) / 16144][Stage 172:==============================>                 (10349 + 92) / 16144][Stage 172:==============================>                 (10349 + 93) / 16144][Stage 172:==============================>                 (10350 + 92) / 16144][Stage 172:==============================>                 (10352 + 92) / 16144][Stage 172:==============================>                 (10353 + 92) / 16144][Stage 172:==============================>                 (10354 + 92) / 16144][Stage 172:==============================>                 (10355 + 92) / 16144][Stage 172:==============================>                 (10356 + 92) / 16144][Stage 172:==============================>                 (10356 + 93) / 16144][Stage 172:==============================>                 (10357 + 92) / 16144][Stage 172:==============================>                 (10357 + 93) / 16144][Stage 172:==============================>                 (10358 + 92) / 16144][Stage 172:==============================>                 (10358 + 93) / 16144][Stage 172:==============================>                 (10359 + 92) / 16144][Stage 172:==============================>                 (10359 + 93) / 16144][Stage 172:==============================>                 (10359 + 94) / 16144][Stage 172:==============================>                 (10361 + 92) / 16144][Stage 172:==============================>                 (10361 + 93) / 16144][Stage 172:==============================>                 (10362 + 92) / 16144][Stage 172:==============================>                 (10362 + 93) / 16144][Stage 172:==============================>                 (10363 + 93) / 16144][Stage 172:==============================>                 (10364 + 92) / 16144][Stage 172:==============================>                 (10365 + 92) / 16144][Stage 172:==============================>                 (10365 + 93) / 16144][Stage 172:==============================>                 (10366 + 92) / 16144][Stage 172:==============================>                 (10366 + 93) / 16144][Stage 172:==============================>                 (10367 + 92) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space	at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156)	at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124)	at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189)	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space	at java.util.Arrays.copyOfRange(Arrays.java:3664)	at java.lang.String.<init>(String.java:207)	at java.lang.StringBuilder.toString(StringBuilder.java:407)	at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430)	at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101)	at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71)	at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55)	at java.util.TimerThread.mainLoop(Timer.java:555)	at java.util.TimerThread.run(Timer.java:505)
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Driver hung and happend out of memory while writing to console progress bar

Ryan Blue
This isn't related to the progress bar, it just happened while in that section of code. Something else is taking memory in the driver, usually a broadcast table or something else that requires a lot of memory and happens on the driver.

You should check your driver memory settings and the query plan (if this was SparkSQL) for this stage to investigate further.

rb

On Thu, Feb 9, 2017 at 8:41 PM, John Fang <[hidden email]> wrote:
the spark version is 2.1.0

------------------------------------------------------------------
发件人:方孝健(玄弟) <[hidden email]>
发送时间:2017年2月10日(星期五) 12:35
收件人:spark-dev <[hidden email]>; spark-user <[hidden email]>
主 题:Driver hung and happend out of memory while writing to console progress bar

[Stage 172:==============================>                 (10328 + 93) / 16144][Stage 172:==============================>                 (10329 + 93) / 16144][Stage 172:==============================>                 (10330 + 93) / 16144][Stage 172:==============================>                 (10331 + 93) / 16144][Stage 172:==============================>                 (10333 + 92) / 16144][Stage 172:==============================>                 (10333 + 93) / 16144][Stage 172:==============================>                 (10333 + 94) / 16144][Stage 172:==============================>                 (10334 + 94) / 16144][Stage 172:==============================>                 (10338 + 93) / 16144][Stage 172:==============================>                 (10339 + 92) / 16144][Stage 172:==============================>                 (10340 + 93) / 16144][Stage 172:==============================>                 (10341 + 92) / 16144][Stage 172:==============================>                 (10341 + 93) / 16144][Stage 172:==============================>                 (10342 + 93) / 16144][Stage 172:==============================>                 (10343 + 93) / 16144][Stage 172:==============================>                 (10344 + 92) / 16144][Stage 172:==============================>                 (10345 + 92) / 16144][Stage 172:==============================>                 (10345 + 93) / 16144][Stage 172:==============================>                 (10346 + 93) / 16144][Stage 172:==============================>                 (10348 + 92) / 16144][Stage 172:==============================>                 (10348 + 93) / 16144][Stage 172:==============================>                 (10349 + 92) / 16144][Stage 172:==============================>                 (10349 + 93) / 16144][Stage 172:==============================>                 (10350 + 92) / 16144][Stage 172:==============================>                 (10352 + 92) / 16144][Stage 172:==============================>                 (10353 + 92) / 16144][Stage 172:==============================>                 (10354 + 92) / 16144][Stage 172:==============================>                 (10355 + 92) / 16144][Stage 172:==============================>                 (10356 + 92) / 16144][Stage 172:==============================>                 (10356 + 93) / 16144][Stage 172:==============================>                 (10357 + 92) / 16144][Stage 172:==============================>                 (10357 + 93) / 16144][Stage 172:==============================>                 (10358 + 92) / 16144][Stage 172:==============================>                 (10358 + 93) / 16144][Stage 172:==============================>                 (10359 + 92) / 16144][Stage 172:==============================>                 (10359 + 93) / 16144][Stage 172:==============================>                 (10359 + 94) / 16144][Stage 172:==============================>                 (10361 + 92) / 16144][Stage 172:==============================>                 (10361 + 93) / 16144][Stage 172:==============================>                 (10362 + 92) / 16144][Stage 172:==============================>                 (10362 + 93) / 16144][Stage 172:==============================>                 (10363 + 93) / 16144][Stage 172:==============================>                 (10364 + 92) / 16144][Stage 172:==============================>                 (10365 + 92) / 16144][Stage 172:==============================>                 (10365 + 93) / 16144][Stage 172:==============================>                 (10366 + 92) / 16144][Stage 172:==============================>                 (10366 + 93) / 16144][Stage 172:==============================>                 (10367 + 92) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144][Stage 172:==============================>                 (10367 + 93) / 16144]Exception in thread "JobGenerator" java.lang.OutOfMemoryError: Java heap space	at com.fasterxml.jackson.core.util.BufferRecycler.calloc(BufferRecycler.java:156)	at com.fasterxml.jackson.core.util.BufferRecycler.allocCharBuffer(BufferRecycler.java:124)	at com.fasterxml.jackson.core.io.IOContext.allocTokenBuffer(IOContext.java:189)	at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:879)Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "JobGenerator"Exception in thread "refresh progress" java.lang.OutOfMemoryError: Java heap space	at java.util.Arrays.copyOfRange(Arrays.java:3664)	at java.lang.String.<init>(String.java:207)	at java.lang.StringBuilder.toString(StringBuilder.java:407)	at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430)	at org.apache.spark.ui.ConsoleProgressBar.show(ConsoleProgressBar.scala:101)	at org.apache.spark.ui.ConsoleProgressBar.org$apache$spark$ui$ConsoleProgressBar$$refresh(ConsoleProgressBar.scala:71)	at org.apache.spark.ui.ConsoleProgressBar$$anon$1.run(ConsoleProgressBar.scala:55)	at java.util.TimerThread.mainLoop(Timer.java:555)	at java.util.TimerThread.run(Timer.java:505)



--
Ryan Blue
Software Engineer
Netflix
Loading...