Logging exception of custom spark app

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Logging exception of custom spark app

Alessandro Liparoti
Hi everyone,

I have an application that uses spark to perform some computation. It can be used both in a spark-shell or in a spark-submit. I want to log all exceptions throw by my code inside a file in order to have some detailed info when user have an error.

I tried with this

Thread.currentThread().setUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
def uncaughtException(t: Thread, e: Throwable): Unit = {
  logger.error("exception logged")
  println("exception logged")
but it is not working. I saw that Spark already sets an uncaughtExceptionHandler, so probably this code is not effective. 
The other option would be to try-catch all public methods of my API, log an exception when it happens and then throw it. But I think this is not optimal.

Do you have any suggestion?
Alessandro Liparoti