Is it possible to get a job end kind of notification on the executor (slave)

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Is it possible to get a job end kind of notification on the executor (slave)

Keith Chapman
Hi ,

Is it possible for an executor (or slave) to know when an actual job ends? I'm running spark on a cluster (with yarn) and my workers create some temporary files that I would like to clean up once the job ends. Is there a way for the worker to detect that a job has finished? I tried doing it in the JobProgressListener but it does not seem to work in a cluster. The event is not triggered in the worker.

Regards,
Keith.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Is it possible to get a job end kind of notification on the executor (slave)

Ryan Blue
I think this is usually done with `FileSystem.deleteOnExit`, which cleans up when an executor exits. That's not quite what you want (job finish) but is probably a reasonable fix.

rb

On Thu, Jan 19, 2017 at 9:31 PM, Keith Chapman <[hidden email]> wrote:
Hi ,

Is it possible for an executor (or slave) to know when an actual job ends? I'm running spark on a cluster (with yarn) and my workers create some temporary files that I would like to clean up once the job ends. Is there a way for the worker to detect that a job has finished? I tried doing it in the JobProgressListener but it does not seem to work in a cluster. The event is not triggered in the worker.

Regards,
Keith.



--
Ryan Blue
Software Engineer
Netflix
Loading...