(no subject)

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
Report Content as Inappropriate

(no subject)

Keith Chapman
Hi ,

Is it possible for an executor (or slave) to know when an actual job ends? I'm running spark on a cluster (with yarn) and my workers create some temporary files that I would like to clean up once the job ends. Is there a way for the worker to detect that a job has finished? I tried doing it in the JobProgressListener but it does not seem to work in a cluster. The event is not triggered in the worker.