Run an OS command or script supplied by the user at the start of each executor

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Run an OS command or script supplied by the user at the start of each executor

Luca Canali
Hi,

I have recently experimented with a few ways to run OS commands from the executors (in a YARN deployment) for a specific use case where we want to interact with an external system of interest for our environment. From that experience I thought that having the possibility to spawn a script at the start of each executors can be quite handy in a few cases and maybe more people are interested. For example I am think of case when interacting with external systems/APIs, or for injecting custom configurations via scripts distributed to the executors and/or for spawning custom monitoring tasks, etc.
They are probable all niche cases but the feature seems quite easy to implement.
I just wanted to check with the list if something like this has already come up in the past and/or there are thoughts about it or details that I have overlooked.
My simple proof of concept for implementing a "startup command" on the executors can be found at:
https://github.com/LucaCanali/spark/commit/e294a1f0d55af115f45fa6d2d7dcf81f751955fa
I can put all this in a Jira in case people here think it makes sense.

Thanks,
Luca


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Loading...