DISCUSS [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
I have been working on coming up with a proposal for supporting stage level resource configuration and scheduling. The basic idea is to allow the user to specify executor and task resource requirements for each stage to allow the user to control the resources required at a finer grain. One good example here is doing some ETL to preprocess your data in one stage and then feed that data into an ML algorithm (like tensorflow) that would run as a separate stage. The ETL could need totally different resource requirements for the executors/tasks than the ML stage does.
If you are interested please take a look at the SPIP and give me feedback. The text for the SPIP is in the jira description: