Apache Spark Serverless pool and preemptive execution support

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache Spark Serverless pool and preemptive execution support

Mania Abdi
Hi everyone,

I came across Databricks serverless analytics on Spark and Serverless pools. I was wondering if the Apache Spark also supports these pools and if the Apache Spark resource manager supports preemptive execution and auto-scaling?

Regards
Mania
Reply | Threaded
Open this post in threaded view
|

Re: Apache Spark Serverless pool and preemptive execution support

Sean Owen-2
The built-in resource managers do support dynamic allocation for auto-scaling. https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation

Provisioning the resources for the resource manager is a different question that Spark itself doesn't address; it won't go run EC2/Azure instances for you, but gives you the resource manager to run on those. The 'serverless' part you refer to here is really answering the question of provisioning VMs for a particular hosted Spark service. That isn't part of Spark per se.


On Sat, Mar 28, 2020 at 10:38 AM Mania Abdi <[hidden email]> wrote:
Hi everyone,

I came across Databricks serverless analytics on Spark and Serverless pools. I was wondering if the Apache Spark also supports these pools and if the Apache Spark resource manager supports preemptive execution and auto-scaling?

Regards
Mania