[VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Thomas graves-2
Hey everyone,

I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
resource configuration and scheduling

This is for supporting stage level resource configuration and
scheduling.  The basic idea is to allow the user to specify executor
and task resource requirements for each stage to allow the user to
control the resources required at a finer grain. One good example here
is doing some ETL to preprocess your data in one stage and then feed
that data into an ML algorithm (like tensorflow) that would run as a
separate stage.  The ETL could need totally different resource
requirements for the executors/tasks than the ML stage does.

The text for the SPIP is in the jira description:

https://issues.apache.org/jira/browse/SPARK-27495

I split the API and Design parts into a google doc that is linked to
from the jira.

This vote is open until next Fri (Sept 13th).

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Thanks,
Tom

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Bobby Evans
I am +1.

I like the proposal and it feels like it will provide what users need for the use cases in the SPIP. Having done similar things on other projects in the past I know this is just the beginning to get something that is truly user and admin friendly, but this is a great start.

Thanks,

Bobby

On Wed, Sep 4, 2019 at 9:24 AM Thomas graves <[hidden email]> wrote:
Hey everyone,

I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
resource configuration and scheduling

This is for supporting stage level resource configuration and
scheduling.  The basic idea is to allow the user to specify executor
and task resource requirements for each stage to allow the user to
control the resources required at a finer grain. One good example here
is doing some ETL to preprocess your data in one stage and then feed
that data into an ML algorithm (like tensorflow) that would run as a
separate stage.  The ETL could need totally different resource
requirements for the executors/tasks than the ML stage does.

The text for the SPIP is in the jira description:

https://issues.apache.org/jira/browse/SPARK-27495

I split the API and Design parts into a google doc that is linked to
from the jira.

This vote is open until next Fri (Sept 13th).

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Thanks,
Tom

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Felix Cheung
In reply to this post by Thomas graves-2
+1


From: Thomas graves <[hidden email]>
Sent: Wednesday, September 4, 2019 7:24:26 AM
To: dev <[hidden email]>
Subject: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
 
Hey everyone,

I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
resource configuration and scheduling

This is for supporting stage level resource configuration and
scheduling.  The basic idea is to allow the user to specify executor
and task resource requirements for each stage to allow the user to
control the resources required at a finer grain. One good example here
is doing some ETL to preprocess your data in one stage and then feed
that data into an ML algorithm (like tensorflow) that would run as a
separate stage.  The ETL could need totally different resource
requirements for the executors/tasks than the ML stage does.

The text for the SPIP is in the jira description:

https://issues.apache.org/jira/browse/SPARK-27495

I split the API and Design parts into a google doc that is linked to
from the jira.

This vote is open until next Fri (Sept 13th).

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Thanks,
Tom

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Ryan Blue
+1

This is going to be really useful. Thanks for working on it!

On Wed, Sep 11, 2019 at 9:38 AM Felix Cheung <[hidden email]> wrote:
+1


From: Thomas graves <[hidden email]>
Sent: Wednesday, September 4, 2019 7:24:26 AM
To: dev <[hidden email]>
Subject: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
 
Hey everyone,

I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
resource configuration and scheduling

This is for supporting stage level resource configuration and
scheduling.  The basic idea is to allow the user to specify executor
and task resource requirements for each stage to allow the user to
control the resources required at a finer grain. One good example here
is doing some ETL to preprocess your data in one stage and then feed
that data into an ML algorithm (like tensorflow) that would run as a
separate stage.  The ETL could need totally different resource
requirements for the executors/tasks than the ML stage does.

The text for the SPIP is in the jira description:

https://issues.apache.org/jira/browse/SPARK-27495

I split the API and Design parts into a google doc that is linked to
from the jira.

This vote is open until next Fri (Sept 13th).

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Thanks,
Tom

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Bryan Cutler
+1 (non-binding), looks good!

On Wed, Sep 11, 2019 at 10:05 AM Ryan Blue <[hidden email]> wrote:
+1

This is going to be really useful. Thanks for working on it!

On Wed, Sep 11, 2019 at 9:38 AM Felix Cheung <[hidden email]> wrote:
+1


From: Thomas graves <[hidden email]>
Sent: Wednesday, September 4, 2019 7:24:26 AM
To: dev <[hidden email]>
Subject: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
 
Hey everyone,

I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
resource configuration and scheduling

This is for supporting stage level resource configuration and
scheduling.  The basic idea is to allow the user to specify executor
and task resource requirements for each stage to allow the user to
control the resources required at a finer grain. One good example here
is doing some ETL to preprocess your data in one stage and then feed
that data into an ML algorithm (like tensorflow) that would run as a
separate stage.  The ETL could need totally different resource
requirements for the executors/tasks than the ML stage does.

The text for the SPIP is in the jira description:

https://issues.apache.org/jira/browse/SPARK-27495

I split the API and Design parts into a google doc that is linked to
from the jira.

This vote is open until next Fri (Sept 13th).

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Thanks,
Tom

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Thomas Graves
Thanks everyone so far for the voting and the feedback, bumping this
up as vote is scheduling to end today.

Tom

On Wed, Sep 11, 2019 at 1:10 PM Bryan Cutler <[hidden email]> wrote:

>
> +1 (non-binding), looks good!
>
> On Wed, Sep 11, 2019 at 10:05 AM Ryan Blue <[hidden email]> wrote:
>>
>> +1
>>
>> This is going to be really useful. Thanks for working on it!
>>
>> On Wed, Sep 11, 2019 at 9:38 AM Felix Cheung <[hidden email]> wrote:
>>>
>>> +1
>>>
>>> ________________________________
>>> From: Thomas graves <[hidden email]>
>>> Sent: Wednesday, September 4, 2019 7:24:26 AM
>>> To: dev <[hidden email]>
>>> Subject: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
>>>
>>> Hey everyone,
>>>
>>> I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
>>> resource configuration and scheduling
>>>
>>> This is for supporting stage level resource configuration and
>>> scheduling.  The basic idea is to allow the user to specify executor
>>> and task resource requirements for each stage to allow the user to
>>> control the resources required at a finer grain. One good example here
>>> is doing some ETL to preprocess your data in one stage and then feed
>>> that data into an ML algorithm (like tensorflow) that would run as a
>>> separate stage.  The ETL could need totally different resource
>>> requirements for the executors/tasks than the ML stage does.
>>>
>>> The text for the SPIP is in the jira description:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-27495
>>>
>>> I split the API and Design parts into a google doc that is linked to
>>> from the jira.
>>>
>>> This vote is open until next Fri (Sept 13th).
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don't think this is a good idea because ...
>>>
>>> I'll start with my +1
>>>
>>> Thanks,
>>> Tom
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling

Dongjoon Hyun-2
+1

Thanks,
Dongjoon.

On Fri, Sep 13, 2019 at 9:59 AM Thomas Graves <[hidden email]> wrote:
Thanks everyone so far for the voting and the feedback, bumping this
up as vote is scheduling to end today.

Tom

On Wed, Sep 11, 2019 at 1:10 PM Bryan Cutler <[hidden email]> wrote:
>
> +1 (non-binding), looks good!
>
> On Wed, Sep 11, 2019 at 10:05 AM Ryan Blue <[hidden email]> wrote:
>>
>> +1
>>
>> This is going to be really useful. Thanks for working on it!
>>
>> On Wed, Sep 11, 2019 at 9:38 AM Felix Cheung <[hidden email]> wrote:
>>>
>>> +1
>>>
>>> ________________________________
>>> From: Thomas graves <[hidden email]>
>>> Sent: Wednesday, September 4, 2019 7:24:26 AM
>>> To: dev <[hidden email]>
>>> Subject: [VOTE] [SPARK-27495] SPIP: Support Stage level resource configuration and scheduling
>>>
>>> Hey everyone,
>>>
>>> I'd like to call for a vote on SPARK-27495 SPIP: Support Stage level
>>> resource configuration and scheduling
>>>
>>> This is for supporting stage level resource configuration and
>>> scheduling.  The basic idea is to allow the user to specify executor
>>> and task resource requirements for each stage to allow the user to
>>> control the resources required at a finer grain. One good example here
>>> is doing some ETL to preprocess your data in one stage and then feed
>>> that data into an ML algorithm (like tensorflow) that would run as a
>>> separate stage.  The ETL could need totally different resource
>>> requirements for the executors/tasks than the ML stage does.
>>>
>>> The text for the SPIP is in the jira description:
>>>
>>> https://issues.apache.org/jira/browse/SPARK-27495
>>>
>>> I split the API and Design parts into a google doc that is linked to
>>> from the jira.
>>>
>>> This vote is open until next Fri (Sept 13th).
>>>
>>> [ ] +1: Accept the proposal as an official SPIP
>>> [ ] +0
>>> [ ] -1: I don't think this is a good idea because ...
>>>
>>> I'll start with my +1
>>>
>>> Thanks,
>>> Tom
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>>
>>
>> --
>> Ryan Blue
>> Software Engineer
>> Netflix

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]