Is there any plan to develop an application level fair scheduler?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Is there any plan to develop an application level fair scheduler?

Nan Zhu
Hi, All  

Is there any plan to develop an application level fair scheduler?

I think it will have more value than a fair scheduler within the application (actually I didn’t understand why we want to fairly share the resource among jobs within the application, in usual, users submit different applications, not jobs)…

Best,  

--  
Nan Zhu

Reply | Threaded
Open this post in threaded view
|

RE: Is there any plan to develop an application level fair scheduler?

Xia, Junluan
Are you sure that you must deploy spark in standalone mode?(it currently only support FIFO)

If you could setup Spark on Yarn or Mesos, then it has supported Fair scheduler in application level.

-----Original Message-----
From: Nan Zhu [mailto:[hidden email]]
Sent: Tuesday, January 14, 2014 10:13 AM
To: [hidden email]
Subject: Is there any plan to develop an application level fair scheduler?

Hi, All  

Is there any plan to develop an application level fair scheduler?

I think it will have more value than a fair scheduler within the application (actually I didn’t understand why we want to fairly share the resource among jobs within the application, in usual, users submit different applications, not jobs)…

Best,  

--  
Nan Zhu

Reply | Threaded
Open this post in threaded view
|

Re: Is there any plan to develop an application level fair scheduler?

Nan Zhu
Hi, Junluan,  

Thank you for the reply  

but for the long-term plan, Spark will depend on Yarn and Mesos for application level scheduling in the coming versions?

Best,  

--  
Nan Zhu


On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote:

> Are you sure that you must deploy spark in standalone mode?(it currently only support FIFO)
>  
> If you could setup Spark on Yarn or Mesos, then it has supported Fair scheduler in application level.
>  
> -----Original Message-----
> From: Nan Zhu [mailto:[hidden email]]  
> Sent: Tuesday, January 14, 2014 10:13 AM
> To: [hidden email] (mailto:[hidden email])
> Subject: Is there any plan to develop an application level fair scheduler?
>  
> Hi, All  
>  
> Is there any plan to develop an application level fair scheduler?
>  
> I think it will have more value than a fair scheduler within the application (actually I didn’t understand why we want to fairly share the resource among jobs within the application, in usual, users submit different applications, not jobs)…
>  
> Best,  
>  
> --  
> Nan Zhu
>  
>  


Reply | Threaded
Open this post in threaded view
|

RE: Is there any plan to develop an application level fair scheduler?

Xia, Junluan
Yes, Spark depends on Yarn or Mesos for application level scheduling.

-----Original Message-----
From: Nan Zhu [mailto:[hidden email]]
Sent: Tuesday, January 14, 2014 9:43 PM
To: [hidden email]
Subject: Re: Is there any plan to develop an application level fair scheduler?

Hi, Junluan,  

Thank you for the reply  

but for the long-term plan, Spark will depend on Yarn and Mesos for application level scheduling in the coming versions?

Best,  

--  
Nan Zhu


On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote:

> Are you sure that you must deploy spark in standalone mode?(it currently only support FIFO)
>  
> If you could setup Spark on Yarn or Mesos, then it has supported Fair scheduler in application level.
>  
> -----Original Message-----
> From: Nan Zhu [mailto:[hidden email]]  
> Sent: Tuesday, January 14, 2014 10:13 AM
> To: [hidden email] (mailto:[hidden email])
> Subject: Is there any plan to develop an application level fair scheduler?
>  
> Hi, All  
>  
> Is there any plan to develop an application level fair scheduler?
>  
> I think it will have more value than a fair scheduler within the application (actually I didn’t understand why we want to fairly share the resource among jobs within the application, in usual, users submit different applications, not jobs)…
>  
> Best,  
>  
> --  
> Nan Zhu
>  
>  


Reply | Threaded
Open this post in threaded view
|

Re: Is there any plan to develop an application level fair scheduler?

Matei Zaharia
Administrator
This is true for now, we didn’t want to replicate those systems. But it may change if we see demand for fair scheduling in our standalone cluster manager.

Matei

On Jan 14, 2014, at 6:32 PM, Xia, Junluan <[hidden email]> wrote:

> Yes, Spark depends on Yarn or Mesos for application level scheduling.
>
> -----Original Message-----
> From: Nan Zhu [mailto:[hidden email]]
> Sent: Tuesday, January 14, 2014 9:43 PM
> To: [hidden email]
> Subject: Re: Is there any plan to develop an application level fair scheduler?
>
> Hi, Junluan,  
>
> Thank you for the reply  
>
> but for the long-term plan, Spark will depend on Yarn and Mesos for application level scheduling in the coming versions?
>
> Best,  
>
> --  
> Nan Zhu
>
>
> On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote:
>
>> Are you sure that you must deploy spark in standalone mode?(it currently only support FIFO)
>>
>> If you could setup Spark on Yarn or Mesos, then it has supported Fair scheduler in application level.
>>
>> -----Original Message-----
>> From: Nan Zhu [mailto:[hidden email]]  
>> Sent: Tuesday, January 14, 2014 10:13 AM
>> To: [hidden email] (mailto:[hidden email])
>> Subject: Is there any plan to develop an application level fair scheduler?
>>
>> Hi, All  
>>
>> Is there any plan to develop an application level fair scheduler?
>>
>> I think it will have more value than a fair scheduler within the application (actually I didn’t understand why we want to fairly share the resource among jobs within the application, in usual, users submit different applications, not jobs)…
>>
>> Best,  
>>
>> --  
>> Nan Zhu
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Is there any plan to develop an application level fair scheduler?

Evan Chan
What is the reason that standalone mode doesn't support the fair scheduler?
Does that mean that Mesos coarse mode also doesn't support the fair
scheduler?


On Tue, Jan 14, 2014 at 8:10 PM, Matei Zaharia <[hidden email]>wrote:

> This is true for now, we didn’t want to replicate those systems. But it
> may change if we see demand for fair scheduling in our standalone cluster
> manager.
>
> Matei
>
> On Jan 14, 2014, at 6:32 PM, Xia, Junluan <[hidden email]> wrote:
>
> > Yes, Spark depends on Yarn or Mesos for application level scheduling.
> >
> > -----Original Message-----
> > From: Nan Zhu [mailto:[hidden email]]
> > Sent: Tuesday, January 14, 2014 9:43 PM
> > To: [hidden email]
> > Subject: Re: Is there any plan to develop an application level fair
> scheduler?
> >
> > Hi, Junluan,
> >
> > Thank you for the reply
> >
> > but for the long-term plan, Spark will depend on Yarn and Mesos for
> application level scheduling in the coming versions?
> >
> > Best,
> >
> > --
> > Nan Zhu
> >
> >
> > On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote:
> >
> >> Are you sure that you must deploy spark in standalone mode?(it
> currently only support FIFO)
> >>
> >> If you could setup Spark on Yarn or Mesos, then it has supported Fair
> scheduler in application level.
> >>
> >> -----Original Message-----
> >> From: Nan Zhu [mailto:[hidden email]]
> >> Sent: Tuesday, January 14, 2014 10:13 AM
> >> To: [hidden email] (mailto:
> [hidden email])
> >> Subject: Is there any plan to develop an application level fair
> scheduler?
> >>
> >> Hi, All
> >>
> >> Is there any plan to develop an application level fair scheduler?
> >>
> >> I think it will have more value than a fair scheduler within the
> application (actually I didn’t understand why we want to fairly share the
> resource among jobs within the application, in usual, users submit
> different applications, not jobs)…
> >>
> >> Best,
> >>
> >> --
> >> Nan Zhu
> >>
> >>
> >
> >
>
>


--
--
Evan Chan
Staff Engineer
[hidden email]  |

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
Reply | Threaded
Open this post in threaded view
|

Re: Is there any plan to develop an application level fair scheduler?

rxin
It does.

There are two scheduling levels here.

The first level is what the cluster manager does. The standalone cluster
manager for Spark only supports FIFO at the moment at the level of
applications.

Regarding Spark itself. Within a single Spark application, both FIFO and
fair scheduling are supported, regardless of what your cluster manager is



On Fri, Jan 17, 2014 at 12:17 PM, Evan Chan <[hidden email]> wrote:

> What is the reason that standalone mode doesn't support the fair scheduler?
> Does that mean that Mesos coarse mode also doesn't support the fair
> scheduler?
>
>
> On Tue, Jan 14, 2014 at 8:10 PM, Matei Zaharia <[hidden email]
> >wrote:
>
> > This is true for now, we didn’t want to replicate those systems. But it
> > may change if we see demand for fair scheduling in our standalone cluster
> > manager.
> >
> > Matei
> >
> > On Jan 14, 2014, at 6:32 PM, Xia, Junluan <[hidden email]> wrote:
> >
> > > Yes, Spark depends on Yarn or Mesos for application level scheduling.
> > >
> > > -----Original Message-----
> > > From: Nan Zhu [mailto:[hidden email]]
> > > Sent: Tuesday, January 14, 2014 9:43 PM
> > > To: [hidden email]
> > > Subject: Re: Is there any plan to develop an application level fair
> > scheduler?
> > >
> > > Hi, Junluan,
> > >
> > > Thank you for the reply
> > >
> > > but for the long-term plan, Spark will depend on Yarn and Mesos for
> > application level scheduling in the coming versions?
> > >
> > > Best,
> > >
> > > --
> > > Nan Zhu
> > >
> > >
> > > On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote:
> > >
> > >> Are you sure that you must deploy spark in standalone mode?(it
> > currently only support FIFO)
> > >>
> > >> If you could setup Spark on Yarn or Mesos, then it has supported Fair
> > scheduler in application level.
> > >>
> > >> -----Original Message-----
> > >> From: Nan Zhu [mailto:[hidden email]]
> > >> Sent: Tuesday, January 14, 2014 10:13 AM
> > >> To: [hidden email] (mailto:
> > [hidden email])
> > >> Subject: Is there any plan to develop an application level fair
> > scheduler?
> > >>
> > >> Hi, All
> > >>
> > >> Is there any plan to develop an application level fair scheduler?
> > >>
> > >> I think it will have more value than a fair scheduler within the
> > application (actually I didn’t understand why we want to fairly share the
> > resource among jobs within the application, in usual, users submit
> > different applications, not jobs)…
> > >>
> > >> Best,
> > >>
> > >> --
> > >> Nan Zhu
> > >>
> > >>
> > >
> > >
> >
> >
>
>
> --
> --
> Evan Chan
> Staff Engineer
> [hidden email]  |
>
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><
> http://www.twitter.com/ooyala>
>
Reply | Threaded
Open this post in threaded view
|

Re: Is there any plan to develop an application level fair scheduler?

Matei Zaharia
Administrator
Yup, this page describes the two levels of scheduling (within and across applications): http://spark.incubator.apache.org/docs/latest/job-scheduling.html


On Jan 17, 2014, at 12:21 PM, Reynold Xin <[hidden email]> wrote:

> It does.
>
> There are two scheduling levels here.
>
> The first level is what the cluster manager does. The standalone cluster
> manager for Spark only supports FIFO at the moment at the level of
> applications.
>
> Regarding Spark itself. Within a single Spark application, both FIFO and
> fair scheduling are supported, regardless of what your cluster manager is
>
>
>
> On Fri, Jan 17, 2014 at 12:17 PM, Evan Chan <[hidden email]> wrote:
>
>> What is the reason that standalone mode doesn't support the fair scheduler?
>> Does that mean that Mesos coarse mode also doesn't support the fair
>> scheduler?
>>
>>
>> On Tue, Jan 14, 2014 at 8:10 PM, Matei Zaharia <[hidden email]
>>> wrote:
>>
>>> This is true for now, we didn’t want to replicate those systems. But it
>>> may change if we see demand for fair scheduling in our standalone cluster
>>> manager.
>>>
>>> Matei
>>>
>>> On Jan 14, 2014, at 6:32 PM, Xia, Junluan <[hidden email]> wrote:
>>>
>>>> Yes, Spark depends on Yarn or Mesos for application level scheduling.
>>>>
>>>> -----Original Message-----
>>>> From: Nan Zhu [mailto:[hidden email]]
>>>> Sent: Tuesday, January 14, 2014 9:43 PM
>>>> To: [hidden email]
>>>> Subject: Re: Is there any plan to develop an application level fair
>>> scheduler?
>>>>
>>>> Hi, Junluan,
>>>>
>>>> Thank you for the reply
>>>>
>>>> but for the long-term plan, Spark will depend on Yarn and Mesos for
>>> application level scheduling in the coming versions?
>>>>
>>>> Best,
>>>>
>>>> --
>>>> Nan Zhu
>>>>
>>>>
>>>> On Tuesday, January 14, 2014 at 12:56 AM, Xia, Junluan wrote:
>>>>
>>>>> Are you sure that you must deploy spark in standalone mode?(it
>>> currently only support FIFO)
>>>>>
>>>>> If you could setup Spark on Yarn or Mesos, then it has supported Fair
>>> scheduler in application level.
>>>>>
>>>>> -----Original Message-----
>>>>> From: Nan Zhu [mailto:[hidden email]]
>>>>> Sent: Tuesday, January 14, 2014 10:13 AM
>>>>> To: [hidden email] (mailto:
>>> [hidden email])
>>>>> Subject: Is there any plan to develop an application level fair
>>> scheduler?
>>>>>
>>>>> Hi, All
>>>>>
>>>>> Is there any plan to develop an application level fair scheduler?
>>>>>
>>>>> I think it will have more value than a fair scheduler within the
>>> application (actually I didn’t understand why we want to fairly share the
>>> resource among jobs within the application, in usual, users submit
>>> different applications, not jobs)…
>>>>>
>>>>> Best,
>>>>>
>>>>> --
>>>>> Nan Zhu
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> --
>> --
>> Evan Chan
>> Staff Engineer
>> [hidden email]  |
>>
>> <http://www.ooyala.com/>
>> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><
>> http://www.twitter.com/ooyala>
>>