Reviewers for Stage level Scheduling prs

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Reviewers for Stage level Scheduling prs

Tom Graves-2
Hey everyone,

I'm trying to get reviewers for the Stage Level Scheduling pull requests.  I was hoping to get this into Spark 3.0.
The code is mostly complete - its just missing the webUI and final doc changes.

If anyone has time, reviews from committers would be appreciated.

You can find information about the overall feature and design here:



Regards,
Tom Graves
Reply | Threaded
Open this post in threaded view
|

How executor Understand which RDDs needed to be persist from the submitted Task

Jack Kolokasis
Hello all,

I want to find when a Task that is sended by Driver to executor contains
a call to function persist(). I am trying to read the submitted function
that driver send to executor but I could not find any call to persist()
method. Do you know how executor understand which RDDs needed to be persist?

Thanks,
Iacovos Kolokasis

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How executor Understand which RDDs needed to be persist from the submitted Task

cloud0fan
RDD has a flag `storageLevel` which will be set by calling persist(). RDD will be serialized and sent to executors for running tasks. So executors just look at RDD.storageLevel and store output in its block manager when needed.

On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis <[hidden email]> wrote:
Hello all,

I want to find when a Task that is sended by Driver to executor contains
a call to function persist(). I am trying to read the submitted function
that driver send to executor but I could not find any call to persist()
method. Do you know how executor understand which RDDs needed to be persist?

Thanks,
Iacovos Kolokasis

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How executor Understand which RDDs needed to be persist from the submitted Task

Jack Kolokasis

Thanks for the help. I read that driver only send a function (task) to executors and the executors apply this function to their local RDD partitions.

Iacovos

On 1/9/20 5:03 PM, Wenchen Fan wrote:
RDD has a flag `storageLevel` which will be set by calling persist(). RDD will be serialized and sent to executors for running tasks. So executors just look at RDD.storageLevel and store output in its block manager when needed.

On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis <[hidden email]> wrote:
Hello all,

I want to find when a Task that is sended by Driver to executor contains
a call to function persist(). I am trying to read the submitted function
that driver send to executor but I could not find any call to persist()
method. Do you know how executor understand which RDDs needed to be persist?

Thanks,
Iacovos Kolokasis

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How executor Understand which RDDs needed to be persist from the submitted Task

cloud0fan
You can take a look at ShuffleMapTask.runTask. It's not just a function.

On Thu, Jan 9, 2020 at 11:25 PM Jack Kolokasis <[hidden email]> wrote:

Thanks for the help. I read that driver only send a function (task) to executors and the executors apply this function to their local RDD partitions.

Iacovos

On 1/9/20 5:03 PM, Wenchen Fan wrote:
RDD has a flag `storageLevel` which will be set by calling persist(). RDD will be serialized and sent to executors for running tasks. So executors just look at RDD.storageLevel and store output in its block manager when needed.

On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis <[hidden email]> wrote:
Hello all,

I want to find when a Task that is sended by Driver to executor contains
a call to function persist(). I am trying to read the submitted function
that driver send to executor but I could not find any call to persist()
method. Do you know how executor understand which RDDs needed to be persist?

Thanks,
Iacovos Kolokasis

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: How executor Understand which RDDs needed to be persist from the submitted Task

Jack Kolokasis

Thanks for your help!

Iacovos

On 1/9/20 5:49 PM, Wenchen Fan wrote:
You can take a look at ShuffleMapTask.runTask. It's not just a function.

On Thu, Jan 9, 2020 at 11:25 PM Jack Kolokasis <[hidden email]> wrote:

Thanks for the help. I read that driver only send a function (task) to executors and the executors apply this function to their local RDD partitions.

Iacovos

On 1/9/20 5:03 PM, Wenchen Fan wrote:
RDD has a flag `storageLevel` which will be set by calling persist(). RDD will be serialized and sent to executors for running tasks. So executors just look at RDD.storageLevel and store output in its block manager when needed.

On Thu, Jan 9, 2020 at 5:53 PM Jack Kolokasis <[hidden email]> wrote:
Hello all,

I want to find when a Task that is sended by Driver to executor contains
a call to function persist(). I am trying to read the submitted function
that driver send to executor but I could not find any call to persist()
method. Do you know how executor understand which RDDs needed to be persist?

Thanks,
Iacovos Kolokasis

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]