[3.0.1] ExecutorMonitor.onJobStart and StageInfo.shuffleDepId that's never used?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

[3.0.1] ExecutorMonitor.onJobStart and StageInfo.shuffleDepId that's never used?

Jacek Laskowski
Hi,

It's been a while. Glad to be back Sparkians!

I've been exploring ExecutorMonitor.onJobStart in 3.0.1 and noticed that it uses StageInfo.shuffleDepId [1] that is None by default and moreover never "written to" according to IntelliJ IDEA.

Is this the case and intentional?

I'm wondering how much IDEA knows about codegen and that's where it's used (?)

I've just stumbled upon it and before I spend more time on this I thought I'd ask (perhaps it's going to change in 3.1?). Help appreciated.

Reply | Threaded
Open this post in threaded view
|

Re: [3.0.1] ExecutorMonitor.onJobStart and StageInfo.shuffleDepId that's never used?

Jacek Laskowski
Hi,

Sorry. A false alarm. Got mistaken with what IDEA calls "unused" may not really be unused. It is (re)assigned in StageInfo.fromStage for a ShuffleMapStage [1] and then caught in ExecutorMonitor [2] (since it's a SparkListener).


On Wed, Dec 30, 2020 at 3:34 PM Jacek Laskowski <[hidden email]> wrote:
Hi,

It's been a while. Glad to be back Sparkians!

I've been exploring ExecutorMonitor.onJobStart in 3.0.1 and noticed that it uses StageInfo.shuffleDepId [1] that is None by default and moreover never "written to" according to IntelliJ IDEA.

Is this the case and intentional?

I'm wondering how much IDEA knows about codegen and that's where it's used (?)

I've just stumbled upon it and before I spend more time on this I thought I'd ask (perhaps it's going to change in 3.1?). Help appreciated.