BlockManager and ShuffleManager = can getLocalBytes be ever used for shuffle blocks?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

BlockManager and ShuffleManager = can getLocalBytes be ever used for shuffle blocks?

Jacek Laskowski
Hi,

While trying to understand the relationship of BlockManager and ShuffleManager I found that ShuffleManager is used for shuffle block data [1] (and that makes sense).

What I found quite surprising is that BlockManager can call getLocalBytes for non-shuffle blocks that in turn does...fetching shuffle blocks (!) [2].

That begs the question, who could want this? In other words, what other components would want to call BlockManager.getLocalBytes for shuffle blocks? The only caller is TorrentBroadcast which is not for shuffle blocks, is it?

In other words, getLocalBytes should NOT be bothering itself with fetching local shuffle blocks as that's a responsibility of someone else, i.e. ShuffleManager.

Is this correct?

I'd appreciate any help you can provide so I can understand the Spark core better. Thanks!

Reply | Threaded
Open this post in threaded view
|

Re: BlockManager and ShuffleManager = can getLocalBytes be ever used for shuffle blocks?

wuyi
Hi Jacek,

code in link [2] is the out of date. The commit
https://github.com/apache/spark/commit/32ec528e63cb768f85644282978040157c3c2fb7
has already removed unreachable branch.


Best,
Yi Wu



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: BlockManager and ShuffleManager = can getLocalBytes be ever used for shuffle blocks?

Jacek Laskowski

On Sat, Apr 18, 2020 at 12:17 PM wuyi <[hidden email]> wrote:
Hi Jacek,

code in link [2] is the out of date. The commit
https://github.com/apache/spark/commit/32ec528e63cb768f85644282978040157c3c2fb7
has already removed unreachable branch.


Best,
Yi Wu



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]