Bucketing and catalyst

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Bucketing and catalyst

Long, Andrew-2

Hey Friends,

 

How aware of bucketing is Catalyst? I’ve been trying to piece together how Catalyst knows that it can remove a sort and shuffle given that both tables are bucketed and sorted the same way. Is there any classes in particular I should look at?

 

Cheers Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Bucketing and catalyst

Ryan Blue
Andrew,

Here's an umbrella issue that is a good starting point for looking at the project to add Hive bucketing support: https://issues.apache.org/jira/browse/SPARK-19256

rb

On Thu, May 2, 2019 at 11:40 AM Long, Andrew <[hidden email]> wrote:

Hey Friends,

 

How aware of bucketing is Catalyst? I’ve been trying to piece together how Catalyst knows that it can remove a sort and shuffle given that both tables are bucketed and sorted the same way. Is there any classes in particular I should look at?

 

Cheers Andrew



--
Ryan Blue
Software Engineer
Netflix