Aggregate pushdown for data source

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Aggregate pushdown for data source

Arun Khetarpal
Hi Folks: 

I have implemented a data source v2 API for an internal source. As a consequence of generating the data source, we have bunch of statistical information about the source which i can potentially use, only if spark pushes down the aggregates down to the data source itself. 

I see that there is already a comprehensive Jira for the same: https://issues.apache.org/jira/browse/SPARK-22390. Are there any more blockers for this Jira? I'll be more than happy to contribute if no one else had picked it up. 

Regards,
Arun