FlatMapGroupsInPandasExec with multiple record batches

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

FlatMapGroupsInPandasExec with multiple record batches

Terry Kim
Hi,

I see the following comment inĀ FlatMapGroupsInPandasExec.scala:
"It's possible to further split one group into multiple record batches to reduce the memory footprint on the Java side, this is left as future work."

I checked the JIRA but could not find anything related to this. Is there a plan to support this scenario?

Thanks,
Terry