Revisiting Python / pandas UDF

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Revisiting Python / pandas UDF

rxin
Hi all,

In the past two years, the pandas UDFs are perhaps the most important changes to Spark for Python data science. However, these functionalities have evolved organically, leading to some inconsistencies and confusions among users. I created a ticket and a document summarizing the issues, and a concrete proposal to fix them (the changes are pretty small). Thanks Xiangrui for initially bringing this to my attention, and Li Jin, Hyukjin, for offline discussions.

Please take a look: