Python API for mapGroupsWithState

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Python API for mapGroupsWithState

Nicholas Chammas
Can someone succinctly describe the challenge in adding the `mapGroupsWithState()` API to PySpark?

I was hoping for some suboptimal but nonetheless working solution to be available in Python, as there are with Python UDFs for example, but that doesn't seem to be case. The JIRA ticket for arbitrary stateful operations in Structured Streaming doesn't give any indication that a Python version of the API is coming.

Is this something that will likely be added in the near future, or is it a major undertaking? Can someone briefly describe the problem?