Query regarding stateless aggregations

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Query regarding stateless aggregations

Chitral Verma
Hi Devs,
I have a query regarding stateless aggregations.

I understand that its possible to do stateless aggregation using mapGroups and flatMapGroups API in Spark 2.x+. I want to use aggregate queries on a streaming registered temporary view. Is there any way to do the same using spark.sql(" ... ") ?

Also posted here,

Any helps will be appreciated.

Regards,
Chitral Verma
Reply | Threaded
Open this post in threaded view
|

Re: Query regarding stateless aggregations

Jungtaek Lim-2
I'm not sure the meaning of stateless aggregation for structured streaming.

The ground rule is that both batch query and streaming query should provide same output if there's no late rows and no indeterministic functions/operations - the query should not produce different outputs depending on how the input is broken down to multiple batches. By the definition of ground rule, streaming aggregation is required to be stateful.

Thanks,
Jungtaek Lim (HeartSaVioR)

On Thu, Nov 28, 2019 at 9:17 PM Chitral Verma <[hidden email]> wrote:
Hi Devs,
I have a query regarding stateless aggregations.

I understand that its possible to do stateless aggregation using mapGroups and flatMapGroups API in Spark 2.x+. I want to use aggregate queries on a streaming registered temporary view. Is there any way to do the same using spark.sql(" ... ") ?

Also posted here,

Any helps will be appreciated.

Regards,
Chitral Verma