RE: [SS] Why does ConsoleSink's addBatch convert input DataFrame to show it?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

RE: [SS] Why does ConsoleSink's addBatch convert input DataFrame to show it?

assaf.mendelson

I actually asked the same thing a couple of weeks ago.

Apparently, when you create a structured streaming plan, it is different than the batch plan and is fixed in order to properly aggregate. If you perform most operations on the dataframe it will recalculate the plan as a batch plan and will therefore not work properly. Therefore, you must either collect or turn to RDD and then create a new dataframe from the RDD.

It would be very useful IMO if we can “freeze” the plan for the input portion and work as if it was a new dataframe (similar to turning it to RDD and then creating a new dataframe from the RDD but without the overhead of converting to RDD and back to dataframe), however, this is not currently possible.

 

Thanks,

              Assaf.

 

From: Jacek Laskowski [via Apache Spark Developers List] [mailto:ml+[hidden email]]
Sent: Friday, July 07, 2017 11:44 AM
To: Mendelson, Assaf
Subject: [SS] Why does ConsoleSink's addBatch convert input DataFrame to show it?

 

Hi,

Just noticed that the input DataFrame is collect'ed and then
parallelize'd simply to show it to the console [1]. Why so many fairly
expensive operations for show?

I'd appreciate some help understanding this code. Thanks.

[1] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/console.scala#L51-L53

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



To start a new topic under Apache Spark Developers List, email [hidden email]
To unsubscribe from Apache Spark Developers List, click here.
NAML