[DOCS] Spark SQL Upgrading Guide

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[DOCS] Spark SQL Upgrading Guide

Jacek Laskowski
Hi,

Just noticed that http://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html (Spark 2.4.5) has formatting issues in "Upgrading from Spark SQL 2.4.3 to 2.4.4" [1] which got fixed in master [2]. That's OK.

What made me wonder was the other change to the section "Upgrading from Spark SQL 2.4 to 2.4.5" [3] that had the following item included:

"Starting from 2.4.5, SQL configurations are effective also when a Dataset is converted to an RDD and its plan is executed due to action on the derived RDD. The previous behavior can be restored setting spark.sql.legacy.rdd.applyConf to false: in this case, SQL configurations are ignored for operations performed on a RDD derived from a Dataset."

Why was this removed in master [4]? It was mentioned in "Notable changes" of Spark Release 2.4.5 [5].

Reply | Threaded
Open this post in threaded view
|

Re: [DOCS] Spark SQL Upgrading Guide

Jacek Laskowski
Hi,

Never mind. Found this [1]:

> This config is deprecated and it will be removed in 3.0.0.

And so it has :) Thanks and sorry for the trouble.



On Sat, Feb 15, 2020 at 7:44 PM Jacek Laskowski <[hidden email]> wrote:
Hi,

Just noticed that http://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html (Spark 2.4.5) has formatting issues in "Upgrading from Spark SQL 2.4.3 to 2.4.4" [1] which got fixed in master [2]. That's OK.

What made me wonder was the other change to the section "Upgrading from Spark SQL 2.4 to 2.4.5" [3] that had the following item included:

"Starting from 2.4.5, SQL configurations are effective also when a Dataset is converted to an RDD and its plan is executed due to action on the derived RDD. The previous behavior can be restored setting spark.sql.legacy.rdd.applyConf to false: in this case, SQL configurations are ignored for operations performed on a RDD derived from a Dataset."

Why was this removed in master [4]? It was mentioned in "Notable changes" of Spark Release 2.4.5 [5].

Reply | Threaded
Open this post in threaded view
|

Re: [DOCS] Spark SQL Upgrading Guide

Hyukjin Kwon
Thanks for checking it, Jacek.

2020년 2월 16일 (일) 오후 7:23, Jacek Laskowski <[hidden email]>님이 작성:
Hi,

Never mind. Found this [1]:

> This config is deprecated and it will be removed in 3.0.0.

And so it has :) Thanks and sorry for the trouble.



On Sat, Feb 15, 2020 at 7:44 PM Jacek Laskowski <[hidden email]> wrote:
Hi,

Just noticed that http://spark.apache.org/docs/latest/sql-migration-guide-upgrade.html (Spark 2.4.5) has formatting issues in "Upgrading from Spark SQL 2.4.3 to 2.4.4" [1] which got fixed in master [2]. That's OK.

What made me wonder was the other change to the section "Upgrading from Spark SQL 2.4 to 2.4.5" [3] that had the following item included:

"Starting from 2.4.5, SQL configurations are effective also when a Dataset is converted to an RDD and its plan is executed due to action on the derived RDD. The previous behavior can be restored setting spark.sql.legacy.rdd.applyConf to false: in this case, SQL configurations are ignored for operations performed on a RDD derived from a Dataset."

Why was this removed in master [4]? It was mentioned in "Notable changes" of Spark Release 2.4.5 [5].