Quantcast

Outstanding Spark 2.1.1 issues

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Outstanding Spark 2.1.1 issues

Holden Karau
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Daniel Siegmann-2
Any chance of back-porting

SPARK-14536 - NPE in JDBCRDD when array column contains nulls (postgresql)

It just adds a null check - just a simple bug fix - so it really belongs in Spark 2.1.x.

On Mon, Mar 20, 2017 at 6:12 PM, Holden Karau <[hidden email]> wrote:
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" target="_blank">425-233-8271

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Felix Cheung
In reply to this post by Holden Karau

From: [hidden email] <[hidden email]> on behalf of Holden Karau <[hidden email]>
Sent: Monday, March 20, 2017 3:12:35 PM
To: [hidden email]
Subject: Outstanding Spark 2.1.1 issues
 
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Nan Zhu
I think https://issues.apache.org/jira/browse/SPARK-19280 should be a blocker

Best,

Nan 

On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <[hidden email]> wrote:

From: [hidden email] <[hidden email]> on behalf of Holden Karau <[hidden email]>
Sent: Monday, March 20, 2017 3:12:35 PM
To: [hidden email]
Subject: Outstanding Spark 2.1.1 issues
 
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" target="_blank">425-233-8271

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Holden Karau
I'm not super sure it should be a blocker for 2.1.1 -- is it a regression? Maybe we can get TDs input on it?

On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <[hidden email]> wrote:
I think https://issues.apache.org/jira/browse/SPARK-19280 should be a blocker

Best,

Nan 

On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <[hidden email]> wrote:

From: [hidden email] <[hidden email]> on behalf of Holden Karau <[hidden email]>
Sent: Monday, March 20, 2017 3:12:35 PM
To: [hidden email]
Subject: Outstanding Spark 2.1.1 issues
 
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="gmail_msg" target="_blank">425-233-8271

--
Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Michael Armbrust
Please speak up if I'm wrong, but none of these seem like critical regressions from 2.1.  As such I'll start the RC process later today.

On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <[hidden email]> wrote:
I'm not super sure it should be a blocker for 2.1.1 -- is it a regression? Maybe we can get TDs input on it?

On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <[hidden email]> wrote:
I think https://issues.apache.org/jira/browse/SPARK-19280 should be a blocker

Best,

Nan 

On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <[hidden email]> wrote:

From: [hidden email] <[hidden email]> on behalf of Holden Karau <[hidden email]>
Sent: Monday, March 20, 2017 3:12:35 PM
To: [hidden email]
Subject: Outstanding Spark 2.1.1 issues
 
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="m_-4336284996730264084gmail_msg" target="_blank">425-233-8271

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" target="_blank">425-233-8271

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Holden Karau
I agree with Michael, I think we've got some outstanding issues but none of them seem like regression from 2.1 so we should be good to start the RC process.

On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <[hidden email]> wrote:
Please speak up if I'm wrong, but none of these seem like critical regressions from 2.1.  As such I'll start the RC process later today.

On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <[hidden email]> wrote:
I'm not super sure it should be a blocker for 2.1.1 -- is it a regression? Maybe we can get TDs input on it?

On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <[hidden email]> wrote:
I think https://issues.apache.org/jira/browse/SPARK-19280 should be a blocker

Best,

Nan 

On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <[hidden email]> wrote:

From: [hidden email] <[hidden email]> on behalf of Holden Karau <[hidden email]>
Sent: Monday, March 20, 2017 3:12:35 PM
To: [hidden email]
Subject: Outstanding Spark 2.1.1 issues
 
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="m_-289239107677719864m_-4336284996730264084gmail_msg" target="_blank">425-233-8271

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" target="_blank">425-233-8271




--
Cell : 425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Nick Pentreath
As for SPARK-19759, I don't think that needs to be targeted for 2.1.1 so we don't need to worry about it

On Tue, 21 Mar 2017 at 13:49 Holden Karau <[hidden email]> wrote:
I agree with Michael, I think we've got some outstanding issues but none of them seem like regression from 2.1 so we should be good to start the RC process.

On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <[hidden email]> wrote:
Please speak up if I'm wrong, but none of these seem like critical regressions from 2.1.  As such I'll start the RC process later today.

On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <[hidden email]> wrote:
I'm not super sure it should be a blocker for 2.1.1 -- is it a regression? Maybe we can get TDs input on it?

On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <[hidden email]> wrote:
I think https://issues.apache.org/jira/browse/SPARK-19280 should be a blocker

Best,

Nan 

On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <[hidden email]> wrote:

From: [hidden email] <[hidden email]> on behalf of Holden Karau <[hidden email]>
Sent: Monday, March 20, 2017 3:12:35 PM
To: [hidden email]
Subject: Outstanding Spark 2.1.1 issues
 
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="m_-812936943720441423m_-289239107677719864m_-4336284996730264084gmail_msg gmail_msg" target="_blank">425-233-8271

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="gmail_msg" target="_blank">425-233-8271




--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="gmail_msg" target="_blank">425-233-8271
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Outstanding Spark 2.1.1 issues

Michael Armbrust
An update: I cut the tag for RC1 last night.  Currently fighting with the release process.  Will post RC1 once I get it working.

On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <[hidden email]> wrote:
As for SPARK-19759, I don't think that needs to be targeted for 2.1.1 so we don't need to worry about it


On Tue, 21 Mar 2017 at 13:49 Holden Karau <[hidden email]> wrote:
I agree with Michael, I think we've got some outstanding issues but none of them seem like regression from 2.1 so we should be good to start the RC process.

On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <[hidden email]> wrote:
Please speak up if I'm wrong, but none of these seem like critical regressions from 2.1.  As such I'll start the RC process later today.

On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau <[hidden email]> wrote:
I'm not super sure it should be a blocker for 2.1.1 -- is it a regression? Maybe we can get TDs input on it?

On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu <[hidden email]> wrote:
I think https://issues.apache.org/jira/browse/SPARK-19280 should be a blocker

Best,

Nan 

On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <[hidden email]> wrote:

From: [hidden email] <[hidden email]> on behalf of Holden Karau <[hidden email]>
Sent: Monday, March 20, 2017 3:12:35 PM
To: [hidden email]
Subject: Outstanding Spark 2.1.1 issues
 
Hi Spark Developers!

As we start working on the Spark 2.1.1 release I've been looking at our outstanding issues still targeted for it. I've tried to break it down by component so that people in charge of each component can take a quick look and see if any of these things can/should be re-targeted to 2.2 or 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only look at explicitly tagged) :)

If your working on something for Spark 2.1.1 and it doesn't show up in this list please speak up now :) We have a lot of issues (including "in progress") that are listed as impacting 2.1.0, but they aren't targeted for 2.1.1 - if there is something you are working in their which should be targeted for 2.1.1 please let us know so it doesn't slip through the cracks.

The query string I used for looking at the 2.1.1 open issues is:

((affectedVersion = 2.1.1 AND cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND project = spark AND resolution = Unresolved ORDER BY priority DESC

None of the open issues appear to be a regression from 2.1.0, but those seem more likely to show up during the RC process (thanks in advance to everyone testing their workloads :)) & generally none of them seem to be 

(Note: the cfs are for Target Version/s field)

Critical Issues:
 SQL:
  SPARK-19690 - Join a streaming DataFrame with a batch DataFrame may not work - PR https://github.com/apache/spark/pull/17052 (review in progress by zsxwing, currently failing Jenkins)*

Major Issues:
 SQL:
  SPARK-19035 - rand() function in case when cause failed - no outstanding PR (consensus on JIRA seems to be leaning towards it being a real issue but not necessarily everyone agrees just yet - maybe we should slip this?)*
 Deploy:
  SPARK-19522 - --executor-memory flag doesn't work in local-cluster mode - https://github.com/apache/spark/pull/16975 (review in progress by vanzin, but PR currently stalled waiting on response) *
 Core:
  SPARK-20025 - Driver fail over will not work, if SPARK_LOCAL* env is set. - https://github.com/apache/spark/pull/17357 (waiting on review) *
 PySpark:
 SPARK-19955 - Update run-tests to support conda [ Part of Dropping 2.6 support -- which we shouldn't do in a minor release -- but also fixes pip installability tests to run in Jenkins ]-  PR failing Jenkins (I need to poke this some more, but seems like 2.7 support works but some other issues. Maybe slip to 2.2?)

Minor issues:
 Tests:
  SPARK-19612 - Tests failing with timeout - No PR per-se but it seems unrelated to the 2.1.1 release. It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 PySpark:
  SPARK-19570 - Allow to disable hive in pyspark shell - https://github.com/apache/spark/pull/16906 PR exists but its difficult to add automated tests for this (although if SPARK-19955 gets in would make testing this easier) - no reviewers yet. Possible re-target?*
 Structured Streaming:
  SPARK-19613 - Flaky test: StateStoreRDDSuite.versioning and immutability - It's not targetted for 2.1.1 but listed as affecting 2.1.1 - I'd consider explicitly targeting this for 2.2?
 ML:
  SPARK-19759 - ALSModel.predict on Dataframes : potential optimization by not using blas - No PR consider re-targeting unless someone has a PR waiting in the wings?

Explicitly targeted issues are marked with a *, the remaining issues are listed as impacting 2.1.1 and don't have a specific target version set.

Since 2.1.1 continues the 2.1.0 branch, looking at 2.1.0 shows 1 open blocker in SQL( SPARK-19983 ), 

Query string is:

affectedVersion = 2.1.0 AND cf[12310320] is EMPTY AND project = spark AND resolution = Unresolved AND priority = targetPriority

Continuing on for unresolved 2.1.0 issues in Major there are 163 (76 of them in progress), 65 Minor (26 in progress), and 9 trivial (6 in progress).

I'll be going through the 2.1.0 major issues with open PRs that impact the PySpark component and seeing if any of them should be targeted for 2.1.1, if anyone from the other components wants to take a look through we might find some easy wins to be merged.

Cheers,

Holden :)

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="m_2890494928288814842m_-812936943720441423m_-289239107677719864m_-4336284996730264084gmail_msg m_2890494928288814842gmail_msg" target="_blank">425-233-8271

--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="m_2890494928288814842gmail_msg" target="_blank">425-233-8271




--
Cell : <a href="tel:(425)%20233-8271" value="+14252338271" class="m_2890494928288814842gmail_msg" target="_blank">425-233-8271

Loading...