[DISCUSS] Deprecate Python < 3.6 in Spark 3.0

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Deprecate Python < 3.6 in Spark 3.0

zero323

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Dongjoon Hyun-2
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Xiangrui Meng-2
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Hyukjin Kwon
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

shane knapp
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Holden Karau
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Takeshi Yamamuro
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Dongjoon Hyun-2
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

shane knapp
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

shane knapp
also, here's my PR for dropping 2.7 tests:

On Wed, Oct 30, 2019 at 1:45 PM Shane Knapp <[hidden email]> wrote:
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.

Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

zero323
In reply to this post by shane knapp

Could we upgrade to PyPy3.6 v7.2.0?

On 10/30/19 9:45 PM, Shane Knapp wrote:
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.
Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
-- 
Best regards,
Maciej
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

shane knapp
sure.  that shouldn't be too hard, but we've historically given very little support to it.

On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz <[hidden email]> wrote:

Could we upgrade to PyPy3.6 v7.2.0?

On 10/30/19 9:45 PM, Shane Knapp wrote:
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.
Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Bryan Cutler
+1 for deprecating

On Wed, Oct 30, 2019 at 2:46 PM Shane Knapp <[hidden email]> wrote:
sure.  that shouldn't be too hard, but we've historically given very little support to it.

On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz <[hidden email]> wrote:

Could we upgrade to PyPy3.6 v7.2.0?

On 10/30/19 9:45 PM, Shane Knapp wrote:
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.
Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

Takuya UESHIN
+1

On Thu, Oct 31, 2019 at 11:21 AM Bryan Cutler <[hidden email]> wrote:
+1 for deprecating

On Wed, Oct 30, 2019 at 2:46 PM Shane Knapp <[hidden email]> wrote:
sure.  that shouldn't be too hard, but we've historically given very little support to it.

On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz <[hidden email]> wrote:

Could we upgrade to PyPy3.6 v7.2.0?

On 10/30/19 9:45 PM, Shane Knapp wrote:
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.
Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Takuya UESHIN
Tokyo, Japan

http://twitter.com/ueshin
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

shane knapp
In reply to this post by zero323
i'm currently testing PyPy3.6 v7.2.0 w/this pull request:  https://github.com/apache/spark/pull/26330

On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz <[hidden email]> wrote:

Could we upgrade to PyPy3.6 v7.2.0?

On 10/30/19 9:45 PM, Shane Knapp wrote:
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.
Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Deprecate Python < 3.6 in Spark 3.0

shane knapp
so...  i've updated the python testing framework to run on python 3.6, and drop support for testing against python 2.7 (and pypy 2.5.1).


anyways:  this is ready to merge, but i'm more than happy to hold off for the time being if we're still a little shy about dropping 2.7 so quickly.

thoughts?  opinions?

On Thu, Oct 31, 2019 at 1:00 PM Shane Knapp <[hidden email]> wrote:
i'm currently testing PyPy3.6 v7.2.0 w/this pull request:  https://github.com/apache/spark/pull/26330

On Wed, Oct 30, 2019 at 2:31 PM Maciej Szymkiewicz <[hidden email]> wrote:

Could we upgrade to PyPy3.6 v7.2.0?

On 10/30/19 9:45 PM, Shane Knapp wrote:
one quick thing:  we currently test against python2.7, 3.6 *and* pypy2.5.1 (python2.7).

what are our plans for pypy?


On Wed, Oct 30, 2019 at 12:26 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you all. I made a PR for that.


On Tue, Oct 29, 2019 at 5:45 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Tue, Oct 29, 2019 at 4:16 PM Holden Karau <[hidden email]> wrote:
+1 to deprecating but not yet removing support for 3.6

On Tue, Oct 29, 2019 at 3:47 AM Shane Knapp <[hidden email]> wrote:
+1 to testing the absolute minimum number of python variants as possible.  ;)

On Mon, Oct 28, 2019 at 7:46 PM Hyukjin Kwon <[hidden email]> wrote:
+1 from me as well.

2019년 10월 29일 (화) 오전 5:34, Xiangrui Meng <[hidden email]>님이 작성:
+1. And we should start testing 3.7 and maybe 3.8 in Jenkins.

On Thu, Oct 24, 2019 at 9:34 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you for starting the thread.

In addition to that, we currently are testing Python 3.6 only in Apache Spark Jenkins environment.

Given that Python 3.8 is already out and Apache Spark 3.0.0 RC1 will start next January
(https://spark.apache.org/versioning-policy.html), I'm +1 for the deprecation (Python < 3.6) at Apache Spark 3.0.0.

It's just a deprecation to prepare the next-step development cycle.
Bests,
Dongjoon.


On Thu, Oct 24, 2019 at 1:10 AM Maciej Szymkiewicz <[hidden email]> wrote:

Hi everyone,

While deprecation of Python 2 in 3.0.0 has been announced, there is no clear statement about specific continuing support of different Python 3 version.

Specifically:

  • Python 3.4 has been retired this year.
  • Python 3.5 is already in the "security fixes only" mode and should be retired in the middle of 2020.

Continued support of these two blocks adoption of many new Python features (PEP 468)  and it is hard to justify beyond 2020.

Should these two be deprecated in 3.0.0 as well?

-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
---
Takeshi Yamamuro


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
-- 
Best regards,
Maciej


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead