[DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

Weichen Xu
Hi all,

I would like to discuss the compatibility for dev scripts. Because we already decided to deprecate python2 in spark 3.0, for development scripts under dev/ , we have two choice:
1) Migration from Python 2 to Python 3
2) Support both Python 2 and Python 3

I tend to option (2) which is more friendly to maintenance.

Regards,
Weichen
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

Hyukjin Kwon
We didn't drop Python 2 yet although it's deprecated. So I think It should support both Python 2 and Python 3 at the current status.

2019년 8월 7일 (수) 오후 6:54, Weichen Xu <[hidden email]>님이 작성:
Hi all,

I would like to discuss the compatibility for dev scripts. Because we already decided to deprecate python2 in spark 3.0, for development scripts under dev/ , we have two choice:
1) Migration from Python 2 to Python 3
2) Support both Python 2 and Python 3

I tend to option (2) which is more friendly to maintenance.

Regards,
Weichen
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

Weichen Xu
All right we could support both Python 2 and Python 3 for spark 3.0.

On Wed, Aug 7, 2019 at 6:10 PM Hyukjin Kwon <[hidden email]> wrote:
We didn't drop Python 2 yet although it's deprecated. So I think It should support both Python 2 and Python 3 at the current status.

2019년 8월 7일 (수) 오후 6:54, Weichen Xu <[hidden email]>님이 작성:
Hi all,

I would like to discuss the compatibility for dev scripts. Because we already decided to deprecate python2 in spark 3.0, for development scripts under dev/ , we have two choice:
1) Migration from Python 2 to Python 3
2) Support both Python 2 and Python 3

I tend to option (2) which is more friendly to maintenance.

Regards,
Weichen
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

Driesprong, Fokko
Sorry for the late reply, was a bit busy lately, but I still would like to share my thoughts on this.

For Apache Airflow we're dropping support for Python 2 in the next major release. We're now supporting Python 3.5+. Mostly because:
  • Easier to maintain and test, and less if/else constructions for the different Python versions. Also, not having to test against Python 2.x reduces the build matrix.
  • Python 3 has support for typing. From Python 3.5 you can include provisional type hints. An excellent presentation by Guido himself: https://www.youtube.com/watch?v=2wDvzy6Hgxg. From Python 3.5 it is still provisional, but it is a really good idea. From Airflow we've noticed that using mypy is catching bugs early:
    • This will put less stress on the (boring part of the) reviewing process since a lot of this stuff is checked automatically.
    • For new developers, it is easier to read the code because of the annotations.
    • Can be used as an input for generated documentation (or check if it still in sync with the docstrings)
    • Easier to extend the code since you know what kind of types you can expect, and your IDE will also pick up the hinting.
  • Python 2.x will be EOL end this year
I have a strong preference to migrate everything to Python 3.

Cheers, Fokko


Op wo 7 aug. 2019 om 12:14 schreef Weichen Xu <[hidden email]>:
All right we could support both Python 2 and Python 3 for spark 3.0.

On Wed, Aug 7, 2019 at 6:10 PM Hyukjin Kwon <[hidden email]> wrote:
We didn't drop Python 2 yet although it's deprecated. So I think It should support both Python 2 and Python 3 at the current status.

2019년 8월 7일 (수) 오후 6:54, Weichen Xu <[hidden email]>님이 작성:
Hi all,

I would like to discuss the compatibility for dev scripts. Because we already decided to deprecate python2 in spark 3.0, for development scripts under dev/ , we have two choice:
1) Migration from Python 2 to Python 3
2) Support both Python 2 and Python 3

I tend to option (2) which is more friendly to maintenance.

Regards,
Weichen
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

Hyukjin Kwon
Yeah, we will probably drop Python 2 entirely after 3.0.0. Python 2 is already deprecated.

On Thu, 15 Aug 2019, 18:25 Driesprong, Fokko, <[hidden email]> wrote:
Sorry for the late reply, was a bit busy lately, but I still would like to share my thoughts on this.

For Apache Airflow we're dropping support for Python 2 in the next major release. We're now supporting Python 3.5+. Mostly because:
  • Easier to maintain and test, and less if/else constructions for the different Python versions. Also, not having to test against Python 2.x reduces the build matrix.
  • Python 3 has support for typing. From Python 3.5 you can include provisional type hints. An excellent presentation by Guido himself: https://www.youtube.com/watch?v=2wDvzy6Hgxg. From Python 3.5 it is still provisional, but it is a really good idea. From Airflow we've noticed that using mypy is catching bugs early:
    • This will put less stress on the (boring part of the) reviewing process since a lot of this stuff is checked automatically.
    • For new developers, it is easier to read the code because of the annotations.
    • Can be used as an input for generated documentation (or check if it still in sync with the docstrings)
    • Easier to extend the code since you know what kind of types you can expect, and your IDE will also pick up the hinting.
  • Python 2.x will be EOL end this year
I have a strong preference to migrate everything to Python 3.

Cheers, Fokko


Op wo 7 aug. 2019 om 12:14 schreef Weichen Xu <[hidden email]>:
All right we could support both Python 2 and Python 3 for spark 3.0.

On Wed, Aug 7, 2019 at 6:10 PM Hyukjin Kwon <[hidden email]> wrote:
We didn't drop Python 2 yet although it's deprecated. So I think It should support both Python 2 and Python 3 at the current status.

2019년 8월 7일 (수) 오후 6:54, Weichen Xu <[hidden email]>님이 작성:
Hi all,

I would like to discuss the compatibility for dev scripts. Because we already decided to deprecate python2 in spark 3.0, for development scripts under dev/ , we have two choice:
1) Migration from Python 2 to Python 3
2) Support both Python 2 and Python 3

I tend to option (2) which is more friendly to maintenance.

Regards,
Weichen
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3

Hyukjin Kwon
I mean python 2 _will be_ deprecated in Spark 3.

On Thu, 15 Aug 2019, 18:37 Hyukjin Kwon, <[hidden email]> wrote:
Yeah, we will probably drop Python 2 entirely after 3.0.0. Python 2 is already deprecated.

On Thu, 15 Aug 2019, 18:25 Driesprong, Fokko, <[hidden email]> wrote:
Sorry for the late reply, was a bit busy lately, but I still would like to share my thoughts on this.

For Apache Airflow we're dropping support for Python 2 in the next major release. We're now supporting Python 3.5+. Mostly because:
  • Easier to maintain and test, and less if/else constructions for the different Python versions. Also, not having to test against Python 2.x reduces the build matrix.
  • Python 3 has support for typing. From Python 3.5 you can include provisional type hints. An excellent presentation by Guido himself: https://www.youtube.com/watch?v=2wDvzy6Hgxg. From Python 3.5 it is still provisional, but it is a really good idea. From Airflow we've noticed that using mypy is catching bugs early:
    • This will put less stress on the (boring part of the) reviewing process since a lot of this stuff is checked automatically.
    • For new developers, it is easier to read the code because of the annotations.
    • Can be used as an input for generated documentation (or check if it still in sync with the docstrings)
    • Easier to extend the code since you know what kind of types you can expect, and your IDE will also pick up the hinting.
  • Python 2.x will be EOL end this year
I have a strong preference to migrate everything to Python 3.

Cheers, Fokko


Op wo 7 aug. 2019 om 12:14 schreef Weichen Xu <[hidden email]>:
All right we could support both Python 2 and Python 3 for spark 3.0.

On Wed, Aug 7, 2019 at 6:10 PM Hyukjin Kwon <[hidden email]> wrote:
We didn't drop Python 2 yet although it's deprecated. So I think It should support both Python 2 and Python 3 at the current status.

2019년 8월 7일 (수) 오후 6:54, Weichen Xu <[hidden email]>님이 작성:
Hi all,

I would like to discuss the compatibility for dev scripts. Because we already decided to deprecate python2 in spark 3.0, for development scripts under dev/ , we have two choice:
1) Migration from Python 2 to Python 3
2) Support both Python 2 and Python 3

I tend to option (2) which is more friendly to maintenance.

Regards,
Weichen