Use /usr/bin/env python3 in scripts?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Use /usr/bin/env python3 in scripts?

Sean Owen-2
So, we are on Python 3 entirely now right?
It might be just my local Mac env, but "/usr/bin/env python" uses
Python 2 on my mac.
Some scripts write "/usr/bin/env python3" now. Should that be the case
in all scripts?
Right now the merge script doesn't work for me b/c it was just updated
to be Python 3 only.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Use /usr/bin/env python3 in scripts?

Oli McCormack
[Warning: not spark+python specific information]

It's recommended that you should explicitly call out python3 in a case like this (see PEP-0394, and SO). Your environment is typical: python is often a pointer to python2 for tooling compatibility reasons (other tools or scripts that expect they're going to get python2 when they call python), and you should use python3 to use the new version. What python points to will change over time, so it's recommended to use python2 if explicitly depending on that.

More generally: It's common/recommended to use a virtual environment + explicitly stated versions of Python and dependencies, rather than system Python, so that python means exactly what you intend it to. I know very little about the Spark python dev stack and how challenging it may be to do this, so please take this with a dose of naiveté.

- Oli


On Fri, Jul 17, 2020 at 9:58 AM Sean Owen <[hidden email]> wrote:
So, we are on Python 3 entirely now right?
It might be just my local Mac env, but "/usr/bin/env python" uses
Python 2 on my mac.
Some scripts write "/usr/bin/env python3" now. Should that be the case
in all scripts?
Right now the merge script doesn't work for me b/c it was just updated
to be Python 3 only.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Use /usr/bin/env python3 in scripts?

Sean Owen-2
Yeah I figured it's a best practice, so I'll raise a PR unless
somebody tells me not to. This is about build scripts, not Pyspark
itself, and half the scripts already specify python3.

On Fri, Jul 17, 2020 at 12:36 PM Oli McCormack <[hidden email]> wrote:

>
> [Warning: not spark+python specific information]
>
> It's recommended that you should explicitly call out python3 in a case like this (see PEP-0394, and SO). Your environment is typical: python is often a pointer to python2 for tooling compatibility reasons (other tools or scripts that expect they're going to get python2 when they call python), and you should use python3 to use the new version. What python points to will change over time, so it's recommended to use python2 if explicitly depending on that.
>
> More generally: It's common/recommended to use a virtual environment + explicitly stated versions of Python and dependencies, rather than system Python, so that python means exactly what you intend it to. I know very little about the Spark python dev stack and how challenging it may be to do this, so please take this with a dose of naiveté.
>
> - Oli
>
>
> On Fri, Jul 17, 2020 at 9:58 AM Sean Owen <[hidden email]> wrote:
>>
>> So, we are on Python 3 entirely now right?
>> It might be just my local Mac env, but "/usr/bin/env python" uses
>> Python 2 on my mac.
>> Some scripts write "/usr/bin/env python3" now. Should that be the case
>> in all scripts?
>> Right now the merge script doesn't work for me b/c it was just updated
>> to be Python 3 only.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Use /usr/bin/env python3 in scripts?

Driesprong, Fokko
+1 I'm in favor of using python3

Cheers, Fokko

Op vr 17 jul. 2020 om 19:49 schreef Sean Owen <[hidden email]>:
Yeah I figured it's a best practice, so I'll raise a PR unless
somebody tells me not to. This is about build scripts, not Pyspark
itself, and half the scripts already specify python3.

On Fri, Jul 17, 2020 at 12:36 PM Oli McCormack <[hidden email]> wrote:
>
> [Warning: not spark+python specific information]
>
> It's recommended that you should explicitly call out python3 in a case like this (see PEP-0394, and SO). Your environment is typical: python is often a pointer to python2 for tooling compatibility reasons (other tools or scripts that expect they're going to get python2 when they call python), and you should use python3 to use the new version. What python points to will change over time, so it's recommended to use python2 if explicitly depending on that.
>
> More generally: It's common/recommended to use a virtual environment + explicitly stated versions of Python and dependencies, rather than system Python, so that python means exactly what you intend it to. I know very little about the Spark python dev stack and how challenging it may be to do this, so please take this with a dose of naiveté.
>
> - Oli
>
>
> On Fri, Jul 17, 2020 at 9:58 AM Sean Owen <[hidden email]> wrote:
>>
>> So, we are on Python 3 entirely now right?
>> It might be just my local Mac env, but "/usr/bin/env python" uses
>> Python 2 on my mac.
>> Some scripts write "/usr/bin/env python3" now. Should that be the case
>> in all scripts?
>> Right now the merge script doesn't work for me b/c it was just updated
>> to be Python 3 only.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Use /usr/bin/env python3 in scripts?

Jungtaek Lim-2
For me merge script worked for python 2.7, but I got some trouble with the encoding issue (probably from contributor's name) so now I use the merge script with virtualenv & python 3.7.7.

"python3" would be OK for me as well as it doesn't break virtualenv with python 3.

On Sat, Jul 18, 2020 at 6:13 AM Driesprong, Fokko <[hidden email]> wrote:
+1 I'm in favor of using python3

Cheers, Fokko

Op vr 17 jul. 2020 om 19:49 schreef Sean Owen <[hidden email]>:
Yeah I figured it's a best practice, so I'll raise a PR unless
somebody tells me not to. This is about build scripts, not Pyspark
itself, and half the scripts already specify python3.

On Fri, Jul 17, 2020 at 12:36 PM Oli McCormack <[hidden email]> wrote:
>
> [Warning: not spark+python specific information]
>
> It's recommended that you should explicitly call out python3 in a case like this (see PEP-0394, and SO). Your environment is typical: python is often a pointer to python2 for tooling compatibility reasons (other tools or scripts that expect they're going to get python2 when they call python), and you should use python3 to use the new version. What python points to will change over time, so it's recommended to use python2 if explicitly depending on that.
>
> More generally: It's common/recommended to use a virtual environment + explicitly stated versions of Python and dependencies, rather than system Python, so that python means exactly what you intend it to. I know very little about the Spark python dev stack and how challenging it may be to do this, so please take this with a dose of naiveté.
>
> - Oli
>
>
> On Fri, Jul 17, 2020 at 9:58 AM Sean Owen <[hidden email]> wrote:
>>
>> So, we are on Python 3 entirely now right?
>> It might be just my local Mac env, but "/usr/bin/env python" uses
>> Python 2 on my mac.
>> Some scripts write "/usr/bin/env python3" now. Should that be the case
>> in all scripts?
>> Right now the merge script doesn't work for me b/c it was just updated
>> to be Python 3 only.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]