Tests failing with run-tests.py SyntaxError

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Tests failing with run-tests.py SyntaxError

Sean Owen
  File "./dev/run-tests.py", line 124
    {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)
                                                            ^
SyntaxError: invalid syntax

It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?)

Does the error suggest anything to anyone who knows Python better than I?

The line has been around for a year so I don't think it's the script per se.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests failing with run-tests.py SyntaxError

Hyukjin Kwon
That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before -https://issues.apache.org/jira/browse/SPARK-20149

On 28 Jul 2017 9:56 pm, "Sean Owen" <[hidden email]> wrote:
  File "./dev/run-tests.py", line 124
    {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)
                                                            ^
SyntaxError: invalid syntax

It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?)

Does the error suggest anything to anyone who knows Python better than I?

The line has been around for a year so I don't think it's the script per se.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests failing with run-tests.py SyntaxError

Sean Owen
I see, does that suggest that a machine has 2.6, when it should use 2.7?

On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <[hidden email]> wrote:
That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before -https://issues.apache.org/jira/browse/SPARK-20149

On 28 Jul 2017 9:56 pm, "Sean Owen" <[hidden email]> wrote:
  File "./dev/run-tests.py", line 124
    {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)
                                                            ^
SyntaxError: invalid syntax

It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?)

Does the error suggest anything to anyone who knows Python better than I?

The line has been around for a year so I don't think it's the script per se.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests failing with run-tests.py SyntaxError

Hyukjin Kwon
Yes, that's my guess just given information here without a close look.

On 28 Jul 2017 11:03 pm, "Sean Owen" <[hidden email]> wrote:
I see, does that suggest that a machine has 2.6, when it should use 2.7?

On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <[hidden email]> wrote:
That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before -https://issues.apache.org/jira/browse/SPARK-20149

On 28 Jul 2017 9:56 pm, "Sean Owen" <[hidden email]> wrote:
  File "./dev/run-tests.py", line 124
    {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)
                                                            ^
SyntaxError: invalid syntax

It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?)

Does the error suggest anything to anyone who knows Python better than I?

The line has been around for a year so I don't think it's the script per se.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests failing with run-tests.py SyntaxError

Dong Joon Hyun

I saw that error in the latest branch-2.1 build failure, too.

 

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.7/579/console

 

But, the code was written in Jan 2016. Didn’t we run it on Python 2.6 without any problem?

 

ee74498de37 (Josh Rosen       2016-01-26 14:20:11 -0800 124)         {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)

 

Bests,

Dongjoon.

 

From: Hyukjin Kwon <[hidden email]>
Date: Friday, July 28, 2017 at 7:06 AM
To: Sean Owen <[hidden email]>
Cc: dev <[hidden email]>
Subject: Re: Tests failing with run-tests.py SyntaxError

 

Yes, that's my guess just given information here without a close look.

 

On 28 Jul 2017 11:03 pm, "Sean Owen" <[hidden email]> wrote:

I see, does that suggest that a machine has 2.6, when it should use 2.7?

 

On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <[hidden email]> wrote:

That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before -https://issues.apache.org/jira/browse/SPARK-20149

 

On 28 Jul 2017 9:56 pm, "Sean Owen" <[hidden email]> wrote:

  File "./dev/run-tests.py", line 124
    {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)
                                                            ^
SyntaxError: invalid syntax


It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?)


Does the error suggest anything to anyone who knows Python better than I?

 

The line has been around for a year so I don't think it's the script per se.

 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests failing with run-tests.py SyntaxError

Hyukjin Kwon
I am sorry for saying just based on my wild guess because I have no way to check and take a look into Jenkins but I think we might have to set the explicit Python version in https://github.com/apache/spark/blob/master/dev/run-tests-jenkins#L29

I guess we set the explicit Python version for running unit tests but it looks the script itself is dependent python executable in the path.



On 29 Jul 2017 7:39 am, "Dong Joon Hyun" <[hidden email]> wrote:

I saw that error in the latest branch-2.1 build failure, too.

 

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.7/579/console

 

But, the code was written in Jan 2016. Didn’t we run it on Python 2.6 without any problem?

 

ee74498de37 (Josh Rosen       2016-01-26 14:20:11 -0800 124)         {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)

 

Bests,

Dongjoon.

 

From: Hyukjin Kwon <[hidden email]>
Date: Friday, July 28, 2017 at 7:06 AM
To: Sean Owen <[hidden email]>
Cc: dev <[hidden email]>
Subject: Re: Tests failing with run-tests.py SyntaxError

 

Yes, that's my guess just given information here without a close look.

 

On 28 Jul 2017 11:03 pm, "Sean Owen" <[hidden email]> wrote:

I see, does that suggest that a machine has 2.6, when it should use 2.7?

 

On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <[hidden email]> wrote:

That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before -https://issues.apache.org/jira/browse/SPARK-20149

 

On 28 Jul 2017 9:56 pm, "Sean Owen" <[hidden email]> wrote:

  File "./dev/run-tests.py", line 124
    {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)
                                                            ^
SyntaxError: invalid syntax


It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?)


Does the error suggest anything to anyone who knows Python better than I?

 

The line has been around for a year so I don't think it's the script per se.

 


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests failing with run-tests.py SyntaxError

Hyukjin Kwon
Or maybe in https://github.com/apache/spark/blob/master/dev/run-tests#L23

On 29 Jul 2017 11:16 am, "Hyukjin Kwon" <[hidden email]> wrote:
I am sorry for saying just based on my wild guess because I have no way to check and take a look into Jenkins but I think we might have to set the explicit Python version in https://github.com/apache/spark/blob/master/dev/run-tests-jenkins#L29

I guess we set the explicit Python version for running unit tests but it looks the script itself is dependent python executable in the path.



On 29 Jul 2017 7:39 am, "Dong Joon Hyun" <[hidden email]> wrote:

I saw that error in the latest branch-2.1 build failure, too.

 

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.7/579/console

 

But, the code was written in Jan 2016. Didn’t we run it on Python 2.6 without any problem?

 

ee74498de37 (Josh Rosen       2016-01-26 14:20:11 -0800 124)         {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)

 

Bests,

Dongjoon.

 

From: Hyukjin Kwon <[hidden email]>
Date: Friday, July 28, 2017 at 7:06 AM
To: Sean Owen <[hidden email]>
Cc: dev <[hidden email]>
Subject: Re: Tests failing with run-tests.py SyntaxError

 

Yes, that's my guess just given information here without a close look.

 

On 28 Jul 2017 11:03 pm, "Sean Owen" <[hidden email]> wrote:

I see, does that suggest that a machine has 2.6, when it should use 2.7?

 

On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <[hidden email]> wrote:

That looks appearently due to dict comprehension which is, IIRC, not allowed in Python 2.6.x. I checked the release note for sure before -https://issues.apache.org/jira/browse/SPARK-20149

 

On 28 Jul 2017 9:56 pm, "Sean Owen" <[hidden email]> wrote:

  File "./dev/run-tests.py", line 124
    {m: set(m.dependencies).intersection(modules_to_test) for m in modules_to_test}, sort=True)
                                                            ^
SyntaxError: invalid syntax


It seems like tests are failing intermittently with this type of error, which leads me to guess there's some difference in the Python interpreter on one or more machines but not all (?)


Does the error suggest anything to anyone who knows Python better than I?

 

The line has been around for a year so I don't think it's the script per se.

 



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Tests failing with run-tests.py SyntaxError

shane knapp
(for the wider dev audience)

there's an apache issue open for this, which i have commented on.

https://issues.apache.org/jira/browse/SPARK-21573

i have a workaround and will be getting this sorted asap.


On Fri, Jul 28, 2017 at 7:27 PM, Hyukjin Kwon <[hidden email]> wrote:

> Or maybe in https://github.com/apache/spark/blob/master/dev/run-tests#L23
>
>
> On 29 Jul 2017 11:16 am, "Hyukjin Kwon" <[hidden email]> wrote:
>
> I am sorry for saying just based on my wild guess because I have no way to
> check and take a look into Jenkins but I think we might have to set the
> explicit Python version in
> https://github.com/apache/spark/blob/master/dev/run-tests-jenkins#L29
>
> I guess we set the explicit Python version for running unit tests but it
> looks the script itself is dependent python executable in the path.
>
>
>
> On 29 Jul 2017 7:39 am, "Dong Joon Hyun" <[hidden email]> wrote:
>
> I saw that error in the latest branch-2.1 build failure, too.
>
>
>
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.7/579/console
>
>
>
> But, the code was written in Jan 2016. Didn’t we run it on Python 2.6
> without any problem?
>
>
>
> ee74498de37 (Josh Rosen       2016-01-26 14:20:11 -0800 124)         {m:
> set(m.dependencies).intersection(modules_to_test) for m in modules_to_test},
> sort=True)
>
>
>
> Bests,
>
> Dongjoon.
>
>
>
> From: Hyukjin Kwon <[hidden email]>
> Date: Friday, July 28, 2017 at 7:06 AM
> To: Sean Owen <[hidden email]>
> Cc: dev <[hidden email]>
> Subject: Re: Tests failing with run-tests.py SyntaxError
>
>
>
> Yes, that's my guess just given information here without a close look.
>
>
>
> On 28 Jul 2017 11:03 pm, "Sean Owen" <[hidden email]> wrote:
>
> I see, does that suggest that a machine has 2.6, when it should use 2.7?
>
>
>
> On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <[hidden email]> wrote:
>
> That looks appearently due to dict comprehension which is, IIRC, not allowed
> in Python 2.6.x. I checked the release note for sure before
> -https://issues.apache.org/jira/browse/SPARK-20149
>
>
>
> On 28 Jul 2017 9:56 pm, "Sean Owen" <[hidden email]> wrote:
>
>   File "./dev/run-tests.py", line 124
>
>     {m: set(m.dependencies).intersection(modules_to_test) for m in
> modules_to_test}, sort=True)
>
>                                                             ^
>
> SyntaxError: invalid syntax
>
>
> It seems like tests are failing intermittently with this type of error,
> which leads me to guess there's some difference in the Python interpreter on
> one or more machines but not all (?)
>
>
> Does the error suggest anything to anyone who knows Python better than I?
>
>
>
> The line has been around for a year so I don't think it's the script per se.
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Loading...