./dev/run-tests failing at master

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

./dev/run-tests failing at master

Sudharshann D
Hello! ;)

I'm new to spark development and have been trying to set up my dev environment for hours without much success. :-(

Firstly, I'm wondering why my  ./dev/run-tests fails even though in on the master branch.

This is the error:

flake8 checks failed:
./examples/src/main/python/sql/arrow.py:67:16: E999 SyntaxError: invalid syntax
    def func(s1: pd.Series, s2: pd.Series, s3: pd.DataFrame) -> pd.DataFrame:
               ^
./python/run-tests.py:123:48: E999 SyntaxError: invalid syntax
                        print(decoded_line, end='')
                                               ^
./dev/run-tests-jenkins.py:43:20: E999 SyntaxError: invalid syntax
    print(msg, file=sys.stderr)
                   ^
./dev/run-tests.py:273:44: E999 SyntaxError: invalid syntax
            print(line.decode('utf-8'), end='')
                                           ^
./dev/pip-sanity-check.py:31:76: E999 SyntaxError: invalid syntax
        print("Value {0} did not match expected value.".format(value), file=sys.stderr)
                                                                           ^
5     E999 SyntaxError: invalid syntax
5
1
[error] running /home/vagrant/gtSync/suddhuApache/spark/dev/lint-python ; received return code 1



Would love to have some help with setting up my dev environment. I've done some other open source contributions and setting up the dev env there was easier...

Regards
Sudha
Reply | Threaded
Open this post in threaded view
|

Re: ./dev/run-tests failing at master

Sean Owen-2
Hm, flake8 works OK for me locally in master, and on Jenkins it seems. Could be a version issue?

On Thu, May 14, 2020 at 1:34 PM Sudharshann D <[hidden email]> wrote:
Hello! ;)

I'm new to spark development and have been trying to set up my dev environment for hours without much success. :-(

Firstly, I'm wondering why my  ./dev/run-tests fails even though in on the master branch.

This is the error:

flake8 checks failed:
./examples/src/main/python/sql/arrow.py:67:16: E999 SyntaxError: invalid syntax
    def func(s1: pd.Series, s2: pd.Series, s3: pd.DataFrame) -> pd.DataFrame:
               ^
./python/run-tests.py:123:48: E999 SyntaxError: invalid syntax
                        print(decoded_line, end='')
                                               ^
./dev/run-tests-jenkins.py:43:20: E999 SyntaxError: invalid syntax
    print(msg, file=sys.stderr)
                   ^
./dev/run-tests.py:273:44: E999 SyntaxError: invalid syntax
            print(line.decode('utf-8'), end='')
                                           ^
./dev/pip-sanity-check.py:31:76: E999 SyntaxError: invalid syntax
        print("Value {0} did not match expected value.".format(value), file=sys.stderr)
                                                                           ^
5     E999 SyntaxError: invalid syntax
5
1
[error] running /home/vagrant/gtSync/suddhuApache/spark/dev/lint-python ; received return code 1



Would love to have some help with setting up my dev environment. I've done some other open source contributions and setting up the dev env there was easier...

Regards
Sudha
Reply | Threaded
Open this post in threaded view
|

Re: ./dev/run-tests failing at master

Jeff Evans
In reply to this post by Sudharshann D
Are you positive you set up your Python environment correctly?  To me, those error messages look like you are running Python 2, but it should be Python 3.

On Thu, May 14, 2020 at 1:34 PM Sudharshann D <[hidden email]> wrote:
Hello! ;)

I'm new to spark development and have been trying to set up my dev environment for hours without much success. :-(

Firstly, I'm wondering why my  ./dev/run-tests fails even though in on the master branch.

This is the error:

flake8 checks failed:
./examples/src/main/python/sql/arrow.py:67:16: E999 SyntaxError: invalid syntax
    def func(s1: pd.Series, s2: pd.Series, s3: pd.DataFrame) -> pd.DataFrame:
               ^
./python/run-tests.py:123:48: E999 SyntaxError: invalid syntax
                        print(decoded_line, end='')
                                               ^
./dev/run-tests-jenkins.py:43:20: E999 SyntaxError: invalid syntax
    print(msg, file=sys.stderr)
                   ^
./dev/run-tests.py:273:44: E999 SyntaxError: invalid syntax
            print(line.decode('utf-8'), end='')
                                           ^
./dev/pip-sanity-check.py:31:76: E999 SyntaxError: invalid syntax
        print("Value {0} did not match expected value.".format(value), file=sys.stderr)
                                                                           ^
5     E999 SyntaxError: invalid syntax
5
1
[error] running /home/vagrant/gtSync/suddhuApache/spark/dev/lint-python ; received return code 1



Would love to have some help with setting up my dev environment. I've done some other open source contributions and setting up the dev env there was easier...

Regards
Sudha
Reply | Threaded
Open this post in threaded view
|

Re: ./dev/run-tests failing at master

suddhu
Thanks for the response Jeff and Sean.

It has been quite frustrating setting up the dev environment without any
help. Its comforting to have some help finally.

I've added "alias python=python3"  in my bashrc. So the default python
accessed is 3.7.2 I have flake8 installed in both python2 and python3.



The same error persists.

How should I proceed?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ./dev/run-tests failing at master

suddhu
In reply to this post by Jeff Evans
Thanks for the response Jeff and Sean.

It has been quite frustrating setting up the dev environment without any
help. Its comforting to have some help finally.

I've added "alias python=python3"  in my bashrc. So the default python
accessed is 3.7.2 I have flake8 installed in both python2 and python3.



The same error persists.

How should I proceed?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ./dev/run-tests failing at master

shane knapp ☠
In reply to this post by suddhu
this is the flake8 versioning from a jenkins worker:
$ flake8 --version
3.6.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 3.6.8 on Linux

be sure you've got all the right versions of packages in there.

On Thu, May 14, 2020 at 12:19 PM suddhu <[hidden email]> wrote:
Thanks for the response Jeff and Sean.

It has been quite frustrating setting up the dev environment without any
help. Its comforting to have some help finally.

I've added "alias python=python3"  in my bashrc. So the default python
accessed is 3.7.2 I have flake8 installed in both python2 and python3.



The same error persists.

How should I proceed?



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Reply | Threaded
Open this post in threaded view
|

Re: ./dev/run-tests failing at master

suddhu
Thanks for the response!! Here is my configuration:
flake8 --version
3.8.1 (mccabe: 0.6.1, pycodestyle: 2.6.0, pyflakes: 2.2.0) CPython 2.7.16 on
Linux

It seems I need to upgrade my CPython...



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: ./dev/run-tests failing at master

suddhu
I did this and it seems to work now.

sudo update-alternatives --install /usr/bin/python python /usr/bin/python3
10

even the cpython version got upgraded. Sorry if you feel I was asking too
many rudimentary questions, setting up spark has been rough without a
mentor. Thanks :-D

FYI, I'm working on this issue:
https://issues.apache.org/jira/browse/SPARK-31579?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20Reopened%2C%20Closed)%20AND%20labels%20%3D%20starter#




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]