[FYI] CI Infra issues (in both GitHub Action and Jenkins)

[Dongjoon Hyun-2]

Hi, All.

There are two issues currently.

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

2. Lack of GitHub Action backend machines.

    There is a JIRA to ask us to reduce the usage.
    https://issues.apache.org/jira/browse/SPARK-34053

    "Please reduce GitHub Actions matrix or improve the build time"

Bests,

Dongjoon.

[shane knapp ☠]

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

examples please?

--

[Dongjoon Hyun-2]

The followings?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/1836/console

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/1887/console

On Fri, Jan 8, 2021 at 2:13 PM shane knapp ☠ <[hidden email]> wrote:

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

examples please?

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

[Dongjoon Hyun-2]

BTW, Shane, do you think we can utilize some of UCB machines as GitHub Action runners?

Bests,

Dongjoon.

On Fri, Jan 8, 2021 at 2:14 PM Dongjoon Hyun <[hidden email]> wrote:

The followings?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/1836/console

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/1887/console

On Fri, Jan 8, 2021 at 2:13 PM shane knapp ☠ <[hidden email]> wrote:

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

examples please?

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

[shane knapp ☠]

hmm, the ubuntu16 machines are acting up.  i pinned the sbt master builds to ubuntu20 and they're happily building while i investigate wtf is up.

On Fri, Jan 8, 2021 at 2:15 PM Dongjoon Hyun <[hidden email]> wrote:

The followings?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/1836/console

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/1887/console

On Fri, Jan 8, 2021 at 2:13 PM shane knapp ☠ <[hidden email]> wrote:

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

examples please?

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

--

[shane knapp ☠]

no, i don't think that'd be a good idea...  adding additional dependencies to our cluster won't scale one bit.

On Fri, Jan 8, 2021 at 2:16 PM Dongjoon Hyun <[hidden email]> wrote:

BTW, Shane, do you think we can utilize some of UCB machines as GitHub Action runners?

Bests,

Dongjoon.

On Fri, Jan 8, 2021 at 2:14 PM Dongjoon Hyun <[hidden email]> wrote:

The followings?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/1836/console

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/1887/console

On Fri, Jan 8, 2021 at 2:13 PM shane knapp ☠ <[hidden email]> wrote:

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

examples please?

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

--

[Hyukjin Kwon]

For GitHub resources of ASF repo, I have been contacting GitHub to address the issue few days ago. This is not a repo level problem cc [hidden email].

ASF organisation in GitHub has already too many repos, and we should have a way to increase the limit, or set the separare limit specifically for the repo.

On Sat, 9 Jan 2021, 07:34 shane knapp ☠, <[hidden email]> wrote:

no, i don't think that'd be a good idea...  adding additional dependencies to our cluster won't scale one bit.

On Fri, Jan 8, 2021 at 2:16 PM Dongjoon Hyun <[hidden email]> wrote:

BTW, Shane, do you think we can utilize some of UCB machines as GitHub Action runners?

Bests,

Dongjoon.

On Fri, Jan 8, 2021 at 2:14 PM Dongjoon Hyun <[hidden email]> wrote:

The followings?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/1836/console

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/1887/console

On Fri, Jan 8, 2021 at 2:13 PM shane knapp ☠ <[hidden email]> wrote:

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

examples please?

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

[Hyukjin Kwon]

To share the update about GitHub Actions:
- I am informed that having more resources is now being discussed in org level. Hopefully we get the situation better.
- Duplicated workflows will be canceled to save resources, see https://github.com/apache/spark/pull/31104

cc [hidden email] [hidden email] too who were involved

2021년 1월 9일 (토) 오전 8:34, Hyukjin Kwon <[hidden email]>님이 작성:

For GitHub resources of ASF repo, I have been contacting GitHub to address the issue few days ago. This is not a repo level problem cc [hidden email].

ASF organisation in GitHub has already too many repos, and we should have a way to increase the limit, or set the separare limit specifically for the repo.

On Sat, 9 Jan 2021, 07:34 shane knapp ☠, <[hidden email]> wrote:

no, i don't think that'd be a good idea...  adding additional dependencies to our cluster won't scale one bit.

On Fri, Jan 8, 2021 at 2:16 PM Dongjoon Hyun <[hidden email]> wrote:

BTW, Shane, do you think we can utilize some of UCB machines as GitHub Action runners?

Bests,

Dongjoon.

On Fri, Jan 8, 2021 at 2:14 PM Dongjoon Hyun <[hidden email]> wrote:

The followings?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-3.2/1836/console

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/1887/console

On Fri, Jan 8, 2021 at 2:13 PM shane knapp ☠ <[hidden email]> wrote:

1. Jenkins machines start to fail with the following recently.

    (master branch)

    Python versions prior to 3.6 are not supported.
    Build step 'Execute shell' marked build as failure

examples please?

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

--

Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu