Migration `Spark QA Compile` Jenkins jobs to GitHub Action

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Migration `Spark QA Compile` Jenkins jobs to GitHub Action

Dongjoon Hyun-2
Hi, All.

Apache Spark community used the following dashboard as post-hook verifications.

    https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/

There are six registered jobs.

    1. spark-branch-2.4-compile-maven-hadoop-2.6
    2. spark-branch-2.4-compile-maven-hadoop-2.7
    3. spark-branch-2.4-lint
    4. spark-master-compile-maven-hadoop-2.7
    5. spark-master-compile-maven-hadoop-3.2
    6. spark-master-lint

Now, we added `GitHub Action` jobs. You can see the green check at every commit.

    https://github.com/apache/spark/commits/master
    https://github.com/apache/spark/commits/branch-2.4

If you click the green check, you can see the detail.
The followings are the example runs at the last commits on both branches.

    https://github.com/apache/spark/runs/310411948 (master)
    https://github.com/apache/spark/runs/309522646 (branch-2.4)

New `GitHub Action` have more combination than the old Jenkins jobs.

    - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
    - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
    - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
    - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
    - branch-2.4-linters (Scala/Java/Python/R)
    - master-scala-2.12-hadoop-2.7 (compile/package/install)
    - master-scala-2.12-hadoop-3.2 (compile/package/install)
    - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
    - master-linters (Scala/Java/Python/R)

In addition, this is a part of Apache Spark code base and everyone can make contributions on this.

Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.

    https://issues.apache.org/jira/browse/SPARK-29935

Please let me know if you have any concerns on this.
(We can keep the legacy jobs, but two of them are already broken.)

Bests,
Dongjoon.
Reply | Threaded
Open this post in threaded view
|

Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action

Sean Owen-2
I would favor moving whatever we can to Github. It's difficult to
modify the Jenkins instances without Shane's valiant help, and over
time makes more sense to modernize and integrate it into the project.

On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <[hidden email]> wrote:

>
> Hi, All.
>
> Apache Spark community used the following dashboard as post-hook verifications.
>
>     https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
>
> There are six registered jobs.
>
>     1. spark-branch-2.4-compile-maven-hadoop-2.6
>     2. spark-branch-2.4-compile-maven-hadoop-2.7
>     3. spark-branch-2.4-lint
>     4. spark-master-compile-maven-hadoop-2.7
>     5. spark-master-compile-maven-hadoop-3.2
>     6. spark-master-lint
>
> Now, we added `GitHub Action` jobs. You can see the green check at every commit.
>
>     https://github.com/apache/spark/commits/master
>     https://github.com/apache/spark/commits/branch-2.4
>
> If you click the green check, you can see the detail.
> The followings are the example runs at the last commits on both branches.
>
>     https://github.com/apache/spark/runs/310411948 (master)
>     https://github.com/apache/spark/runs/309522646 (branch-2.4)
>
> New `GitHub Action` have more combination than the old Jenkins jobs.
>
>     - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
>     - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
>     - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
>     - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
>     - branch-2.4-linters (Scala/Java/Python/R)
>     - master-scala-2.12-hadoop-2.7 (compile/package/install)
>     - master-scala-2.12-hadoop-3.2 (compile/package/install)
>     - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
>     - master-linters (Scala/Java/Python/R)
>
> In addition, this is a part of Apache Spark code base and everyone can make contributions on this.
>
> Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.
>
>     https://issues.apache.org/jira/browse/SPARK-29935
>
> Please let me know if you have any concerns on this.
> (We can keep the legacy jobs, but two of them are already broken.)
>
> Bests,
> Dongjoon.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action

shane knapp
thank sean!

i am all for moving these jobs to github actions, and will be doing
this 'soon' as i'm @ kubecon this week.

btw the R ecosystem definitely needs some attention, however, but
that's an issue for another time.  :)

On Tue, Nov 19, 2019 at 1:49 PM Sean Owen <[hidden email]> wrote:

>
> I would favor moving whatever we can to Github. It's difficult to
> modify the Jenkins instances without Shane's valiant help, and over
> time makes more sense to modernize and integrate it into the project.
>
> On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <[hidden email]> wrote:
> >
> > Hi, All.
> >
> > Apache Spark community used the following dashboard as post-hook verifications.
> >
> >     https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
> >
> > There are six registered jobs.
> >
> >     1. spark-branch-2.4-compile-maven-hadoop-2.6
> >     2. spark-branch-2.4-compile-maven-hadoop-2.7
> >     3. spark-branch-2.4-lint
> >     4. spark-master-compile-maven-hadoop-2.7
> >     5. spark-master-compile-maven-hadoop-3.2
> >     6. spark-master-lint
> >
> > Now, we added `GitHub Action` jobs. You can see the green check at every commit.
> >
> >     https://github.com/apache/spark/commits/master
> >     https://github.com/apache/spark/commits/branch-2.4
> >
> > If you click the green check, you can see the detail.
> > The followings are the example runs at the last commits on both branches.
> >
> >     https://github.com/apache/spark/runs/310411948 (master)
> >     https://github.com/apache/spark/runs/309522646 (branch-2.4)
> >
> > New `GitHub Action` have more combination than the old Jenkins jobs.
> >
> >     - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
> >     - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
> >     - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
> >     - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
> >     - branch-2.4-linters (Scala/Java/Python/R)
> >     - master-scala-2.12-hadoop-2.7 (compile/package/install)
> >     - master-scala-2.12-hadoop-3.2 (compile/package/install)
> >     - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
> >     - master-linters (Scala/Java/Python/R)
> >
> > In addition, this is a part of Apache Spark code base and everyone can make contributions on this.
> >
> > Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.
> >
> >     https://issues.apache.org/jira/browse/SPARK-29935
> >
> > Please let me know if you have any concerns on this.
> > (We can keep the legacy jobs, but two of them are already broken.)
> >
> > Bests,
> > Dongjoon.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action

shane knapp
i had a few minutes aaaaand everything has been deleted!

On Tue, Nov 19, 2019 at 2:02 PM Shane Knapp <[hidden email]> wrote:

>
> thank sean!
>
> i am all for moving these jobs to github actions, and will be doing
> this 'soon' as i'm @ kubecon this week.
>
> btw the R ecosystem definitely needs some attention, however, but
> that's an issue for another time.  :)
>
> On Tue, Nov 19, 2019 at 1:49 PM Sean Owen <[hidden email]> wrote:
> >
> > I would favor moving whatever we can to Github. It's difficult to
> > modify the Jenkins instances without Shane's valiant help, and over
> > time makes more sense to modernize and integrate it into the project.
> >
> > On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <[hidden email]> wrote:
> > >
> > > Hi, All.
> > >
> > > Apache Spark community used the following dashboard as post-hook verifications.
> > >
> > >     https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
> > >
> > > There are six registered jobs.
> > >
> > >     1. spark-branch-2.4-compile-maven-hadoop-2.6
> > >     2. spark-branch-2.4-compile-maven-hadoop-2.7
> > >     3. spark-branch-2.4-lint
> > >     4. spark-master-compile-maven-hadoop-2.7
> > >     5. spark-master-compile-maven-hadoop-3.2
> > >     6. spark-master-lint
> > >
> > > Now, we added `GitHub Action` jobs. You can see the green check at every commit.
> > >
> > >     https://github.com/apache/spark/commits/master
> > >     https://github.com/apache/spark/commits/branch-2.4
> > >
> > > If you click the green check, you can see the detail.
> > > The followings are the example runs at the last commits on both branches.
> > >
> > >     https://github.com/apache/spark/runs/310411948 (master)
> > >     https://github.com/apache/spark/runs/309522646 (branch-2.4)
> > >
> > > New `GitHub Action` have more combination than the old Jenkins jobs.
> > >
> > >     - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
> > >     - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
> > >     - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
> > >     - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
> > >     - branch-2.4-linters (Scala/Java/Python/R)
> > >     - master-scala-2.12-hadoop-2.7 (compile/package/install)
> > >     - master-scala-2.12-hadoop-3.2 (compile/package/install)
> > >     - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
> > >     - master-linters (Scala/Java/Python/R)
> > >
> > > In addition, this is a part of Apache Spark code base and everyone can make contributions on this.
> > >
> > > Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.
> > >
> > >     https://issues.apache.org/jira/browse/SPARK-29935
> > >
> > > Please let me know if you have any concerns on this.
> > > (We can keep the legacy jobs, but two of them are already broken.)
> > >
> > > Bests,
> > > Dongjoon.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: [hidden email]
> >
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action

Dongjoon Hyun-2
Thank you, Sean, Shane, and Xiao!

Bests,
Dongjoon.

On Tue, Nov 19, 2019 at 2:15 PM Shane Knapp <[hidden email]> wrote:
i had a few minutes aaaaand everything has been deleted!

On Tue, Nov 19, 2019 at 2:02 PM Shane Knapp <[hidden email]> wrote:
>
> thank sean!
>
> i am all for moving these jobs to github actions, and will be doing
> this 'soon' as i'm @ kubecon this week.
>
> btw the R ecosystem definitely needs some attention, however, but
> that's an issue for another time.  :)
>
> On Tue, Nov 19, 2019 at 1:49 PM Sean Owen <[hidden email]> wrote:
> >
> > I would favor moving whatever we can to Github. It's difficult to
> > modify the Jenkins instances without Shane's valiant help, and over
> > time makes more sense to modernize and integrate it into the project.
> >
> > On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <[hidden email]> wrote:
> > >
> > > Hi, All.
> > >
> > > Apache Spark community used the following dashboard as post-hook verifications.
> > >
> > >     https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
> > >
> > > There are six registered jobs.
> > >
> > >     1. spark-branch-2.4-compile-maven-hadoop-2.6
> > >     2. spark-branch-2.4-compile-maven-hadoop-2.7
> > >     3. spark-branch-2.4-lint
> > >     4. spark-master-compile-maven-hadoop-2.7
> > >     5. spark-master-compile-maven-hadoop-3.2
> > >     6. spark-master-lint
> > >
> > > Now, we added `GitHub Action` jobs. You can see the green check at every commit.
> > >
> > >     https://github.com/apache/spark/commits/master
> > >     https://github.com/apache/spark/commits/branch-2.4
> > >
> > > If you click the green check, you can see the detail.
> > > The followings are the example runs at the last commits on both branches.
> > >
> > >     https://github.com/apache/spark/runs/310411948 (master)
> > >     https://github.com/apache/spark/runs/309522646 (branch-2.4)
> > >
> > > New `GitHub Action` have more combination than the old Jenkins jobs.
> > >
> > >     - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
> > >     - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
> > >     - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
> > >     - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
> > >     - branch-2.4-linters (Scala/Java/Python/R)
> > >     - master-scala-2.12-hadoop-2.7 (compile/package/install)
> > >     - master-scala-2.12-hadoop-3.2 (compile/package/install)
> > >     - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
> > >     - master-linters (Scala/Java/Python/R)
> > >
> > > In addition, this is a part of Apache Spark code base and everyone can make contributions on this.
> > >
> > > Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.
> > >
> > >     https://issues.apache.org/jira/browse/SPARK-29935
> > >
> > > Please let me know if you have any concerns on this.
> > > (We can keep the legacy jobs, but two of them are already broken.)
> > >
> > > Bests,
> > > Dongjoon.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: [hidden email]
> >
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu