Spark Packaging Jenkins

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Spark Packaging Jenkins

Dongjoon Hyun-2
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.

Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

shane knapp
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

Dongjoon Hyun-2
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

shane knapp
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

shane knapp

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

Dongjoon Hyun-2
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

cloud0fan
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

Dongjoon Hyun-2
Thank you, Wenchen.

I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too.

Bests,
Dongjoon.

On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <[hidden email]> wrote:
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

Felix Cheung
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed)

The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it)

 

From: Dongjoon Hyun <[hidden email]>
Sent: Saturday, January 5, 2019 9:46 PM
To: Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
Thank you, Wenchen.

I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too.

Bests,
Dongjoon.

On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <[hidden email]> wrote:
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

Felix Cheung
https://spark.apache.org/release-process.html

Look for do-release-docker.sh script

 

From: Felix Cheung <[hidden email]>
Sent: Sunday, January 6, 2019 11:17 AM
To: Dongjoon Hyun; Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed)

The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it)

 

From: Dongjoon Hyun <[hidden email]>
Sent: Saturday, January 5, 2019 9:46 PM
To: Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
Thank you, Wenchen.

I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too.

Bests,
Dongjoon.

On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <[hidden email]> wrote:
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

shane knapp
noted.  i like the idea of building (but not signing) the release and will update the job(s) this week.

On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung <[hidden email]> wrote:

Look for do-release-docker.sh script

 

From: Felix Cheung <[hidden email]>
Sent: Sunday, January 6, 2019 11:17 AM
To: Dongjoon Hyun; Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed)

The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it)

 

From: Dongjoon Hyun <[hidden email]>
Sent: Saturday, January 5, 2019 9:46 PM
To: Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
Thank you, Wenchen.

I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too.

Bests,
Dongjoon.

On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <[hidden email]> wrote:
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

Felix Cheung
Awesome Shane!

 

From: shane knapp <[hidden email]>
Sent: Sunday, January 6, 2019 11:38 AM
To: Felix Cheung
Cc: Dongjoon Hyun; Wenchen Fan; dev
Subject: Re: Spark Packaging Jenkins
 
noted.  i like the idea of building (but not signing) the release and will update the job(s) this week.

On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung <[hidden email]> wrote:

Look for do-release-docker.sh script

 

From: Felix Cheung <[hidden email]>
Sent: Sunday, January 6, 2019 11:17 AM
To: Dongjoon Hyun; Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed)

The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it)

 

From: Dongjoon Hyun <[hidden email]>
Sent: Saturday, January 5, 2019 9:46 PM
To: Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
Thank you, Wenchen.

I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too.

Bests,
Dongjoon.

On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <[hidden email]> wrote:
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

Dongjoon Hyun-2
Thank you, Felix and Shane.

+1 for packaging except signing in Jenkins. That sounds perfect for the next releases (2.2.3, 2.3.3, 3.0.0).

BTW, thanks to Shane, the followings are recovered from today.

1. For all live branches (master to branch-2.2), Jenkins Packaging jobs are recovered to build and packaging. Only there exist signing errors.


2. Daily snapshot maven publishing is recovered. This is helpful for partial module testing.


Thank you all. Everything becomes healthier during 2.2.3 release.
This will reduce the burden of 2.3.3/3.0.0 releases. :D

Bests,
Dongjoon.


On Sun, Jan 6, 2019 at 11:42 AM Felix Cheung <[hidden email]> wrote:
Awesome Shane!

 

From: shane knapp <[hidden email]>
Sent: Sunday, January 6, 2019 11:38 AM
To: Felix Cheung
Cc: Dongjoon Hyun; Wenchen Fan; dev
Subject: Re: Spark Packaging Jenkins
 
noted.  i like the idea of building (but not signing) the release and will update the job(s) this week.

On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung <[hidden email]> wrote:

Look for do-release-docker.sh script

 

From: Felix Cheung <[hidden email]>
Sent: Sunday, January 6, 2019 11:17 AM
To: Dongjoon Hyun; Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed)

The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it)

 

From: Dongjoon Hyun <[hidden email]>
Sent: Saturday, January 5, 2019 9:46 PM
To: Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
Thank you, Wenchen.

I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too.

Bests,
Dongjoon.

On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <[hidden email]> wrote:
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

shane knapp
after spending the last three hours unraveling how dev/create-release/release-build.sh works, i think i figured out what needs to happen.

2) create world-wide campaign to stop bash scripting

my brain...  it hurts.

On Sun, Jan 6, 2019 at 1:07 PM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Felix and Shane.

+1 for packaging except signing in Jenkins. That sounds perfect for the next releases (2.2.3, 2.3.3, 3.0.0).

BTW, thanks to Shane, the followings are recovered from today.

1. For all live branches (master to branch-2.2), Jenkins Packaging jobs are recovered to build and packaging. Only there exist signing errors.


2. Daily snapshot maven publishing is recovered. This is helpful for partial module testing.


Thank you all. Everything becomes healthier during 2.2.3 release.
This will reduce the burden of 2.3.3/3.0.0 releases. :D

Bests,
Dongjoon.


On Sun, Jan 6, 2019 at 11:42 AM Felix Cheung <[hidden email]> wrote:
Awesome Shane!

 

From: shane knapp <[hidden email]>
Sent: Sunday, January 6, 2019 11:38 AM
To: Felix Cheung
Cc: Dongjoon Hyun; Wenchen Fan; dev
Subject: Re: Spark Packaging Jenkins
 
noted.  i like the idea of building (but not signing) the release and will update the job(s) this week.

On Sun, Jan 6, 2019 at 11:22 AM Felix Cheung <[hidden email]> wrote:

Look for do-release-docker.sh script

 

From: Felix Cheung <[hidden email]>
Sent: Sunday, January 6, 2019 11:17 AM
To: Dongjoon Hyun; Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
The release process doc should have been updated on this - as mentioned we do not use Jenkins for release signing (take this offline if further discussion is needed)

The release build on Jenkins can still be useful for pre-validating the release build process (without actually signing it)

 

From: Dongjoon Hyun <[hidden email]>
Sent: Saturday, January 5, 2019 9:46 PM
To: Wenchen Fan
Cc: dev; shane knapp
Subject: Re: Spark Packaging Jenkins
 
Thank you, Wenchen.

I see. I'll update the doc and proceed to the next step manually as you advise. And it seems that we can stop the outdated Jenkins jobs, too.

Bests,
Dongjoon.

On Sat, Jan 5, 2019 at 20:15 Wenchen Fan <[hidden email]> wrote:
IIRC there was a change to the release process: we stop using the shared gpg key on Jenkins, but use the personal key of the release manager. I'm not sure Jenkins can help testing package anymore.

BTW release manager needs to run the packaging script by himself. If there is a problem, the release manager will find it out sooner or later.



On Sun, Jan 6, 2019 at 6:34 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All.

It turns out that `gpg signing` is the next huddle in Spark Packaging Jenkins.
Since 2.4.0 release, is there something changed in our Jenkins machine?

      gpg: skipped "/home/jenkins/workspace/spark-master-package/spark-utils/new-release-scripts/jenkins/jenkins-credentials-JEtz0nyn/gpg.tmp": No secret key
      gpg: signing failed: No secret key

Bests,
Dongjoon.


On Fri, Jan 4, 2019 at 11:52 AM shane knapp <[hidden email]> wrote:

On Fri, Jan 4, 2019 at 11:31 AM shane knapp <[hidden email]> wrote:
this may push in to early next week...  these builds were set up before my time, and i'm currently unraveling how they all work before pushing a commit to fix stuff.

nothing like some code archaeology to make my friday more exciting!  :)

shane

On Fri, Jan 4, 2019 at 11:08 AM Dongjoon Hyun <[hidden email]> wrote:
Thank you, Shane!

Bests,
Dongjoon.

On Fri, Jan 4, 2019 at 10:50 AM shane knapp <[hidden email]> wrote:
yeah, i'll get on that today.  thanks for the heads up.

On Fri, Jan 4, 2019 at 10:46 AM Dongjoon Hyun <[hidden email]> wrote:
Hi, All

As a part of release process, we need to check Packaging/Compile/Test Jenkins status.



Currently, (2) and (3) are working because it uses GitHub (https://github.com/apache/spark.git).
But, (1) seems to be broken because it's looking for old repo(https://git-wip-us.apache.org/repos/asf/spark.git/info/refs) instead of new GitBox.

Can we fix this in this week?

Bests,
Dongjoon.



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: Spark Packaging Jenkins

shane knapp
2) create world-wide campaign to stop bash scripting

3) re-create these builds w/o calling release-build.sh at all (preferred)