[DISCUSS] SparkR support on k8s back-end for Spark 2.4

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] SparkR support on k8s back-end for Spark 2.4

Erik Erlandson-2
The SparkR support PR is finished, along with integration testing, however Shane has requested that the integration testing not be enabled until after the 2.4 release because it requires the OS updates he wants to test *after* the release.

The integration testing can be run locally, and so the question at hand is: would the PMC be willing to consider inclusion of the SparkR for 2.4, based on local verification of the testing? The PySpark PR was merged under similar circumstances: the testing was verified locally and the PR was merged before the testing was enabled for jenkins.

Cheers,
Erik
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

ifilonenko
The SparkR support PR includes integration testing that can be tested on a local Minikube instance by merely running the distribution with appropriate flags (--r) and running the integration-tests similarly to as you would on any k8s test. Maybe some others could locally test this, if there is any hesitation. I, for one, don't see a problem with merging as is and having the Jenkins build be configured for R distribution support via the Ubuntu update when Shane gets to it, similar to as we did for PySpark. The PR can be found here

Furthermore, I am wondering if there is any desire to ensure SparkR support (for k8s) is merged by 2.4 by the community, and if so, then maybe merging this without the Jenkins build seems even more appropriate. 

Best,
Ilan Filonenko

On Wed, Aug 15, 2018 at 3:33 PM, Erik Erlandson <[hidden email]> wrote:
The SparkR support PR is finished, along with integration testing, however Shane has requested that the integration testing not be enabled until after the 2.4 release because it requires the OS updates he wants to test *after* the release.

The integration testing can be run locally, and so the question at hand is: would the PMC be willing to consider inclusion of the SparkR for 2.4, based on local verification of the testing? The PySpark PR was merged under similar circumstances: the testing was verified locally and the PR was merged before the testing was enabled for jenkins.

Cheers,
Erik

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

rxin
In reply to this post by Erik Erlandson-2
What's the reason we don't want to do the OS updates right now? Is it due to the unpredictability of potential issues that might happen and end up delaying 2.4 release?


On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson <[hidden email]> wrote:
The SparkR support PR is finished, along with integration testing, however Shane has requested that the integration testing not be enabled until after the 2.4 release because it requires the OS updates he wants to test *after* the release.

The integration testing can be run locally, and so the question at hand is: would the PMC be willing to consider inclusion of the SparkR for 2.4, based on local verification of the testing? The PySpark PR was merged under similar circumstances: the testing was verified locally and the PR was merged before the testing was enabled for jenkins.

Cheers,
Erik
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

rxin
Personally I'd love for R support to be in 2.4, but I don't consider something "Done" unless tests are running ... Is the proposal: the release manager manually run the R tests when preparing the release, and switch over to fully integrated Jenkins after 2.4.0 is released?

On Wed, Aug 15, 2018 at 2:45 PM Reynold Xin <[hidden email]> wrote:
What's the reason we don't want to do the OS updates right now? Is it due to the unpredictability of potential issues that might happen and end up delaying 2.4 release?


On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson <[hidden email]> wrote:
The SparkR support PR is finished, along with integration testing, however Shane has requested that the integration testing not be enabled until after the 2.4 release because it requires the OS updates he wants to test *after* the release.

The integration testing can be run locally, and so the question at hand is: would the PMC be willing to consider inclusion of the SparkR for 2.4, based on local verification of the testing? The PySpark PR was merged under similar circumstances: the testing was verified locally and the PR was merged before the testing was enabled for jenkins.

Cheers,
Erik
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

ifilonenko
Correct, the OS change and updates would require more testing, from what Shane has told me, and could potentially surface some issue that could delay a major release. 

So yes, the release manager would need to run the tests manually and after the release we would switch to a fully integrated Jenkins that would run those same tests on the newly updated workers. 

On Wed, Aug 15, 2018 at 3:47 PM, Reynold Xin <[hidden email]> wrote:
Personally I'd love for R support to be in 2.4, but I don't consider something "Done" unless tests are running ... Is the proposal: the release manager manually run the R tests when preparing the release, and switch over to fully integrated Jenkins after 2.4.0 is released?

On Wed, Aug 15, 2018 at 2:45 PM Reynold Xin <[hidden email]> wrote:
What's the reason we don't want to do the OS updates right now? Is it due to the unpredictability of potential issues that might happen and end up delaying 2.4 release?


On Wed, Aug 15, 2018 at 2:33 PM Erik Erlandson <[hidden email]> wrote:
The SparkR support PR is finished, along with integration testing, however Shane has requested that the integration testing not be enabled until after the 2.4 release because it requires the OS updates he wants to test *after* the release.

The integration testing can be run locally, and so the question at hand is: would the PMC be willing to consider inclusion of the SparkR for 2.4, based on local verification of the testing? The PySpark PR was merged under similar circumstances: the testing was verified locally and the PR was merged before the testing was enabled for jenkins.

Cheers,
Erik

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

shane knapp
In reply to this post by rxin
On Wed, Aug 15, 2018 at 12:45 PM, Reynold Xin <[hidden email]> wrote:
What's the reason we don't want to do the OS updates right now? Is it due to the unpredictability of potential issues that might happen and end up delaying 2.4 release?

that is exactly it...  i haven't had a chance to test everything (esp docs, release builds, etc), and since there have been many changes pushed to various branches, some builds are failing in different ways on centos vs ubuntu.  things are very, very fluid right now.

we are SUPER close to being able to move the vast majority of builds to ubuntu, but even after than happens there will almost definitely be a non-zero amount of babysitting and debugging that needs to happen.  all of that babysitting and debugging would potentially delay the 2.4 cut even more.

in fact, i don't see us getting rid of all of the centos machines until EOY (see my above comment, re docs, release etc).  these are the builds that will remain on centos for the near future:

these all run on amp-jenkins-worker-01, which will remain untouched.

HOWEVER:

i'm 99%+ certain that i can port the PRB builder over to ubuntu w/a pared down python 3.5 installation.  i have a build running now (https://rise.cs.berkeley.edu/jenkins/view/RISELab%20Infra/job/ubuntuSparkPRB/74/) and if it behaves appropriately (meaning testing all the right things w/py35), i should be able to get two more non-critical centos build nodes reinstalled w/ubuntu and deployed by EOW.  this will give us enough jenkins executors and allow PRB throughput to not decrease.

if the dev community is ok w/taking this plunge, i will make it happen.

shane (who wants everyone to remember that it's just little old me running this...  not a team of people)  ;)
-- 
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

Marcelo Vanzin-2
On Wed, Aug 15, 2018 at 1:35 PM, shane knapp <[hidden email]> wrote:
> in fact, i don't see us getting rid of all of the centos machines until EOY
> (see my above comment, re docs, release etc).  these are the builds that
> will remain on centos for the near future:
> https://rise.cs.berkeley.edu/jenkins/label/spark-release/
> https://rise.cs.berkeley.edu/jenkins/label/spark-packaging/
> https://rise.cs.berkeley.edu/jenkins/label/spark-docs/

What is the current purpose of these builds?

- spark-release hasn't been triggered in a long time.
- spark-packaging seems to have been failing miserably for the last 10 months.
- spark-docs seems to be building the docs, is that the only place
where the docs build is tested?

In the last many releases we've moved away from using jenkins jobs for
preparing the packages, and the scripts have changed a lot to be
friendlier to people running them locally (they even support docker
now, and have flags to run "test" builds that don't require
credentials such as GPG keys).

Perhaps we should think about revamping these jobs instead of keeping
them as is.

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

shane knapp

What is the current purpose of these builds?

to be honest, i have absolutely no idea.  :)

these were set up a long time ago, in a galaxy far far away, by someone who is not me.
 
- spark-docs seems to be building the docs, is that the only place
where the docs build is tested?

i think so...
 
In the last many releases we've moved away from using jenkins jobs for
preparing the packages, and the scripts have changed a lot to be
friendlier to people running them locally (they even support docker
now, and have flags to run "test" builds that don't require
credentials such as GPG keys).

awesome++
 
Perhaps we should think about revamping these jobs instead of keeping
them as is.

i fully support this.  which is exactly why i punted on even trying to get them ported over to the ubuntu nodes. 

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

cloud0fan
I'm also happy to see we have R support on k8s for Spark 2.4. I'll do the manual testing for it if we don't want to upgrade the OS now. If the Python support is also merged in this way, I think we can merge the R support PR too?

On Thu, Aug 16, 2018 at 7:23 AM shane knapp <[hidden email]> wrote:

What is the current purpose of these builds?

to be honest, i have absolutely no idea.  :)

these were set up a long time ago, in a galaxy far far away, by someone who is not me.
 
- spark-docs seems to be building the docs, is that the only place
where the docs build is tested?

i think so...
 
In the last many releases we've moved away from using jenkins jobs for
preparing the packages, and the scripts have changed a lot to be
friendlier to people running them locally (they even support docker
now, and have flags to run "test" builds that don't require
credentials such as GPG keys).

awesome++
 
Perhaps we should think about revamping these jobs instead of keeping
them as is.

i fully support this.  which is exactly why i punted on even trying to get them ported over to the ubuntu nodes. 

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

Erik Erlandson-2
IMO sparkR support makes sense to merge for 2.4, as long as the release wranglers agree that local integration testing is sufficiently convincing. Part of the intent here is to allow this to happen without Shane having to reorganize his complex upgrade schedule and make it even more complicated.

On Wed, Aug 15, 2018 at 7:08 PM, Wenchen Fan <[hidden email]> wrote:
I'm also happy to see we have R support on k8s for Spark 2.4. I'll do the manual testing for it if we don't want to upgrade the OS now. If the Python support is also merged in this way, I think we can merge the R support PR too?

On Thu, Aug 16, 2018 at 7:23 AM shane knapp <[hidden email]> wrote:

What is the current purpose of these builds?

to be honest, i have absolutely no idea.  :)

these were set up a long time ago, in a galaxy far far away, by someone who is not me.
 
- spark-docs seems to be building the docs, is that the only place
where the docs build is tested?

i think so...
 
In the last many releases we've moved away from using jenkins jobs for
preparing the packages, and the scripts have changed a lot to be
friendlier to people running them locally (they even support docker
now, and have flags to run "test" builds that don't require
credentials such as GPG keys).

awesome++
 
Perhaps we should think about revamping these jobs instead of keeping
them as is.

i fully support this.  which is exactly why i punted on even trying to get them ported over to the ubuntu nodes. 

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

shane knapp
On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson <[hidden email]> wrote:
IMO sparkR support makes sense to merge for 2.4, as long as the release wranglers agree that local integration testing is sufficiently convincing.

i agree w/this.  testing for this stuff specifically will happen within a couple of weeks after the 2.4 cut.
 
Part of the intent here is to allow this to happen without Shane having to reorganize his complex upgrade schedule and make it even more complicated.

this.  exactly.  :) 

--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

ifilonenko
Okay, if there is a consensus in merging the SparkR feature and its respective tests separately, I will split up the current PR to allow for the feature to be merged while the e2e tests to be used for locally testing until we are ready to merge (similar to PySpark). Will give it another day to hear other opinions, but otherwise working under the impression that the OS upgrade will not happen before the 2.4 cut.



On Thu, Aug 16, 2018 at 12:53 PM, shane knapp <[hidden email]> wrote:
On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson <[hidden email]> wrote:
IMO sparkR support makes sense to merge for 2.4, as long as the release wranglers agree that local integration testing is sufficiently convincing.

i agree w/this.  testing for this stuff specifically will happen within a couple of weeks after the 2.4 cut.
 
Part of the intent here is to allow this to happen without Shane having to reorganize his complex upgrade schedule and make it even more complicated.

this.  exactly.  :) 

--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4

Felix Cheung
I fully support merging that SparkR support on k8s.

If Ilan and other are willing to manually validate the RC I’m happy to voucher for it (I’m not 100% sure i have capacity to test it that way but certainly will try)

Also +1 on revisiting Jenkins builds. tbh I’m not sure we depend on them too much during the release process due to reason Marcelo listed, for the last 4+ releases...

 

From: Ilan Filonenko <[hidden email]>
Sent: Thursday, August 16, 2018 10:27 AM
To: shane knapp
Cc: Erik Erlandson; Wenchen Fan; Marcelo Vanzin; Reynold?Xin; Spark dev list
Subject: Re: [DISCUSS] SparkR support on k8s back-end for Spark 2.4
 
Okay, if there is a consensus in merging the SparkR feature and its respective tests separately, I will split up the current PR to allow for the feature to be merged while the e2e tests to be used for locally testing until we are ready to merge (similar to PySpark). Will give it another day to hear other opinions, but otherwise working under the impression that the OS upgrade will not happen before the 2.4 cut.



On Thu, Aug 16, 2018 at 12:53 PM, shane knapp <[hidden email]> wrote:
On Thu, Aug 16, 2018 at 9:49 AM, Erik Erlandson <[hidden email]> wrote:
IMO sparkR support makes sense to merge for 2.4, as long as the release wranglers agree that local integration testing is sufficiently convincing.

i agree w/this.  testing for this stuff specifically will happen within a couple of weeks after the 2.4 cut.
 
Part of the intent here is to allow this to happen without Shane having to reorganize his complex upgrade schedule and make it even more complicated.

this.  exactly.  :) 

--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead