[discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
hey everyone...  i wanted to break this discussion out of the mega-threads for the 2.4.1 RC candidates.

the TL;DR is that we've been trying to update the k8s client libs to something much more modern.  however, for us to do this, we need to update our very old k8s and minikube versions.

the problem here lies in the fact that if we update the client libs on master, but not the 2.4 branch, then the 2.4 branch k8s integration tests will fail if we update our backend minikube/k8s versions.

i've done all of the testing locally for the new k8s client libs, and am ready to pull the trigger on the infrastructure upgrade (which will take all of ~15 mins).

for this to happen, two PRs will need to be merged...  one for 2.4.1 and one for master.

is there a chance that we can get https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 release?  this will also require https://github.com/apache/spark/pull/24002 (for master) to be merged simultaneously.

both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm not entirely sure why).

here's the primary jira we're using to track this upgrade:

thanks in advance,

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

Marcelo Vanzin-2
Do the upgraded minikube/k8s versions break the current master client
version too?

I'm not super concerned about 2.4 integration tests being broken for a
little bit. It's very uncommon for new PRs to be open against
branch-2.4 that would affect k8s.

But I really don't want master to break. So if we can upgrade minikube
first, even if that breaks k8s integration tests on branch-2.4 for a
little bit, that would be optimal IMO.

On Wed, Mar 13, 2019 at 11:26 AM shane knapp <[hidden email]> wrote:

>
> hey everyone...  i wanted to break this discussion out of the mega-threads for the 2.4.1 RC candidates.
>
> the TL;DR is that we've been trying to update the k8s client libs to something much more modern.  however, for us to do this, we need to update our very old k8s and minikube versions.
>
> the problem here lies in the fact that if we update the client libs on master, but not the 2.4 branch, then the 2.4 branch k8s integration tests will fail if we update our backend minikube/k8s versions.
>
> i've done all of the testing locally for the new k8s client libs, and am ready to pull the trigger on the infrastructure upgrade (which will take all of ~15 mins).
>
> for this to happen, two PRs will need to be merged...  one for 2.4.1 and one for master.
>
> is there a chance that we can get https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 release?  this will also require https://github.com/apache/spark/pull/24002 (for master) to be merged simultaneously.
>
> both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm not entirely sure why).
>
> here's the primary jira we're using to track this upgrade:
> https://issues.apache.org/jira/browse/SPARK-26742
>
> thanks in advance,
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
Do the upgraded minikube/k8s versions break the current master client
version too?

yes.
 
I'm not super concerned about 2.4 integration tests being broken for a
little bit. It's very uncommon for new PRs to be open against
branch-2.4 that would affect k8s.

ok.  if the 2.4.1 PR is merged at about the same time as the one for master, we won't have to worry about the 2.4.x tests failing.

But I really don't want master to break. So if we can upgrade minikube
first, even if that breaks k8s integration tests on branch-2.4 for a
little bit, that would be optimal IMO.

i have everything staged on the affected jenkins workers and can do the infra upgrade really quickly...

shane
 
On Wed, Mar 13, 2019 at 11:26 AM shane knapp <[hidden email]> wrote:
>
> hey everyone...  i wanted to break this discussion out of the mega-threads for the 2.4.1 RC candidates.
>
> the TL;DR is that we've been trying to update the k8s client libs to something much more modern.  however, for us to do this, we need to update our very old k8s and minikube versions.
>
> the problem here lies in the fact that if we update the client libs on master, but not the 2.4 branch, then the 2.4 branch k8s integration tests will fail if we update our backend minikube/k8s versions.
>
> i've done all of the testing locally for the new k8s client libs, and am ready to pull the trigger on the infrastructure upgrade (which will take all of ~15 mins).
>
> for this to happen, two PRs will need to be merged...  one for 2.4.1 and one for master.
>
> is there a chance that we can get https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 release?  this will also require https://github.com/apache/spark/pull/24002 (for master) to be merged simultaneously.
>
> both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm not entirely sure why).
>
> here's the primary jira we're using to track this upgrade:
> https://issues.apache.org/jira/browse/SPARK-26742
>
> thanks in advance,
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

Sean Owen-2
In reply to this post by Marcelo Vanzin-2
I'm OK with this take. The problem with back-porting the client update
to 2.4.x at all is that it drops support for some old-but-not-that-old
K8S versions, which feels surprising in a maintenance release. That
said, maybe it's OK, and a little more OK for a 2.4.2 in several
months' time.

On Wed, Mar 13, 2019 at 1:49 PM Marcelo Vanzin <[hidden email]> wrote:

>
> Do the upgraded minikube/k8s versions break the current master client
> version too?
>
> I'm not super concerned about 2.4 integration tests being broken for a
> little bit. It's very uncommon for new PRs to be open against
> branch-2.4 that would affect k8s.
>
> But I really don't want master to break. So if we can upgrade minikube
> first, even if that breaks k8s integration tests on branch-2.4 for a
> little bit, that would be optimal IMO.
>
> On Wed, Mar 13, 2019 at 11:26 AM shane knapp <[hidden email]> wrote:
> >
> > hey everyone...  i wanted to break this discussion out of the mega-threads for the 2.4.1 RC candidates.
> >
> > the TL;DR is that we've been trying to update the k8s client libs to something much more modern.  however, for us to do this, we need to update our very old k8s and minikube versions.
> >
> > the problem here lies in the fact that if we update the client libs on master, but not the 2.4 branch, then the 2.4 branch k8s integration tests will fail if we update our backend minikube/k8s versions.
> >
> > i've done all of the testing locally for the new k8s client libs, and am ready to pull the trigger on the infrastructure upgrade (which will take all of ~15 mins).
> >
> > for this to happen, two PRs will need to be merged...  one for 2.4.1 and one for master.
> >
> > is there a chance that we can get https://github.com/apache/spark/pull/23993 merged in for the 2.4.1 release?  this will also require https://github.com/apache/spark/pull/24002 (for master) to be merged simultaneously.
> >
> > both of those PRs are ready to go (tho 23993 was closed w/o merge and i'm not entirely sure why).
> >
> > here's the primary jira we're using to track this upgrade:
> > https://issues.apache.org/jira/browse/SPARK-26742
> >
> > thanks in advance,
> >
> > shane
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

Marcelo Vanzin-2
In reply to this post by shane knapp
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
ok awesome.  let's shoot for 3pm PST.

On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

Marcelo Vanzin-2
Sounds good.

On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:

>
> ok awesome.  let's shoot for 3pm PST.
>
> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>> >>
>> >> Do the upgraded minikube/k8s versions break the current master client
>> >> version too?
>> >>
>> > yes.
>>
>> Ah, so that part kinda sucks.
>>
>> Let's do this: since the master PR is good to go pending the minikube
>> upgrade, let's try to synchronize things. Set a time to do the
>> minikube upgrade this PM, if that works for you, and I'll merge that
>> PR once it's done. Then I'll take care of backporting it to 2.4 and
>> make sure it passes the integration tests.
>>
>> --
>> Marcelo
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
In reply to this post by shane knapp
okie dokie!  the time approacheth!

i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.

On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
ok awesome.  let's shoot for 3pm PST.

On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.

On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
okie dokie!  the time approacheth!

i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.

On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
ok awesome.  let's shoot for 3pm PST.

On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp

On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.

On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
okie dokie!  the time approacheth!

i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.

On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
ok awesome.  let's shoot for 3pm PST.

On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp

On Wed, Mar 13, 2019 at 3:24 PM shane knapp <[hidden email]> wrote:

On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.

On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
okie dokie!  the time approacheth!

i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.

On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
ok awesome.  let's shoot for 3pm PST.

On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
btw, let's wait and see if the non-k8s PRB tests pass before merging https://github.com/apache/spark/pull/23993 in to 2.4.1

On Wed, Mar 13, 2019 at 3:42 PM shane knapp <[hidden email]> wrote:

On Wed, Mar 13, 2019 at 3:24 PM shane knapp <[hidden email]> wrote:

On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.

On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
okie dokie!  the time approacheth!

i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.

On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
ok awesome.  let's shoot for 3pm PST.

On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
thanks everyone, both PRs are merged.  :)

On Wed, Mar 13, 2019 at 3:51 PM shane knapp <[hidden email]> wrote:
btw, let's wait and see if the non-k8s PRB tests pass before merging https://github.com/apache/spark/pull/23993 in to 2.4.1

On Wed, Mar 13, 2019 at 3:42 PM shane knapp <[hidden email]> wrote:

On Wed, Mar 13, 2019 at 3:24 PM shane knapp <[hidden email]> wrote:

On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.

On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
okie dokie!  the time approacheth!

i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.

On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
ok awesome.  let's shoot for 3pm PST.

On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>
>> Do the upgraded minikube/k8s versions break the current master client
>> version too?
>>
> yes.

Ah, so that part kinda sucks.

Let's do this: since the master PR is good to go pending the minikube
upgrade, let's try to synchronize things. Set a time to do the
minikube upgrade this PM, if that works for you, and I'll merge that
PR once it's done. Then I'll take care of backporting it to 2.4 and
make sure it passes the integration tests.

--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

DB Tsai-6
Since rc8 was already cut without the k8s client upgrade; the build is
ready to vote, and including k8s client upgrade in 2.4.1 implies that
we will drop the old-but-not-that-old
K8S versions as Sean mentioned, should we include this upgrade in 2.4.2?

Thanks.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1

On Thu, Mar 14, 2019 at 9:48 AM shane knapp <[hidden email]> wrote:

>
> thanks everyone, both PRs are merged.  :)
>
> On Wed, Mar 13, 2019 at 3:51 PM shane knapp <[hidden email]> wrote:
>>
>> btw, let's wait and see if the non-k8s PRB tests pass before merging https://github.com/apache/spark/pull/23993 in to 2.4.1
>>
>> On Wed, Mar 13, 2019 at 3:42 PM shane knapp <[hidden email]> wrote:
>>>
>>> 2.4.1 k8s integration test passed:
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/
>>>
>>> thanks everyone!  :)
>>>
>>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp <[hidden email]> wrote:
>>>>
>>>> 2.4.1 integration tests running:  https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/
>>>>
>>>> On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
>>>>>
>>>>> upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.
>>>>>
>>>>> On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
>>>>>>
>>>>>> okie dokie!  the time approacheth!
>>>>>>
>>>>>> i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.
>>>>>>
>>>>>> On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
>>>>>>>
>>>>>>> ok awesome.  let's shoot for 3pm PST.
>>>>>>>
>>>>>>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
>>>>>>>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>>>>>>> >>
>>>>>>>> >> Do the upgraded minikube/k8s versions break the current master client
>>>>>>>> >> version too?
>>>>>>>> >>
>>>>>>>> > yes.
>>>>>>>>
>>>>>>>> Ah, so that part kinda sucks.
>>>>>>>>
>>>>>>>> Let's do this: since the master PR is good to go pending the minikube
>>>>>>>> upgrade, let's try to synchronize things. Set a time to do the
>>>>>>>> minikube upgrade this PM, if that works for you, and I'll merge that
>>>>>>>> PR once it's done. Then I'll take care of backporting it to 2.4 and
>>>>>>>> make sure it passes the integration tests.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Marcelo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Shane Knapp
>>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>>>> https://rise.cs.berkeley.edu
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shane Knapp
>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>>> https://rise.cs.berkeley.edu
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shane Knapp
>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>> https://rise.cs.berkeley.edu
>>>>
>>>>
>>>>
>>>> --
>>>> Shane Knapp
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

Marcelo Vanzin-2
IMO we can vote on rc8; that upgrade should not block it. If it fails
for some other reason, then rc9 will have it.

Supported versions in k8s-land are really weird and don't match well
to our release schedule. There isn't a good solution to this at the
moment... good thing k8s support is still marked as experimental. :-)

On Thu, Mar 14, 2019 at 10:56 AM DB Tsai <[hidden email]> wrote:

>
> Since rc8 was already cut without the k8s client upgrade; the build is
> ready to vote, and including k8s client upgrade in 2.4.1 implies that
> we will drop the old-but-not-that-old
> K8S versions as Sean mentioned, should we include this upgrade in 2.4.2?
>
> Thanks.
>
> Sincerely,
>
> DB Tsai
> ----------------------------------------------------------
> Web: https://www.dbtsai.com
> PGP Key ID: 42E5B25A8F7A82C1
>
> On Thu, Mar 14, 2019 at 9:48 AM shane knapp <[hidden email]> wrote:
> >
> > thanks everyone, both PRs are merged.  :)
> >
> > On Wed, Mar 13, 2019 at 3:51 PM shane knapp <[hidden email]> wrote:
> >>
> >> btw, let's wait and see if the non-k8s PRB tests pass before merging https://github.com/apache/spark/pull/23993 in to 2.4.1
> >>
> >> On Wed, Mar 13, 2019 at 3:42 PM shane knapp <[hidden email]> wrote:
> >>>
> >>> 2.4.1 k8s integration test passed:
> >>>
> >>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/
> >>>
> >>> thanks everyone!  :)
> >>>
> >>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp <[hidden email]> wrote:
> >>>>
> >>>> 2.4.1 integration tests running:  https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/
> >>>>
> >>>> On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
> >>>>>
> >>>>> upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.
> >>>>>
> >>>>> On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
> >>>>>>
> >>>>>> okie dokie!  the time approacheth!
> >>>>>>
> >>>>>> i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.
> >>>>>>
> >>>>>> On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
> >>>>>>>
> >>>>>>> ok awesome.  let's shoot for 3pm PST.
> >>>>>>>
> >>>>>>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
> >>>>>>>>
> >>>>>>>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
> >>>>>>>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
> >>>>>>>> >>
> >>>>>>>> >> Do the upgraded minikube/k8s versions break the current master client
> >>>>>>>> >> version too?
> >>>>>>>> >>
> >>>>>>>> > yes.
> >>>>>>>>
> >>>>>>>> Ah, so that part kinda sucks.
> >>>>>>>>
> >>>>>>>> Let's do this: since the master PR is good to go pending the minikube
> >>>>>>>> upgrade, let's try to synchronize things. Set a time to do the
> >>>>>>>> minikube upgrade this PM, if that works for you, and I'll merge that
> >>>>>>>> PR once it's done. Then I'll take care of backporting it to 2.4 and
> >>>>>>>> make sure it passes the integration tests.
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Marcelo
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Shane Knapp
> >>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
> >>>>>>> https://rise.cs.berkeley.edu
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Shane Knapp
> >>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
> >>>>>> https://rise.cs.berkeley.edu
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Shane Knapp
> >>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
> >>>>> https://rise.cs.berkeley.edu
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Shane Knapp
> >>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
> >>>> https://rise.cs.berkeley.edu
> >>>
> >>>
> >>>
> >>> --
> >>> Shane Knapp
> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead
> >>> https://rise.cs.berkeley.edu
> >>
> >>
> >>
> >> --
> >> Shane Knapp
> >> UC Berkeley EECS Research / RISELab Staff Technical Lead
> >> https://rise.cs.berkeley.edu
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
In reply to this post by DB Tsai-6
i think we should.  the version of k8s we were testing against before was v1.10.x, and we are currently testing master *and* 2.4.1 against  v1.13.1.   given that we're a couple of weeks away from v.1.14.x i feel that this is a safe and prudent thing to include.

On Thu, Mar 14, 2019 at 10:57 AM DB Tsai <[hidden email]> wrote:
Since rc8 was already cut without the k8s client upgrade; the build is
ready to vote, and including k8s client upgrade in 2.4.1 implies that
we will drop the old-but-not-that-old
K8S versions as Sean mentioned, should we include this upgrade in 2.4.2?

Thanks.

Sincerely,

DB Tsai
----------------------------------------------------------
Web: https://www.dbtsai.com
PGP Key ID: 42E5B25A8F7A82C1

On Thu, Mar 14, 2019 at 9:48 AM shane knapp <[hidden email]> wrote:
>
> thanks everyone, both PRs are merged.  :)
>
> On Wed, Mar 13, 2019 at 3:51 PM shane knapp <[hidden email]> wrote:
>>
>> btw, let's wait and see if the non-k8s PRB tests pass before merging https://github.com/apache/spark/pull/23993 in to 2.4.1
>>
>> On Wed, Mar 13, 2019 at 3:42 PM shane knapp <[hidden email]> wrote:
>>>
>>> 2.4.1 k8s integration test passed:
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/
>>>
>>> thanks everyone!  :)
>>>
>>> On Wed, Mar 13, 2019 at 3:24 PM shane knapp <[hidden email]> wrote:
>>>>
>>>> 2.4.1 integration tests running:  https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/
>>>>
>>>> On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
>>>>>
>>>>> upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.
>>>>>
>>>>> On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
>>>>>>
>>>>>> okie dokie!  the time approacheth!
>>>>>>
>>>>>> i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.
>>>>>>
>>>>>> On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
>>>>>>>
>>>>>>> ok awesome.  let's shoot for 3pm PST.
>>>>>>>
>>>>>>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
>>>>>>>>
>>>>>>>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
>>>>>>>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>>>>>>> >>
>>>>>>>> >> Do the upgraded minikube/k8s versions break the current master client
>>>>>>>> >> version too?
>>>>>>>> >>
>>>>>>>> > yes.
>>>>>>>>
>>>>>>>> Ah, so that part kinda sucks.
>>>>>>>>
>>>>>>>> Let's do this: since the master PR is good to go pending the minikube
>>>>>>>> upgrade, let's try to synchronize things. Set a time to do the
>>>>>>>> minikube upgrade this PM, if that works for you, and I'll merge that
>>>>>>>> PR once it's done. Then I'll take care of backporting it to 2.4 and
>>>>>>>> make sure it passes the integration tests.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Marcelo
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Shane Knapp
>>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>>>> https://rise.cs.berkeley.edu
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Shane Knapp
>>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>>> https://rise.cs.berkeley.edu
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Shane Knapp
>>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>>> https://rise.cs.berkeley.edu
>>>>
>>>>
>>>>
>>>> --
>>>> Shane Knapp
>>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>>> https://rise.cs.berkeley.edu
>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

Marcelo Vanzin-2
In reply to this post by shane knapp
Not sure if anyone else is having the same issues, but jenkins seems
to be in a weird state.

I can't connect to the web UI, and it doesn't seem to be responding to
test requests.

On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:

>
> upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.
>
> On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
>>
>> okie dokie!  the time approacheth!
>>
>> i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.
>>
>> On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
>>>
>>> ok awesome.  let's shoot for 3pm PST.
>>>
>>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
>>>>
>>>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
>>>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>>> >>
>>>> >> Do the upgraded minikube/k8s versions break the current master client
>>>> >> version too?
>>>> >>
>>>> > yes.
>>>>
>>>> Ah, so that part kinda sucks.
>>>>
>>>> Let's do this: since the master PR is good to go pending the minikube
>>>> upgrade, let's try to synchronize things. Set a time to do the
>>>> minikube upgrade this PM, if that works for you, and I'll merge that
>>>> PR once it's done. Then I'll take care of backporting it to 2.4 and
>>>> make sure it passes the integration tests.
>>>>
>>>> --
>>>> Marcelo
>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [discuss] 2.4.1-rcX release, k8s client PRs, build system infrastructure update

shane knapp
about to send mail.

On Thu, Mar 14, 2019 at 1:00 PM Marcelo Vanzin <[hidden email]> wrote:
Not sure if anyone else is having the same issues, but jenkins seems
to be in a weird state.

I can't connect to the web UI, and it doesn't seem to be responding to
test requests.

On Wed, Mar 13, 2019 at 3:15 PM shane knapp <[hidden email]> wrote:
>
> upgrade completed, jenkins building again...  master PR merged, waiting for the 2.4.1 PR to launch the k8s integration tests.
>
> On Wed, Mar 13, 2019 at 2:55 PM shane knapp <[hidden email]> wrote:
>>
>> okie dokie!  the time approacheth!
>>
>> i'll pause jenkins @ 3pm to not accept new jobs.  i don't expect the upgrade to take more than 15-20 mins, following which i will re-enable builds.
>>
>> On Wed, Mar 13, 2019 at 12:17 PM shane knapp <[hidden email]> wrote:
>>>
>>> ok awesome.  let's shoot for 3pm PST.
>>>
>>> On Wed, Mar 13, 2019 at 11:59 AM Marcelo Vanzin <[hidden email]> wrote:
>>>>
>>>> On Wed, Mar 13, 2019 at 11:53 AM shane knapp <[hidden email]> wrote:
>>>> > On Wed, Mar 13, 2019 at 11:49 AM Marcelo Vanzin <[hidden email]> wrote:
>>>> >>
>>>> >> Do the upgraded minikube/k8s versions break the current master client
>>>> >> version too?
>>>> >>
>>>> > yes.
>>>>
>>>> Ah, so that part kinda sucks.
>>>>
>>>> Let's do this: since the master PR is good to go pending the minikube
>>>> upgrade, let's try to synchronize things. Set a time to do the
>>>> minikube upgrade this PM, if that works for you, and I'll merge that
>>>> PR once it's done. Then I'll take care of backporting it to 2.4 and
>>>> make sure it passes the integration tests.
>>>>
>>>> --
>>>> Marcelo
>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Marcelo


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead