[build system] our colo is having power issues again. there will be a few 'events' this week

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[build system] our colo is having power issues again. there will be a few 'events' this week

shane knapp
the main transformer for our colo is experiencing major issues, and
campus be performing emergency work on it starting tomorrow morning
(tuesday sept 24, 9am PDT).  it's pretty dire.  :(

there's a lot going on, but please expect some sporadic jenkins
downtime until monday.  here's the abbreviated list of when we can
expect things to happen:

* tomorrow @ 9am we are switching to generator power.  there will be
temporary loss of power to the jenkins workers, and if any don't come
back after the switch i will need to head down there and manually
power them back on.

* thursday @ 9am, switch back to campus power.  again, i will deal
w/any workers that don't come back manually.

* saturday @ 9am, back to generator power.  same deal as above.

* saturday or sunday the repairs will be completed.  i will monitor
the situation, and deal w/any problem servers on my way in to work on
monday.

i apologize for any inconvenience that this will cause...  and i hope
that this is the final piece of the power puzzle that our colo team
has to deal with.

thanks in advance,

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

shane knapp
power switchover is happening now.  more updates to come once machines
come back up.

On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:

>
> the main transformer for our colo is experiencing major issues, and
> campus be performing emergency work on it starting tomorrow morning
> (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
>
> there's a lot going on, but please expect some sporadic jenkins
> downtime until monday.  here's the abbreviated list of when we can
> expect things to happen:
>
> * tomorrow @ 9am we are switching to generator power.  there will be
> temporary loss of power to the jenkins workers, and if any don't come
> back after the switch i will need to head down there and manually
> power them back on.
>
> * thursday @ 9am, switch back to campus power.  again, i will deal
> w/any workers that don't come back manually.
>
> * saturday @ 9am, back to generator power.  same deal as above.
>
> * saturday or sunday the repairs will be completed.  i will monitor
> the situation, and deal w/any problem servers on my way in to work on
> monday.
>
> i apologize for any inconvenience that this will cause...  and i hope
> that this is the final piece of the power puzzle that our colo team
> has to deal with.
>
> thanks in advance,
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

shane knapp
aaaaaand that was quick!  everything is back up and building

On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp <[hidden email]> wrote:

>
> power switchover is happening now.  more updates to come once machines
> come back up.
>
> On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:
> >
> > the main transformer for our colo is experiencing major issues, and
> > campus be performing emergency work on it starting tomorrow morning
> > (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
> >
> > there's a lot going on, but please expect some sporadic jenkins
> > downtime until monday.  here's the abbreviated list of when we can
> > expect things to happen:
> >
> > * tomorrow @ 9am we are switching to generator power.  there will be
> > temporary loss of power to the jenkins workers, and if any don't come
> > back after the switch i will need to head down there and manually
> > power them back on.
> >
> > * thursday @ 9am, switch back to campus power.  again, i will deal
> > w/any workers that don't come back manually.
> >
> > * saturday @ 9am, back to generator power.  same deal as above.
> >
> > * saturday or sunday the repairs will be completed.  i will monitor
> > the situation, and deal w/any problem servers on my way in to work on
> > monday.
> >
> > i apologize for any inconvenience that this will cause...  and i hope
> > that this is the final piece of the power puzzle that our colo team
> > has to deal with.
> >
> > thanks in advance,
> >
> > shane
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

shane knapp
quick update from our colo admin:  they are going to keep the colo on
generator power until monday morning and not switch back and forth.
this is great as we'll have solid uptime until monday morning when we
go back to grid power.

if this changes i will be certain to update everyone here.

thanks,

shane

On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp <[hidden email]> wrote:

>
> aaaaaand that was quick!  everything is back up and building
>
> On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp <[hidden email]> wrote:
> >
> > power switchover is happening now.  more updates to come once machines
> > come back up.
> >
> > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:
> > >
> > > the main transformer for our colo is experiencing major issues, and
> > > campus be performing emergency work on it starting tomorrow morning
> > > (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
> > >
> > > there's a lot going on, but please expect some sporadic jenkins
> > > downtime until monday.  here's the abbreviated list of when we can
> > > expect things to happen:
> > >
> > > * tomorrow @ 9am we are switching to generator power.  there will be
> > > temporary loss of power to the jenkins workers, and if any don't come
> > > back after the switch i will need to head down there and manually
> > > power them back on.
> > >
> > > * thursday @ 9am, switch back to campus power.  again, i will deal
> > > w/any workers that don't come back manually.
> > >
> > > * saturday @ 9am, back to generator power.  same deal as above.
> > >
> > > * saturday or sunday the repairs will be completed.  i will monitor
> > > the situation, and deal w/any problem servers on my way in to work on
> > > monday.
> > >
> > > i apologize for any inconvenience that this will cause...  and i hope
> > > that this is the final piece of the power puzzle that our colo team
> > > has to deal with.
> > >
> > > thanks in advance,
> > >
> > > shane
> > > --
> > > Shane Knapp
> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > > https://rise.cs.berkeley.edu
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

Jungtaek Lim
Hi Shane,

Thanks for the update, and take care of build system!

Looks like some of build just got failed without test failures, looks like env. issue.

+ ./dev/run-tests-jenkins
Python versions prior to 2.7 are not supported.

Could you please check on this?

Thanks,
Jungtaek Lim (HeartSaVioR)


On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp <[hidden email]> wrote:
quick update from our colo admin:  they are going to keep the colo on
generator power until monday morning and not switch back and forth.
this is great as we'll have solid uptime until monday morning when we
go back to grid power.

if this changes i will be certain to update everyone here.

thanks,

shane

On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp <[hidden email]> wrote:
>
> aaaaaand that was quick!  everything is back up and building
>
> On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp <[hidden email]> wrote:
> >
> > power switchover is happening now.  more updates to come once machines
> > come back up.
> >
> > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:
> > >
> > > the main transformer for our colo is experiencing major issues, and
> > > campus be performing emergency work on it starting tomorrow morning
> > > (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
> > >
> > > there's a lot going on, but please expect some sporadic jenkins
> > > downtime until monday.  here's the abbreviated list of when we can
> > > expect things to happen:
> > >
> > > * tomorrow @ 9am we are switching to generator power.  there will be
> > > temporary loss of power to the jenkins workers, and if any don't come
> > > back after the switch i will need to head down there and manually
> > > power them back on.
> > >
> > > * thursday @ 9am, switch back to campus power.  again, i will deal
> > > w/any workers that don't come back manually.
> > >
> > > * saturday @ 9am, back to generator power.  same deal as above.
> > >
> > > * saturday or sunday the repairs will be completed.  i will monitor
> > > the situation, and deal w/any problem servers on my way in to work on
> > > monday.
> > >
> > > i apologize for any inconvenience that this will cause...  and i hope
> > > that this is the final piece of the power puzzle that our colo team
> > > has to deal with.
> > >
> > > thanks in advance,
> > >
> > > shane
> > > --
> > > Shane Knapp
> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > > https://rise.cs.berkeley.edu
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

shane knapp
no worries.  since we deprecated the packaging builds i put that
worker back in to the rotation...  there was a slight env issue but
that's fixed and we should be g2g w/python.

On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim <[hidden email]> wrote:

>
> Hi Shane,
>
> Thanks for the update, and take care of build system!
>
> Looks like some of build just got failed without test failures, looks like env. issue.
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/
>
> + ./dev/run-tests-jenkins
> Python versions prior to 2.7 are not supported.
>
>
> Could you please check on this?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp <[hidden email]> wrote:
>>
>> quick update from our colo admin:  they are going to keep the colo on
>> generator power until monday morning and not switch back and forth.
>> this is great as we'll have solid uptime until monday morning when we
>> go back to grid power.
>>
>> if this changes i will be certain to update everyone here.
>>
>> thanks,
>>
>> shane
>>
>> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp <[hidden email]> wrote:
>> >
>> > aaaaaand that was quick!  everything is back up and building
>> >
>> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp <[hidden email]> wrote:
>> > >
>> > > power switchover is happening now.  more updates to come once machines
>> > > come back up.
>> > >
>> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:
>> > > >
>> > > > the main transformer for our colo is experiencing major issues, and
>> > > > campus be performing emergency work on it starting tomorrow morning
>> > > > (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
>> > > >
>> > > > there's a lot going on, but please expect some sporadic jenkins
>> > > > downtime until monday.  here's the abbreviated list of when we can
>> > > > expect things to happen:
>> > > >
>> > > > * tomorrow @ 9am we are switching to generator power.  there will be
>> > > > temporary loss of power to the jenkins workers, and if any don't come
>> > > > back after the switch i will need to head down there and manually
>> > > > power them back on.
>> > > >
>> > > > * thursday @ 9am, switch back to campus power.  again, i will deal
>> > > > w/any workers that don't come back manually.
>> > > >
>> > > > * saturday @ 9am, back to generator power.  same deal as above.
>> > > >
>> > > > * saturday or sunday the repairs will be completed.  i will monitor
>> > > > the situation, and deal w/any problem servers on my way in to work on
>> > > > monday.
>> > > >
>> > > > i apologize for any inconvenience that this will cause...  and i hope
>> > > > that this is the final piece of the power puzzle that our colo team
>> > > > has to deal with.
>> > > >
>> > > > thanks in advance,
>> > > >
>> > > > shane
>> > > > --
>> > > > Shane Knapp
>> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > > > https://rise.cs.berkeley.edu
>> > >
>> > >
>> > >
>> > > --
>> > > Shane Knapp
>> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > > https://rise.cs.berkeley.edu
>> >
>> >
>> >
>> > --
>> > Shane Knapp
>> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

Jungtaek Lim
Awesome, thanks for the quick update! 

On Wed, Sep 25, 2019 at 6:04 AM Shane Knapp <[hidden email]> wrote:
no worries.  since we deprecated the packaging builds i put that
worker back in to the rotation...  there was a slight env issue but
that's fixed and we should be g2g w/python.

On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim <[hidden email]> wrote:
>
> Hi Shane,
>
> Thanks for the update, and take care of build system!
>
> Looks like some of build just got failed without test failures, looks like env. issue.
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/
>
> + ./dev/run-tests-jenkins
> Python versions prior to 2.7 are not supported.
>
>
> Could you please check on this?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp <[hidden email]> wrote:
>>
>> quick update from our colo admin:  they are going to keep the colo on
>> generator power until monday morning and not switch back and forth.
>> this is great as we'll have solid uptime until monday morning when we
>> go back to grid power.
>>
>> if this changes i will be certain to update everyone here.
>>
>> thanks,
>>
>> shane
>>
>> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp <[hidden email]> wrote:
>> >
>> > aaaaaand that was quick!  everything is back up and building
>> >
>> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp <[hidden email]> wrote:
>> > >
>> > > power switchover is happening now.  more updates to come once machines
>> > > come back up.
>> > >
>> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:
>> > > >
>> > > > the main transformer for our colo is experiencing major issues, and
>> > > > campus be performing emergency work on it starting tomorrow morning
>> > > > (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
>> > > >
>> > > > there's a lot going on, but please expect some sporadic jenkins
>> > > > downtime until monday.  here's the abbreviated list of when we can
>> > > > expect things to happen:
>> > > >
>> > > > * tomorrow @ 9am we are switching to generator power.  there will be
>> > > > temporary loss of power to the jenkins workers, and if any don't come
>> > > > back after the switch i will need to head down there and manually
>> > > > power them back on.
>> > > >
>> > > > * thursday @ 9am, switch back to campus power.  again, i will deal
>> > > > w/any workers that don't come back manually.
>> > > >
>> > > > * saturday @ 9am, back to generator power.  same deal as above.
>> > > >
>> > > > * saturday or sunday the repairs will be completed.  i will monitor
>> > > > the situation, and deal w/any problem servers on my way in to work on
>> > > > monday.
>> > > >
>> > > > i apologize for any inconvenience that this will cause...  and i hope
>> > > > that this is the final piece of the power puzzle that our colo team
>> > > > has to deal with.
>> > > >
>> > > > thanks in advance,
>> > > >
>> > > > shane
>> > > > --
>> > > > Shane Knapp
>> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > > > https://rise.cs.berkeley.edu
>> > >
>> > >
>> > >
>> > > --
>> > > Shane Knapp
>> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > > https://rise.cs.berkeley.edu
>> >
>> >
>> >
>> > --
>> > Shane Knapp
>> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

Takeshi Yamamuro
Shane, thanks for the hard work!

Bests,
Takeshi

On Wed, Sep 25, 2019 at 6:07 AM Jungtaek Lim <[hidden email]> wrote:
Awesome, thanks for the quick update! 

On Wed, Sep 25, 2019 at 6:04 AM Shane Knapp <[hidden email]> wrote:
no worries.  since we deprecated the packaging builds i put that
worker back in to the rotation...  there was a slight env issue but
that's fixed and we should be g2g w/python.

On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim <[hidden email]> wrote:
>
> Hi Shane,
>
> Thanks for the update, and take care of build system!
>
> Looks like some of build just got failed without test failures, looks like env. issue.
> https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/
>
> + ./dev/run-tests-jenkins
> Python versions prior to 2.7 are not supported.
>
>
> Could you please check on this?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp <[hidden email]> wrote:
>>
>> quick update from our colo admin:  they are going to keep the colo on
>> generator power until monday morning and not switch back and forth.
>> this is great as we'll have solid uptime until monday morning when we
>> go back to grid power.
>>
>> if this changes i will be certain to update everyone here.
>>
>> thanks,
>>
>> shane
>>
>> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp <[hidden email]> wrote:
>> >
>> > aaaaaand that was quick!  everything is back up and building
>> >
>> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp <[hidden email]> wrote:
>> > >
>> > > power switchover is happening now.  more updates to come once machines
>> > > come back up.
>> > >
>> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:
>> > > >
>> > > > the main transformer for our colo is experiencing major issues, and
>> > > > campus be performing emergency work on it starting tomorrow morning
>> > > > (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
>> > > >
>> > > > there's a lot going on, but please expect some sporadic jenkins
>> > > > downtime until monday.  here's the abbreviated list of when we can
>> > > > expect things to happen:
>> > > >
>> > > > * tomorrow @ 9am we are switching to generator power.  there will be
>> > > > temporary loss of power to the jenkins workers, and if any don't come
>> > > > back after the switch i will need to head down there and manually
>> > > > power them back on.
>> > > >
>> > > > * thursday @ 9am, switch back to campus power.  again, i will deal
>> > > > w/any workers that don't come back manually.
>> > > >
>> > > > * saturday @ 9am, back to generator power.  same deal as above.
>> > > >
>> > > > * saturday or sunday the repairs will be completed.  i will monitor
>> > > > the situation, and deal w/any problem servers on my way in to work on
>> > > > monday.
>> > > >
>> > > > i apologize for any inconvenience that this will cause...  and i hope
>> > > > that this is the final piece of the power puzzle that our colo team
>> > > > has to deal with.
>> > > >
>> > > > thanks in advance,
>> > > >
>> > > > shane
>> > > > --
>> > > > Shane Knapp
>> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > > > https://rise.cs.berkeley.edu
>> > >
>> > >
>> > >
>> > > --
>> > > Shane Knapp
>> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > > https://rise.cs.berkeley.edu
>> >
>> >
>> >
>> > --
>> > Shane Knapp
>> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>> > https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: [hidden email]
>>
>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--


--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [build system] our colo is having power issues again. there will be a few 'events' this week

shane knapp
alright, they switched over from generator power back to the grid
about an hour ago...  all the workers are back up and building!

shane

On Tue, Sep 24, 2019 at 3:23 PM Takeshi Yamamuro <[hidden email]> wrote:

>
> Shane, thanks for the hard work!
>
> Bests,
> Takeshi
>
> On Wed, Sep 25, 2019 at 6:07 AM Jungtaek Lim <[hidden email]> wrote:
>>
>> Awesome, thanks for the quick update!
>>
>> On Wed, Sep 25, 2019 at 6:04 AM Shane Knapp <[hidden email]> wrote:
>>>
>>> no worries.  since we deprecated the packaging builds i put that
>>> worker back in to the rotation...  there was a slight env issue but
>>> that's fixed and we should be g2g w/python.
>>>
>>> On Tue, Sep 24, 2019 at 1:50 PM Jungtaek Lim <[hidden email]> wrote:
>>> >
>>> > Hi Shane,
>>> >
>>> > Thanks for the update, and take care of build system!
>>> >
>>> > Looks like some of build just got failed without test failures, looks like env. issue.
>>> > https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/111308/
>>> >
>>> > + ./dev/run-tests-jenkins
>>> > Python versions prior to 2.7 are not supported.
>>> >
>>> >
>>> > Could you please check on this?
>>> >
>>> > Thanks,
>>> > Jungtaek Lim (HeartSaVioR)
>>> >
>>> >
>>> > On Wed, Sep 25, 2019 at 2:48 AM Shane Knapp <[hidden email]> wrote:
>>> >>
>>> >> quick update from our colo admin:  they are going to keep the colo on
>>> >> generator power until monday morning and not switch back and forth.
>>> >> this is great as we'll have solid uptime until monday morning when we
>>> >> go back to grid power.
>>> >>
>>> >> if this changes i will be certain to update everyone here.
>>> >>
>>> >> thanks,
>>> >>
>>> >> shane
>>> >>
>>> >> On Tue, Sep 24, 2019 at 9:50 AM Shane Knapp <[hidden email]> wrote:
>>> >> >
>>> >> > aaaaaand that was quick!  everything is back up and building
>>> >> >
>>> >> > On Tue, Sep 24, 2019 at 9:39 AM Shane Knapp <[hidden email]> wrote:
>>> >> > >
>>> >> > > power switchover is happening now.  more updates to come once machines
>>> >> > > come back up.
>>> >> > >
>>> >> > > On Mon, Sep 23, 2019 at 3:16 PM Shane Knapp <[hidden email]> wrote:
>>> >> > > >
>>> >> > > > the main transformer for our colo is experiencing major issues, and
>>> >> > > > campus be performing emergency work on it starting tomorrow morning
>>> >> > > > (tuesday sept 24, 9am PDT).  it's pretty dire.  :(
>>> >> > > >
>>> >> > > > there's a lot going on, but please expect some sporadic jenkins
>>> >> > > > downtime until monday.  here's the abbreviated list of when we can
>>> >> > > > expect things to happen:
>>> >> > > >
>>> >> > > > * tomorrow @ 9am we are switching to generator power.  there will be
>>> >> > > > temporary loss of power to the jenkins workers, and if any don't come
>>> >> > > > back after the switch i will need to head down there and manually
>>> >> > > > power them back on.
>>> >> > > >
>>> >> > > > * thursday @ 9am, switch back to campus power.  again, i will deal
>>> >> > > > w/any workers that don't come back manually.
>>> >> > > >
>>> >> > > > * saturday @ 9am, back to generator power.  same deal as above.
>>> >> > > >
>>> >> > > > * saturday or sunday the repairs will be completed.  i will monitor
>>> >> > > > the situation, and deal w/any problem servers on my way in to work on
>>> >> > > > monday.
>>> >> > > >
>>> >> > > > i apologize for any inconvenience that this will cause...  and i hope
>>> >> > > > that this is the final piece of the power puzzle that our colo team
>>> >> > > > has to deal with.
>>> >> > > >
>>> >> > > > thanks in advance,
>>> >> > > >
>>> >> > > > shane
>>> >> > > > --
>>> >> > > > Shane Knapp
>>> >> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> >> > > > https://rise.cs.berkeley.edu
>>> >> > >
>>> >> > >
>>> >> > >
>>> >> > > --
>>> >> > > Shane Knapp
>>> >> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> >> > > https://rise.cs.berkeley.edu
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Shane Knapp
>>> >> > UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> >> > https://rise.cs.berkeley.edu
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Shane Knapp
>>> >> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> >> https://rise.cs.berkeley.edu
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe e-mail: [hidden email]
>>> >>
>>> >
>>> >
>>> > --
>>> > Name : Jungtaek Lim
>>> > Blog : http://medium.com/@heartsavior
>>> > Twitter : http://twitter.com/heartsavior
>>> > LinkedIn : http://www.linkedin.com/in/heartsavior
>>>
>>>
>>>
>>> --
>>> Shane Knapp
>>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>>> https://rise.cs.berkeley.edu
>>
>>
>>
>> --
>> Name : Jungtaek Lim
>> Blog : http://medium.com/@heartsavior
>> Twitter : http://twitter.com/heartsavior
>> LinkedIn : http://www.linkedin.com/in/heartsavior
>
>
>
> --
> ---
> Takeshi Yamamuro



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]