[build system] IMPORTANT! northern california fire danger, potential power outage(s)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[build system] IMPORTANT! northern california fire danger, potential power outage(s)

shane knapp
here in the lovely bay area, we are currently experiencing some
absolutely lovely weather:  temps around 20C, light winds, and not a
drop of moisture anywhere.

this means that wildfire season is here, and our utilities company
(PG&E) is very concerned about fires like last year's Camp Fire
(https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
(https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
fires (https://en.wikipedia.org/wiki/2017_California_wildfires).

because conditions are absolutely perfect for wildfires, we may lose
power here in berkeley tomorrow and thursday.

there will be little to no notice of then this might happen, and if it
does that means that jenkins will most definitely go down.

i will continue to keep a close eye on this and give updates as they
happen.  sadly, the pg&e website is down because they apparently
didn't think that they needed load balancers.  :\

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

Xiao Li-2
Hi, Shane,

Thank you for letting us know in advance!

Xiao

On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
here in the lovely bay area, we are currently experiencing some
absolutely lovely weather:  temps around 20C, light winds, and not a
drop of moisture anywhere.

this means that wildfire season is here, and our utilities company
(PG&E) is very concerned about fires like last year's Camp Fire
(https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
(https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
fires (https://en.wikipedia.org/wiki/2017_California_wildfires).

because conditions are absolutely perfect for wildfires, we may lose
power here in berkeley tomorrow and thursday.

there will be little to no notice of then this might happen, and if it
does that means that jenkins will most definitely go down.

i will continue to keep a close eye on this and give updates as they
happen.  sadly, the pg&e website is down because they apparently
didn't think that they needed load balancers.  :\

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

--
Databricks Summit - Watch the talks 
Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

shane knapp
In reply to this post by shane knapp
quick update:

we are definitely going to have our power shut off starting early
tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
before it is restored.

i will be shutting jenkins down some time this evening, and will
update everyone here when i get more information.

full service will be restored (i HOPE) by friday morning.

shane (who doesn't ever want to check this list's archives and count
how many times we've had power issues)

On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:

>
> here in the lovely bay area, we are currently experiencing some
> absolutely lovely weather:  temps around 20C, light winds, and not a
> drop of moisture anywhere.
>
> this means that wildfire season is here, and our utilities company
> (PG&E) is very concerned about fires like last year's Camp Fire
> (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
> (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
> fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
>
> because conditions are absolutely perfect for wildfires, we may lose
> power here in berkeley tomorrow and thursday.
>
> there will be little to no notice of then this might happen, and if it
> does that means that jenkins will most definitely go down.
>
> i will continue to keep a close eye on this and give updates as they
> happen.  sadly, the pg&e website is down because they apparently
> didn't think that they needed load balancers.  :\
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

shane knapp
jenkins is going down now.

On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp <[hidden email]> wrote:

>
> quick update:
>
> we are definitely going to have our power shut off starting early
> tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
> before it is restored.
>
> i will be shutting jenkins down some time this evening, and will
> update everyone here when i get more information.
>
> full service will be restored (i HOPE) by friday morning.
>
> shane (who doesn't ever want to check this list's archives and count
> how many times we've had power issues)
>
> On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
> >
> > here in the lovely bay area, we are currently experiencing some
> > absolutely lovely weather:  temps around 20C, light winds, and not a
> > drop of moisture anywhere.
> >
> > this means that wildfire season is here, and our utilities company
> > (PG&E) is very concerned about fires like last year's Camp Fire
> > (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
> > (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
> > fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
> >
> > because conditions are absolutely perfect for wildfires, we may lose
> > power here in berkeley tomorrow and thursday.
> >
> > there will be little to no notice of then this might happen, and if it
> > does that means that jenkins will most definitely go down.
> >
> > i will continue to keep a close eye on this and give updates as they
> > happen.  sadly, the pg&e website is down because they apparently
> > didn't think that they needed load balancers.  :\
> >
> > shane
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

shane knapp
quick update:

campus is losing power @ 8pm.  this is after we were told 4am, 8am,
noon, and 2-4pm.  :)

PG&E expects to start bringing alameda county back online at noon
tomorrow, but i believe that target to be fluid and take longer than
expected.

this means that the earliest that we can bring the build system back
up is friday, but there's a much greater than non-zero chance of this
not happening until monday morning.  i will be leaving town for the
weekend friday afternoon, which means i won't be physically present to
turn on all of our servers in the colo (about ~80 servers including
jenkins) until monday.

more updates as they come.  thanks for your patience!

On Tue, Oct 8, 2019 at 7:32 PM Shane Knapp <[hidden email]> wrote:

>
> jenkins is going down now.
>
> On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp <[hidden email]> wrote:
> >
> > quick update:
> >
> > we are definitely going to have our power shut off starting early
> > tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
> > before it is restored.
> >
> > i will be shutting jenkins down some time this evening, and will
> > update everyone here when i get more information.
> >
> > full service will be restored (i HOPE) by friday morning.
> >
> > shane (who doesn't ever want to check this list's archives and count
> > how many times we've had power issues)
> >
> > On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
> > >
> > > here in the lovely bay area, we are currently experiencing some
> > > absolutely lovely weather:  temps around 20C, light winds, and not a
> > > drop of moisture anywhere.
> > >
> > > this means that wildfire season is here, and our utilities company
> > > (PG&E) is very concerned about fires like last year's Camp Fire
> > > (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
> > > (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
> > > fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
> > >
> > > because conditions are absolutely perfect for wildfires, we may lose
> > > power here in berkeley tomorrow and thursday.
> > >
> > > there will be little to no notice of then this might happen, and if it
> > > does that means that jenkins will most definitely go down.
> > >
> > > i will continue to keep a close eye on this and give updates as they
> > > happen.  sadly, the pg&e website is down because they apparently
> > > didn't think that they needed load balancers.  :\
> > >
> > > shane
> > > --
> > > Shane Knapp
> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > > https://rise.cs.berkeley.edu
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

cloud0fan
Thanks for the updates!

On Thu, Oct 10, 2019 at 5:34 AM Shane Knapp <[hidden email]> wrote:
quick update:

campus is losing power @ 8pm.  this is after we were told 4am, 8am,
noon, and 2-4pm.  :)

PG&E expects to start bringing alameda county back online at noon
tomorrow, but i believe that target to be fluid and take longer than
expected.

this means that the earliest that we can bring the build system back
up is friday, but there's a much greater than non-zero chance of this
not happening until monday morning.  i will be leaving town for the
weekend friday afternoon, which means i won't be physically present to
turn on all of our servers in the colo (about ~80 servers including
jenkins) until monday.

more updates as they come.  thanks for your patience!

On Tue, Oct 8, 2019 at 7:32 PM Shane Knapp <[hidden email]> wrote:
>
> jenkins is going down now.
>
> On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp <[hidden email]> wrote:
> >
> > quick update:
> >
> > we are definitely going to have our power shut off starting early
> > tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
> > before it is restored.
> >
> > i will be shutting jenkins down some time this evening, and will
> > update everyone here when i get more information.
> >
> > full service will be restored (i HOPE) by friday morning.
> >
> > shane (who doesn't ever want to check this list's archives and count
> > how many times we've had power issues)
> >
> > On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
> > >
> > > here in the lovely bay area, we are currently experiencing some
> > > absolutely lovely weather:  temps around 20C, light winds, and not a
> > > drop of moisture anywhere.
> > >
> > > this means that wildfire season is here, and our utilities company
> > > (PG&E) is very concerned about fires like last year's Camp Fire
> > > (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
> > > (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
> > > fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
> > >
> > > because conditions are absolutely perfect for wildfires, we may lose
> > > power here in berkeley tomorrow and thursday.
> > >
> > > there will be little to no notice of then this might happen, and if it
> > > does that means that jenkins will most definitely go down.
> > >
> > > i will continue to keep a close eye on this and give updates as they
> > > happen.  sadly, the pg&e website is down because they apparently
> > > didn't think that they needed load balancers.  :\
> > >
> > > shane
> > > --
> > > Shane Knapp
> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > > https://rise.cs.berkeley.edu
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

shane knapp
In reply to this post by shane knapp
another quick update:

campus lost power ~1130pm, and is closed for the entirety of today.

no word on power restoration, campus status, etc etc.

updates as they come.  :\

On Wed, Oct 9, 2019 at 2:34 PM Shane Knapp <[hidden email]> wrote:

>
> quick update:
>
> campus is losing power @ 8pm.  this is after we were told 4am, 8am,
> noon, and 2-4pm.  :)
>
> PG&E expects to start bringing alameda county back online at noon
> tomorrow, but i believe that target to be fluid and take longer than
> expected.
>
> this means that the earliest that we can bring the build system back
> up is friday, but there's a much greater than non-zero chance of this
> not happening until monday morning.  i will be leaving town for the
> weekend friday afternoon, which means i won't be physically present to
> turn on all of our servers in the colo (about ~80 servers including
> jenkins) until monday.
>
> more updates as they come.  thanks for your patience!
>
> On Tue, Oct 8, 2019 at 7:32 PM Shane Knapp <[hidden email]> wrote:
> >
> > jenkins is going down now.
> >
> > On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp <[hidden email]> wrote:
> > >
> > > quick update:
> > >
> > > we are definitely going to have our power shut off starting early
> > > tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
> > > before it is restored.
> > >
> > > i will be shutting jenkins down some time this evening, and will
> > > update everyone here when i get more information.
> > >
> > > full service will be restored (i HOPE) by friday morning.
> > >
> > > shane (who doesn't ever want to check this list's archives and count
> > > how many times we've had power issues)
> > >
> > > On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
> > > >
> > > > here in the lovely bay area, we are currently experiencing some
> > > > absolutely lovely weather:  temps around 20C, light winds, and not a
> > > > drop of moisture anywhere.
> > > >
> > > > this means that wildfire season is here, and our utilities company
> > > > (PG&E) is very concerned about fires like last year's Camp Fire
> > > > (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
> > > > (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
> > > > fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
> > > >
> > > > because conditions are absolutely perfect for wildfires, we may lose
> > > > power here in berkeley tomorrow and thursday.
> > > >
> > > > there will be little to no notice of then this might happen, and if it
> > > > does that means that jenkins will most definitely go down.
> > > >
> > > > i will continue to keep a close eye on this and give updates as they
> > > > happen.  sadly, the pg&e website is down because they apparently
> > > > didn't think that they needed load balancers.  :\
> > > >
> > > > shane
> > > > --
> > > > Shane Knapp
> > > > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > > > https://rise.cs.berkeley.edu
> > >
> > >
> > >
> > > --
> > > Shane Knapp
> > > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > > https://rise.cs.berkeley.edu
> >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

shane knapp
finally, some good news!  power was just restored to campus.

i'm about to leave town, but jon (CCed) will be heading down to power
things up soon and we should hopefully be building in the next 3-4
hours.

the initial build queue will be massive, so he will purge it before
enabling builds.  this means any open PRs will need to be retriggered.
please don't do this until *after* you get the notice that jenkins is
up.

thanks for your patience,

shane

On Thu, Oct 10, 2019 at 9:05 AM Shane Knapp <[hidden email]> wrote:

>
> another quick update:
>
> campus lost power ~1130pm, and is closed for the entirety of today.
>
> no word on power restoration, campus status, etc etc.
>
> updates as they come.  :\
>
> On Wed, Oct 9, 2019 at 2:34 PM Shane Knapp <[hidden email]> wrote:
> >
> > quick update:
> >
> > campus is losing power @ 8pm.  this is after we were told 4am, 8am,
> > noon, and 2-4pm.  :)
> >
> > PG&E expects to start bringing alameda county back online at noon
> > tomorrow, but i believe that target to be fluid and take longer than
> > expected.
> >
> > this means that the earliest that we can bring the build system back
> > up is friday, but there's a much greater than non-zero chance of this
> > not happening until monday morning.  i will be leaving town for the
> > weekend friday afternoon, which means i won't be physically present to
> > turn on all of our servers in the colo (about ~80 servers including
> > jenkins) until monday.
> >
> > more updates as they come.  thanks for your patience!
> >
> > On Tue, Oct 8, 2019 at 7:32 PM Shane Knapp <[hidden email]> wrote:
> > >
> > > jenkins is going down now.
> > >
> > > On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp <[hidden email]> wrote:
> > > >
> > > > quick update:
> > > >
> > > > we are definitely going to have our power shut off starting early
> > > > tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
> > > > before it is restored.
> > > >
> > > > i will be shutting jenkins down some time this evening, and will
> > > > update everyone here when i get more information.
> > > >
> > > > full service will be restored (i HOPE) by friday morning.
> > > >
> > > > shane (who doesn't ever want to check this list's archives and count
> > > > how many times we've had power issues)
> > > >
> > > > On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
> > > > >
> > > > > here in the lovely bay area, we are currently experiencing some
> > > > > absolutely lovely weather:  temps around 20C, light winds, and not a
> > > > > drop of moisture anywhere.
> > > > >
> > > > > this means that wildfire season is here, and our utilities company
> > > > > (PG&E) is very concerned about fires like last year's Camp Fire
> > > > > (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
> > > > > (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
> > > > > fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
> > > > >
> > > > > because conditions are absolutely perfect for wildfires, we may lose
> > > > > power here in berkeley tomorrow and thursday.
> > > > >
> > > > > there will be little to no notice of then this might happen, and if it
> > > > > does that means that jenkins will most definitely go down.
> > > > >
> > > > > i will continue to keep a close eye on this and give updates as they
> > > > > happen.  sadly, the pg&e website is down because they apparently
> > > > > didn't think that they needed load balancers.  :\
> > > > >
> > > > > shane

--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

Xiao Li-2
That is great news!!!

Shane, have a good trip! 

Xiao

On Fri, Oct 11, 2019 at 1:58 PM Shane Knapp <[hidden email]> wrote:
finally, some good news!  power was just restored to campus.

i'm about to leave town, but jon (CCed) will be heading down to power
things up soon and we should hopefully be building in the next 3-4
hours.

the initial build queue will be massive, so he will purge it before
enabling builds.  this means any open PRs will need to be retriggered.
please don't do this until *after* you get the notice that jenkins is
up.

thanks for your patience,

shane

On Thu, Oct 10, 2019 at 9:05 AM Shane Knapp <[hidden email]> wrote:
>
> another quick update:
>
> campus lost power ~1130pm, and is closed for the entirety of today.
>
> no word on power restoration, campus status, etc etc.
>
> updates as they come.  :\
>
> On Wed, Oct 9, 2019 at 2:34 PM Shane Knapp <[hidden email]> wrote:
> >
> > quick update:
> >
> > campus is losing power @ 8pm.  this is after we were told 4am, 8am,
> > noon, and 2-4pm.  :)
> >
> > PG&E expects to start bringing alameda county back online at noon
> > tomorrow, but i believe that target to be fluid and take longer than
> > expected.
> >
> > this means that the earliest that we can bring the build system back
> > up is friday, but there's a much greater than non-zero chance of this
> > not happening until monday morning.  i will be leaving town for the
> > weekend friday afternoon, which means i won't be physically present to
> > turn on all of our servers in the colo (about ~80 servers including
> > jenkins) until monday.
> >
> > more updates as they come.  thanks for your patience!
> >
> > On Tue, Oct 8, 2019 at 7:32 PM Shane Knapp <[hidden email]> wrote:
> > >
> > > jenkins is going down now.
> > >
> > > On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp <[hidden email]> wrote:
> > > >
> > > > quick update:
> > > >
> > > > we are definitely going to have our power shut off starting early
> > > > tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
> > > > before it is restored.
> > > >
> > > > i will be shutting jenkins down some time this evening, and will
> > > > update everyone here when i get more information.
> > > >
> > > > full service will be restored (i HOPE) by friday morning.
> > > >
> > > > shane (who doesn't ever want to check this list's archives and count
> > > > how many times we've had power issues)
> > > >
> > > > On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
> > > > >
> > > > > here in the lovely bay area, we are currently experiencing some
> > > > > absolutely lovely weather:  temps around 20C, light winds, and not a
> > > > > drop of moisture anywhere.
> > > > >
> > > > > this means that wildfire season is here, and our utilities company
> > > > > (PG&E) is very concerned about fires like last year's Camp Fire
> > > > > (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
> > > > > (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
> > > > > fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
> > > > >
> > > > > because conditions are absolutely perfect for wildfires, we may lose
> > > > > power here in berkeley tomorrow and thursday.
> > > > >
> > > > > there will be little to no notice of then this might happen, and if it
> > > > > does that means that jenkins will most definitely go down.
> > > > >
> > > > > i will continue to keep a close eye on this and give updates as they
> > > > > happen.  sadly, the pg&e website is down because they apparently
> > > > > didn't think that they needed load balancers.  :\
> > > > >
> > > > > shane

--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Databricks Summit - Watch the talks 
Reply | Threaded
Open this post in threaded view
|

Re: [build system] IMPORTANT! northern california fire danger, potential power outage(s)

shane knapp
In reply to this post by shane knapp
thanks jon!

On Fri, Oct 11, 2019 at 3:25 PM Jon Kuroda <[hidden email]> wrote:

>
> Jenkins is back up and building. Build queue purged, builders are building. Fiat Lux --Jon
>
> On Fri, Oct 11, 2019 at 1:58 PM Shane Knapp <[hidden email]> wrote:
>>
>> finally, some good news!  power was just restored to campus.
>>
>> i'm about to leave town, but jon (CCed) will be heading down to power
>> things up soon and we should hopefully be building in the next 3-4
>> hours.
>>
>> the initial build queue will be massive, so he will purge it before
>> enabling builds.  this means any open PRs will need to be retriggered.
>> please don't do this until *after* you get the notice that jenkins is
>> up.
>>
>> thanks for your patience,
>>
>> shane
>>
>> On Thu, Oct 10, 2019 at 9:05 AM Shane Knapp <[hidden email]> wrote:
>> >
>> > another quick update:
>> >
>> > campus lost power ~1130pm, and is closed for the entirety of today.
>> >
>> > no word on power restoration, campus status, etc etc.
>> >
>> > updates as they come.  :\
>> >
>> > On Wed, Oct 9, 2019 at 2:34 PM Shane Knapp <[hidden email]> wrote:
>> > >
>> > > quick update:
>> > >
>> > > campus is losing power @ 8pm.  this is after we were told 4am, 8am,
>> > > noon, and 2-4pm.  :)
>> > >
>> > > PG&E expects to start bringing alameda county back online at noon
>> > > tomorrow, but i believe that target to be fluid and take longer than
>> > > expected.
>> > >
>> > > this means that the earliest that we can bring the build system back
>> > > up is friday, but there's a much greater than non-zero chance of this
>> > > not happening until monday morning.  i will be leaving town for the
>> > > weekend friday afternoon, which means i won't be physically present to
>> > > turn on all of our servers in the colo (about ~80 servers including
>> > > jenkins) until monday.
>> > >
>> > > more updates as they come.  thanks for your patience!
>> > >
>> > > On Tue, Oct 8, 2019 at 7:32 PM Shane Knapp <[hidden email]> wrote:
>> > > >
>> > > > jenkins is going down now.
>> > > >
>> > > > On Tue, Oct 8, 2019 at 4:21 PM Shane Knapp <[hidden email]> wrote:
>> > > > >
>> > > > > quick update:
>> > > > >
>> > > > > we are definitely going to have our power shut off starting early
>> > > > > tomorrow morning (by 4am PDT oct 9th), and expect at least 48 hours
>> > > > > before it is restored.
>> > > > >
>> > > > > i will be shutting jenkins down some time this evening, and will
>> > > > > update everyone here when i get more information.
>> > > > >
>> > > > > full service will be restored (i HOPE) by friday morning.
>> > > > >
>> > > > > shane (who doesn't ever want to check this list's archives and count
>> > > > > how many times we've had power issues)
>> > > > >
>> > > > > On Tue, Oct 8, 2019 at 12:50 PM Shane Knapp <[hidden email]> wrote:
>> > > > > >
>> > > > > > here in the lovely bay area, we are currently experiencing some
>> > > > > > absolutely lovely weather:  temps around 20C, light winds, and not a
>> > > > > > drop of moisture anywhere.
>> > > > > >
>> > > > > > this means that wildfire season is here, and our utilities company
>> > > > > > (PG&E) is very concerned about fires like last year's Camp Fire
>> > > > > > (https://en.wikipedia.org/wiki/Camp_Fire_(2018)), the 2018 fires
>> > > > > > (https://en.wikipedia.org/wiki/2018_California_wildfires) and 2017
>> > > > > > fires (https://en.wikipedia.org/wiki/2017_California_wildfires).
>> > > > > >
>> > > > > > because conditions are absolutely perfect for wildfires, we may lose
>> > > > > > power here in berkeley tomorrow and thursday.
>> > > > > >
>> > > > > > there will be little to no notice of then this might happen, and if it
>> > > > > > does that means that jenkins will most definitely go down.
>> > > > > >
>> > > > > > i will continue to keep a close eye on this and give updates as they
>> > > > > > happen.  sadly, the pg&e website is down because they apparently
>> > > > > > didn't think that they needed load balancers.  :\
>> > > > > >
>> > > > > > shane
>>
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu



--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]