[build system] colo maintenance & outage tomorrow, 10am-2pm PDT

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[build system] colo maintenance & outage tomorrow, 10am-2pm PDT

shane knapp
the berkeley colo had a major power distribution breaker fail, and they've scheduled an emergency repair for tomorrow (thursday) @ 10am.

they expect this to take ~4 hours.

i will be shutting down the machines (again) ~9am, and bringing them all back up once i get the all-clear.

sorry for the inconvenience...

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

shane knapp
quick update:

it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored.

once i hear something i will let everyone know what's up.

On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp <[hidden email]> wrote:
the berkeley colo had a major power distribution breaker fail, and they've scheduled an emergency repair for tomorrow (thursday) @ 10am.

they expect this to take ~4 hours.

i will be shutting down the machines (again) ~9am, and bringing them all back up once i get the all-clear.

sorry for the inconvenience...

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

shane knapp
just got an update:  

there was a problem w/the replacement part, and they're trying to fix it.  if that's successful, the expect to have power restored within the hour.

if that doesn't work, a new (new) replacement part is scheduled to arrive at 8am tomorrow.

shane

On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp <[hidden email]> wrote:
quick update:

it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored.

once i hear something i will let everyone know what's up.

On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp <[hidden email]> wrote:
the berkeley colo had a major power distribution breaker fail, and they've scheduled an emergency repair for tomorrow (thursday) @ 10am.

they expect this to take ~4 hours.

i will be shutting down the machines (again) ~9am, and bringing them all back up once i get the all-clear.

sorry for the inconvenience...

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

cloud0fan
Thanks for tracking it Shane!

On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp <[hidden email]> wrote:
just got an update:  

there was a problem w/the replacement part, and they're trying to fix it.  if that's successful, the expect to have power restored within the hour.

if that doesn't work, a new (new) replacement part is scheduled to arrive at 8am tomorrow.

shane

On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp <[hidden email]> wrote:
quick update:

it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored.

once i hear something i will let everyone know what's up.

On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp <[hidden email]> wrote:
the berkeley colo had a major power distribution breaker fail, and they've scheduled an emergency repair for tomorrow (thursday) @ 10am.

they expect this to take ~4 hours.

i will be shutting down the machines (again) ~9am, and bringing them all back up once i get the all-clear.

sorry for the inconvenience...

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

shane knapp
it's back up!  some of the workers didn't come back cleanly, so i'll have to hit up the colo tomorrow and persuade them in person.

On Thu, Aug 15, 2019 at 6:45 PM Wenchen Fan <[hidden email]> wrote:
Thanks for tracking it Shane!

On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp <[hidden email]> wrote:
just got an update:  

there was a problem w/the replacement part, and they're trying to fix it.  if that's successful, the expect to have power restored within the hour.

if that doesn't work, a new (new) replacement part is scheduled to arrive at 8am tomorrow.

shane

On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp <[hidden email]> wrote:
quick update:

it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored.

once i hear something i will let everyone know what's up.

On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp <[hidden email]> wrote:
the berkeley colo had a major power distribution breaker fail, and they've scheduled an emergency repair for tomorrow (thursday) @ 10am.

they expect this to take ~4 hours.

i will be shutting down the machines (again) ~9am, and bringing them all back up once i get the all-clear.

sorry for the inconvenience...

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] colo maintenance & outage tomorrow, 10am-2pm PDT

shane knapp
a couple of workers needed a bit more time to finish booting up, so no need for my excursion tomorrow.  :)

builds be building, things look happy.  

On Thu, Aug 15, 2019 at 6:46 PM Shane Knapp <[hidden email]> wrote:
it's back up!  some of the workers didn't come back cleanly, so i'll have to hit up the colo tomorrow and persuade them in person.

On Thu, Aug 15, 2019 at 6:45 PM Wenchen Fan <[hidden email]> wrote:
Thanks for tracking it Shane!

On Fri, Aug 16, 2019 at 7:41 AM Shane Knapp <[hidden email]> wrote:
just got an update:  

there was a problem w/the replacement part, and they're trying to fix it.  if that's successful, the expect to have power restored within the hour.

if that doesn't work, a new (new) replacement part is scheduled to arrive at 8am tomorrow.

shane

On Thu, Aug 15, 2019 at 2:07 PM Shane Knapp <[hidden email]> wrote:
quick update:

it's been 4 hours, the colo is still down, and i haven't gotten any news yet as to when they're planning on getting power restored.

once i hear something i will let everyone know what's up.

On Wed, Aug 14, 2019 at 10:22 AM Shane Knapp <[hidden email]> wrote:
the berkeley colo had a major power distribution breaker fail, and they've scheduled an emergency repair for tomorrow (thursday) @ 10am.

they expect this to take ~4 hours.

i will be shutting down the machines (again) ~9am, and bringing them all back up once i get the all-clear.

sorry for the inconvenience...

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead