[build system] intermittent network issues + potential power shutoff over the weekend

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[build system] intermittent network issues + potential power shutoff over the weekend

shane knapp
1) our department is having some issues w/their network and other
services.  this means that if you're at the jenkins site, you may
occasionally get a 503 error.  just hit refresh a couple of times and
it will come back.

2) the OTHER thing is that PG&E will be potentially cutting power to
campus again tomorrow evening and over the weekend.  i am unsure 1) if
this is happening, and 2) if our campus colo will go on to generator
backup power.

if the colo does go on backup power, all of the workers will shut down
and power back on automatically when power is restored.

i don't really have any more information for either of these events.
the situation regarding the power shutoff is fluid and changing, and
depending on the weather and winds, could potentially not happen at
all, or impact us until monday.

once i hear more i will update this message.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

shane knapp
> 1) our department is having some issues w/their network and other
> services.  this means that if you're at the jenkins site, you may
> occasionally get a 503 error.  just hit refresh a couple of times and
> it will come back.
>
btw this might impact some pull request builds, so if your PR or
'retest this please' comment doesn't seem to catch and trigger a build
after ~15 mins, feel free (if you're whitelisted or an admin) to
request another test with 'test this please'.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

Dongjoon Hyun-2
Thank you for notice, Shane.

Bests,
Dongjoon.

On Fri, Oct 25, 2019 at 12:31 PM Shane Knapp <[hidden email]> wrote:
> 1) our department is having some issues w/their network and other
> services.  this means that if you're at the jenkins site, you may
> occasionally get a 503 error.  just hit refresh a couple of times and
> it will come back.
>
btw this might impact some pull request builds, so if your PR or
'retest this please' comment doesn't seem to catch and trigger a build
after ~15 mins, feel free (if you're whitelisted or an admin) to
request another test with 'test this please'.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

shane knapp
In reply to this post by shane knapp
> 2) the OTHER thing is that PG&E will be potentially cutting power to
> campus again tomorrow evening and over the weekend.  i am unsure 1) if
> this is happening, and 2) if our campus colo will go on to generator
> backup power.
>
> if the colo does go on backup power, all of the workers will shut down
> and power back on automatically when power is restored.
>
ok, it looks like the colo will have power until monday morning, and
it will be shut down from 8am to noon to perform some maintenance.

this means jenkins will be up all weekend, but down monday morning.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

Takeshi Yamamuro
ok, thanks, Shane!

Bests,
Takeshi

On Sat, Oct 26, 2019 at 7:01 AM Shane Knapp <[hidden email]> wrote:
> 2) the OTHER thing is that PG&E will be potentially cutting power to
> campus again tomorrow evening and over the weekend.  i am unsure 1) if
> this is happening, and 2) if our campus colo will go on to generator
> backup power.
>
> if the colo does go on backup power, all of the workers will shut down
> and power back on automatically when power is restored.
>
ok, it looks like the colo will have power until monday morning, and
it will be shut down from 8am to noon to perform some maintenance.

this means jenkins will be up all weekend, but down monday morning.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

shane knapp
In reply to this post by shane knapp
ok, it looks like the colo will have power until monday morning, and
it will be shut down from 8am to noon to perform some maintenance.

this means jenkins will be up all weekend, but down monday morning.

jenkins is currently down due to colo maintenance.  expect it to return in ~3.5 hours.

shane 
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

shane knapp
we're back up and building!

On Mon, Oct 28, 2019 at 8:35 AM Shane Knapp <[hidden email]> wrote:
ok, it looks like the colo will have power until monday morning, and
it will be shut down from 8am to noon to perform some maintenance.

this means jenkins will be up all weekend, but down monday morning.

jenkins is currently down due to colo maintenance.  expect it to return in ~3.5 hours.

shane 
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

shane knapp
i will need to restart jenkins -- the worker's ENV vars got borked when they came back up.

this is happening NOW.

shane

On Mon, Oct 28, 2019 at 10:37 AM Shane Knapp <[hidden email]> wrote:
we're back up and building!

On Mon, Oct 28, 2019 at 8:35 AM Shane Knapp <[hidden email]> wrote:
ok, it looks like the colo will have power until monday morning, and
it will be shut down from 8am to noon to perform some maintenance.

this means jenkins will be up all weekend, but down monday morning.

jenkins is currently down due to colo maintenance.  expect it to return in ~3.5 hours.

shane 
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] intermittent network issues + potential power shutoff over the weekend

Dongjoon Hyun-2
Thank you for fixing the worker ENVs, Shane.

Bests,
Dongjoon.

On Mon, Oct 28, 2019 at 10:47 AM Shane Knapp <[hidden email]> wrote:
i will need to restart jenkins -- the worker's ENV vars got borked when they came back up.

this is happening NOW.

shane

On Mon, Oct 28, 2019 at 10:37 AM Shane Knapp <[hidden email]> wrote:
we're back up and building!

On Mon, Oct 28, 2019 at 8:35 AM Shane Knapp <[hidden email]> wrote:
ok, it looks like the colo will have power until monday morning, and
it will be shut down from 8am to noon to perform some maintenance.

this means jenkins will be up all weekend, but down monday morning.

jenkins is currently down due to colo maintenance.  expect it to return in ~3.5 hours.

shane 
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead