[build system] jenkins wedged again, rebooting master node

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[build system] jenkins wedged again, rebooting master node

shane knapp
as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.

jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
well, that box rebooted in record time!  we're back up and building.

and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\

On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.

jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

cloud0fan
cool, thanks!

On Sat, Mar 16, 2019 at 1:08 AM shane knapp <[hidden email]> wrote:
well, that box rebooted in record time!  we're back up and building.

and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\

On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.

jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

Sean Owen-2
In reply to this post by shane knapp
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:

>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
argh.  kicking it again.

On Fri, Mar 15, 2019 at 8:22 PM Sean Owen <[hidden email]> wrote:
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:
>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
In reply to this post by Sean Owen-2
On Fri, Mar 15, 2019 at 8:22 PM Sean Owen <[hidden email]> wrote:
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

it's back up and running now.

btw, the only way to kick it harder would be to do a complete reinstall...

...which really isn't that bad of an idea.  ;)

situations like this seem to happen every couple of years.  last time i 'upgraded' to java8 and put a bunch of JVM GC cleanup parameters in to the java options at launch, and that's kept us going for ~2 years+:

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -XX:+UseG1GC -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -server -XX:+AlwaysPreTouch -Xmx16g -Xms10g -Djava.net.preferIPv4Stack=true"

i'll be sure to spend some QT seeing if i can get this figured out next week.  maybe it's time to move off of java8 and up to turn it up to 11?

shane 
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

Sean Owen-2
In reply to this post by shane knapp
It's wedged against since this morning. Something's clearly gone
wrong-er than usual; any recent changes that could be a culprit?

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:

>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
i kicked the service.  again.  and jenkins seems happy for now.

nothing has changed (system config, packages, etc) on the master node.  i'll dive in to this tomorrow morning.

On Sun, Mar 17, 2019 at 9:46 AM Sean Owen <[hidden email]> wrote:
It's wedged against since this morning. Something's clearly gone
wrong-er than usual; any recent changes that could be a culprit?

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:
>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
In reply to this post by Sean Owen-2
ok, i dug through the logs and noticed that rsyslogd was dropping messages to do imuxsock being spammed by postfix...  which i then tracked down to our installation of fail2ban being incorrectly configured and attempting to send IP ban/unban status emails to '[hidden email]'.

since we're a university, and especially one w/a reputation like ours, we are constantly under attack.  the logs of the attempted dictionary attacks would astound you in their size and scope.  since we have so many ban/unban actions happening for all of these unique IP address, each of which generates an email that was directed to an invalid address, we ended up w/well over 100M of plain-text messages waiting in the mail queue.  postfix was continually trying to send these messages, which was causing the system to behave strangely, including breaking rsyslogd.

so, i disabled email reports in fail2ban, restarted the impacted services, picked my sysadmin's brain and then purged the mail queue (when was the last time anyone actually used postfix?).  jenkins now seems to be behaving (maybe?).

i'm not entirely sure that this will fix the strange GUI hangs, but all reports i found on stackoverflow and other sites detail strange system behavior across the board when rsyslogd starts dropping messages.  at the very least we won't be (potentially) losing system-level log messages anymore, which might actually help me track down what's happening if jenkins gets wedged again.

and finally, the obligatory IT Crowd clip:

shane (who expects jenkins to crash within 5 minutes of this email going out)

On Fri, Mar 15, 2019 at 8:22 PM Sean Owen <[hidden email]> wrote:
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:
>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

Imran Rashid-5
seems wedged again?

sorry for the bad news Shane, thanks for all the work on fixing it

On Mon, Mar 18, 2019 at 4:02 PM shane knapp <[hidden email]> wrote:
ok, i dug through the logs and noticed that rsyslogd was dropping messages to do imuxsock being spammed by postfix...  which i then tracked down to our installation of fail2ban being incorrectly configured and attempting to send IP ban/unban status emails to '[hidden email]'.

since we're a university, and especially one w/a reputation like ours, we are constantly under attack.  the logs of the attempted dictionary attacks would astound you in their size and scope.  since we have so many ban/unban actions happening for all of these unique IP address, each of which generates an email that was directed to an invalid address, we ended up w/well over 100M of plain-text messages waiting in the mail queue.  postfix was continually trying to send these messages, which was causing the system to behave strangely, including breaking rsyslogd.

so, i disabled email reports in fail2ban, restarted the impacted services, picked my sysadmin's brain and then purged the mail queue (when was the last time anyone actually used postfix?).  jenkins now seems to be behaving (maybe?).

i'm not entirely sure that this will fix the strange GUI hangs, but all reports i found on stackoverflow and other sites detail strange system behavior across the board when rsyslogd starts dropping messages.  at the very least we won't be (potentially) losing system-level log messages anymore, which might actually help me track down what's happening if jenkins gets wedged again.

and finally, the obligatory IT Crowd clip:

shane (who expects jenkins to crash within 5 minutes of this email going out)

On Fri, Mar 15, 2019 at 8:22 PM Sean Owen <[hidden email]> wrote:
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:
>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
i tweaked some apache settings (MaxClients increased to fix an error i found buried in the logs, and added 'retry' and 'acquire' to the reverse proxy settings to hopefully combat the dreaded 502 response), restarted httpd and things actually seem quite snappy right now!

i'm not holding my breath, however...  only time will tell.

On Tue, Mar 19, 2019 at 7:18 AM Imran Rashid <[hidden email]> wrote:
seems wedged again?

sorry for the bad news Shane, thanks for all the work on fixing it

On Mon, Mar 18, 2019 at 4:02 PM shane knapp <[hidden email]> wrote:
ok, i dug through the logs and noticed that rsyslogd was dropping messages to do imuxsock being spammed by postfix...  which i then tracked down to our installation of fail2ban being incorrectly configured and attempting to send IP ban/unban status emails to '[hidden email]'.

since we're a university, and especially one w/a reputation like ours, we are constantly under attack.  the logs of the attempted dictionary attacks would astound you in their size and scope.  since we have so many ban/unban actions happening for all of these unique IP address, each of which generates an email that was directed to an invalid address, we ended up w/well over 100M of plain-text messages waiting in the mail queue.  postfix was continually trying to send these messages, which was causing the system to behave strangely, including breaking rsyslogd.

so, i disabled email reports in fail2ban, restarted the impacted services, picked my sysadmin's brain and then purged the mail queue (when was the last time anyone actually used postfix?).  jenkins now seems to be behaving (maybe?).

i'm not entirely sure that this will fix the strange GUI hangs, but all reports i found on stackoverflow and other sites detail strange system behavior across the board when rsyslogd starts dropping messages.  at the very least we won't be (potentially) losing system-level log messages anymore, which might actually help me track down what's happening if jenkins gets wedged again.

and finally, the obligatory IT Crowd clip:

shane (who expects jenkins to crash within 5 minutes of this email going out)

On Fri, Mar 15, 2019 at 8:22 PM Sean Owen <[hidden email]> wrote:
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:
>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
i was right to not hold my breath...  while my apache changes seem to have helped a bit, things are still slowing down after 10-12 hours.

i have a few other things i can look at, and will get as much done as possible before the weekend.  serious troubleshooting will begin anew monday.

apologies again,

shane

On Thu, Mar 21, 2019 at 12:54 PM shane knapp <[hidden email]> wrote:
i tweaked some apache settings (MaxClients increased to fix an error i found buried in the logs, and added 'retry' and 'acquire' to the reverse proxy settings to hopefully combat the dreaded 502 response), restarted httpd and things actually seem quite snappy right now!

i'm not holding my breath, however...  only time will tell.

On Tue, Mar 19, 2019 at 7:18 AM Imran Rashid <[hidden email]> wrote:
seems wedged again?

sorry for the bad news Shane, thanks for all the work on fixing it

On Mon, Mar 18, 2019 at 4:02 PM shane knapp <[hidden email]> wrote:
ok, i dug through the logs and noticed that rsyslogd was dropping messages to do imuxsock being spammed by postfix...  which i then tracked down to our installation of fail2ban being incorrectly configured and attempting to send IP ban/unban status emails to '[hidden email]'.

since we're a university, and especially one w/a reputation like ours, we are constantly under attack.  the logs of the attempted dictionary attacks would astound you in their size and scope.  since we have so many ban/unban actions happening for all of these unique IP address, each of which generates an email that was directed to an invalid address, we ended up w/well over 100M of plain-text messages waiting in the mail queue.  postfix was continually trying to send these messages, which was causing the system to behave strangely, including breaking rsyslogd.

so, i disabled email reports in fail2ban, restarted the impacted services, picked my sysadmin's brain and then purged the mail queue (when was the last time anyone actually used postfix?).  jenkins now seems to be behaving (maybe?).

i'm not entirely sure that this will fix the strange GUI hangs, but all reports i found on stackoverflow and other sites detail strange system behavior across the board when rsyslogd starts dropping messages.  at the very least we won't be (potentially) losing system-level log messages anymore, which might actually help me track down what's happening if jenkins gets wedged again.

and finally, the obligatory IT Crowd clip:

shane (who expects jenkins to crash within 5 minutes of this email going out)

On Fri, Mar 15, 2019 at 8:22 PM Sean Owen <[hidden email]> wrote:
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:
>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins wedged again, rebooting master node

shane knapp
quick update:  

since kicking httpd on the jenkins master "fixes" the GUI hanging, i set up a cron job to restart httpd 4 times per day.  

this is not the final solution, but will definitely help over the weekend as i'm heading out of town.

shane

On Fri, Mar 22, 2019 at 9:50 AM shane knapp <[hidden email]> wrote:
i was right to not hold my breath...  while my apache changes seem to have helped a bit, things are still slowing down after 10-12 hours.

i have a few other things i can look at, and will get as much done as possible before the weekend.  serious troubleshooting will begin anew monday.

apologies again,

shane

On Thu, Mar 21, 2019 at 12:54 PM shane knapp <[hidden email]> wrote:
i tweaked some apache settings (MaxClients increased to fix an error i found buried in the logs, and added 'retry' and 'acquire' to the reverse proxy settings to hopefully combat the dreaded 502 response), restarted httpd and things actually seem quite snappy right now!

i'm not holding my breath, however...  only time will tell.

On Tue, Mar 19, 2019 at 7:18 AM Imran Rashid <[hidden email]> wrote:
seems wedged again?

sorry for the bad news Shane, thanks for all the work on fixing it

On Mon, Mar 18, 2019 at 4:02 PM shane knapp <[hidden email]> wrote:
ok, i dug through the logs and noticed that rsyslogd was dropping messages to do imuxsock being spammed by postfix...  which i then tracked down to our installation of fail2ban being incorrectly configured and attempting to send IP ban/unban status emails to '[hidden email]'.

since we're a university, and especially one w/a reputation like ours, we are constantly under attack.  the logs of the attempted dictionary attacks would astound you in their size and scope.  since we have so many ban/unban actions happening for all of these unique IP address, each of which generates an email that was directed to an invalid address, we ended up w/well over 100M of plain-text messages waiting in the mail queue.  postfix was continually trying to send these messages, which was causing the system to behave strangely, including breaking rsyslogd.

so, i disabled email reports in fail2ban, restarted the impacted services, picked my sysadmin's brain and then purged the mail queue (when was the last time anyone actually used postfix?).  jenkins now seems to be behaving (maybe?).

i'm not entirely sure that this will fix the strange GUI hangs, but all reports i found on stackoverflow and other sites detail strange system behavior across the board when rsyslogd starts dropping messages.  at the very least we won't be (potentially) losing system-level log messages anymore, which might actually help me track down what's happening if jenkins gets wedged again.

and finally, the obligatory IT Crowd clip:

shane (who expects jenkins to crash within 5 minutes of this email going out)

On Fri, Mar 15, 2019 at 8:22 PM Sean Owen <[hidden email]> wrote:
It's not responding again. Is there any way to kick it harder? I know
it's well understood but this means not much can be merged in Spark

On Fri, Mar 15, 2019 at 12:08 PM shane knapp <[hidden email]> wrote:
>
> well, that box rebooted in record time!  we're back up and building.
>
> and as always, i'll keep a close eye on things today...  jenkins usually works great, until it doesn't.  :\
>
> On Fri, Mar 15, 2019 at 9:52 AM shane knapp <[hidden email]> wrote:
>>
>> as some of you may have noticed, jenkins got itself in a bad state multiple times over the past couple of weeks.  usually restarting the service is sufficient, but it appears that i need to hit it w/the reboot hammer.
>>
>> jenkins will be down for the next 20-30 minutes as the node reboots and jenkins spins back up.  i'll reply here w/any updates.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead