[build system] jenkins slow to respond

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[build system] jenkins slow to respond

shane knapp
it's that time of the year again when jenkins decides to take it's time serving pages.

i will post any updates/etc when i get them.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins slow to respond

shane knapp
i kicked apache, and the number of active connections dropped from ~500 to ~100...  things are much snappier now.

On Fri, Aug 16, 2019 at 1:05 PM Shane Knapp <[hidden email]> wrote:
it's that time of the year again when jenkins decides to take it's time serving pages.

i will post any updates/etc when i get them.

shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead


--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
Reply | Threaded
Open this post in threaded view
|

Re: [build system] jenkins slow to respond

R. Tyler Croy
(replies inline)

On Fri, 16 Aug 2019, Shane Knapp wrote:

> i kicked apache, and the number of active connections dropped from ~500 to
> ~100...  things are much snappier now.


Hiya! I maintain ci.jenkins.io for the Jenkins project (among other things) and
have some snippets of code you can use to help reduce load on the Jenkins
instance, which can lead to performance issues.

Snippets usch as these which we run in Apache:
    https://github.com/jenkins-infra/jenkins-infra/blob/staging/dist/profile/manifests/buildmaster.pp#L386-L410


Basically we had found that a number of automated requests would access /api/*
paths, which can access Jenkins' tree-based data (this is okay) with an
arbitrarily deep level of the tree (not okay). We would see thousands of
request per day from bots requesting a 100+ depth view on our data model, which
Jenkins would spend many CPU cycles trying to service.

We just bocked a bunch of anonymous request and that solved many of our load
issues not directly related to running Jenkins Pipelines.


I can be found in #jenkins-infra on Freenode as "rtyler" if you have more
questions on how to improve Jenkins performance :)



--
GitHub:  https://github.com/rtyler

GPG Key ID: 0F2298A980EE31ACCA0A7825E5C92681BEF6CEA2

signature.asc (858 bytes) Download Attachment