[build system] currently experiencing git timeouts when building

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[build system] currently experiencing git timeouts when building

shane knapp
i just noticed we're starting to see the once-yearly rash of git timeouts when building.

i'll be looking in to this today...  i'm at our lab retreat, so my attention will be divided during the day but i will report back here once i have some more information.

in the meantime, if your jobs have a git timeout, please just retrigger them and we will hope for the best.

shane
Reply | Threaded
Open this post in threaded view
|

Re: [build system] currently experiencing git timeouts when building

shane knapp
update:  we're seeing about 3% of builds timing out, but today is (so far) a bad one:

i've reached out to github about this and am waiting to hear back.

$ get_timeouts.py 10
Timeouts by project:
 17    spark-branch-2.3-lint
 10    spark-branch-2.3-test-maven-hadoop-2.6
 7    spark-branch-2.3-compile-maven-hadoop-2.7
 41    spark-master-lint
 23    spark-master-compile-maven-hadoop-2.6
 32    spark-master-compile-maven-hadoop-2.7
 5    spark-branch-2.2-compile-maven-hadoop-2.6
 3    spark-branch-2.2-test-maven-hadoop-2.7
 6    spark-branch-2.2-compile-maven-scala-2.10
 6    spark-master-test-maven-hadoop-2.6
 3    spark-master-test-maven-hadoop-2.7
 14    spark-branch-2.3-compile-maven-hadoop-2.6
 3    spark-branch-2.2-lint
 1    spark-branch-2.3-test-maven-hadoop-2.7

Timeouts by day:
2018-01-09    4
2018-01-10    13
2018-01-11    27
2018-01-12    74
2018-01-13    9
2018-01-14    2
2018-01-15    8
2018-01-16    34

Total builds:    4112
Total timeouts:    171
Percentage of all builds timing out:    4.15856031128

On Wed, Jan 10, 2018 at 9:54 AM, shane knapp <[hidden email]> wrote:
i just noticed we're starting to see the once-yearly rash of git timeouts when building.

i'll be looking in to this today...  i'm at our lab retreat, so my attention will be divided during the day but i will report back here once i have some more information.

in the meantime, if your jobs have a git timeout, please just retrigger them and we will hope for the best.

shane

Reply | Threaded
Open this post in threaded view
|

Re: [build system] currently experiencing git timeouts when building

shane knapp
quick update:

it looks like the timeouts have stopped.  github finally got back to me about this, but (again) after they stopped happening.

i'll be keeping an eye on this for the next few days and will re-escalate if we start having them again.

shane

On Tue, Jan 16, 2018 at 1:18 PM, shane knapp <[hidden email]> wrote:
update:  we're seeing about 3% of builds timing out, but today is (so far) a bad one:

i've reached out to github about this and am waiting to hear back.

$ get_timeouts.py 10
Timeouts by project:
 17    spark-branch-2.3-lint
 10    spark-branch-2.3-test-maven-hadoop-2.6
 7    spark-branch-2.3-compile-maven-hadoop-2.7
 41    spark-master-lint
 23    spark-master-compile-maven-hadoop-2.6
 32    spark-master-compile-maven-hadoop-2.7
 5    spark-branch-2.2-compile-maven-hadoop-2.6
 3    spark-branch-2.2-test-maven-hadoop-2.7
 6    spark-branch-2.2-compile-maven-scala-2.10
 6    spark-master-test-maven-hadoop-2.6
 3    spark-master-test-maven-hadoop-2.7
 14    spark-branch-2.3-compile-maven-hadoop-2.6
 3    spark-branch-2.2-lint
 1    spark-branch-2.3-test-maven-hadoop-2.7

Timeouts by day:
2018-01-09    4
2018-01-10    13
2018-01-11    27
2018-01-12    74
2018-01-13    9
2018-01-14    2
2018-01-15    8
2018-01-16    34

Total builds:    4112
Total timeouts:    171
Percentage of all builds timing out:    4.<a href="tel:(585)%20603-1128" value="+15856031128" target="_blank">15856031128

On Wed, Jan 10, 2018 at 9:54 AM, shane knapp <[hidden email]> wrote:
i just noticed we're starting to see the once-yearly rash of git timeouts when building.

i'll be looking in to this today...  i'm at our lab retreat, so my attention will be divided during the day but i will report back here once i have some more information.

in the meantime, if your jobs have a git timeout, please just retrigger them and we will hope for the best.

shane