Build timed out for `branch-2.3 (hadoop-2.7)`

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Build timed out for `branch-2.3 (hadoop-2.7)`

Dongjoon Hyun-2
Hi, All and Shane.

Can we increase the build time for `branch-2.3` during 2.3 RC period?

There are two known test issues, but the Jenkins on branch-2.3 with hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether the branch is healthy or not.
Reply | Threaded
Open this post in threaded view
|

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

xin lu
seems like someone should investigate what caused the build time to go up an hour and if it's expected or not.  

On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun <[hidden email]> wrote:
Hi, All and Shane.

Can we increase the build time for `branch-2.3` during 2.3 RC period?

There are two known test issues, but the Jenkins on branch-2.3 with hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether the branch is healthy or not.

Reply | Threaded
Open this post in threaded view
|

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

Shixiong(Ryan) Zhu

On Fri, Jan 12, 2018 at 11:45 AM, Xin Lu <[hidden email]> wrote:
seems like someone should investigate what caused the build time to go up an hour and if it's expected or not.  

On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun <[hidden email]> wrote:
Hi, All and Shane.

Can we increase the build time for `branch-2.3` during 2.3 RC period?

There are two known test issues, but the Jenkins on branch-2.3 with hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether the branch is healthy or not.


Reply | Threaded
Open this post in threaded view
|

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

Dongjoon Hyun-2
For this issue, during SPARK-23028, Shane shared that the server limit is already higher.

1. Xiao Li increased the timeout of Spark test script for `master` branch first in the following commit.

[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT

2. Marco Gaido reports a flaky test suite and it turns out that the test suite hangs in SPARK-23055

3. Sameer Agarwal swiftly reverts it.

Thank you all!

Let's wait and see the dashboard.

Bests,
Dongjoon.



On Fri, Jan 12, 2018 at 3:22 PM, Shixiong(Ryan) Zhu <[hidden email]> wrote:

On Fri, Jan 12, 2018 at 11:45 AM, Xin Lu <[hidden email]> wrote:
seems like someone should investigate what caused the build time to go up an hour and if it's expected or not.  

On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun <[hidden email]> wrote:
Hi, All and Shane.

Can we increase the build time for `branch-2.3` during 2.3 RC period?

There are two known test issues, but the Jenkins on branch-2.3 with hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether the branch is healthy or not.



Reply | Threaded
Open this post in threaded view
|

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

Sameer Agarwal-2
FYI, I ended up bumping the build timeouts from 255 to 275 minutes. All successful 2.3 (hadoop-2.7) builds last week were already taking 245-250 mins and had started timing out earlier today (towards the very end; while making consistent progress throughout). Increasing the timeout resolves the issue.

NB: This might be either due to additional tests that were recently added or due to the git delays that Shane reported; we haven't investigated the root cause yet.

On 12 January 2018 at 16:37, Dongjoon Hyun <[hidden email]> wrote:
For this issue, during SPARK-23028, Shane shared that the server limit is already higher.

1. Xiao Li increased the timeout of Spark test script for `master` branch first in the following commit.

[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT

2. Marco Gaido reports a flaky test suite and it turns out that the test suite hangs in SPARK-23055

3. Sameer Agarwal swiftly reverts it.

Thank you all!

Let's wait and see the dashboard.

Bests,
Dongjoon.



On Fri, Jan 12, 2018 at 3:22 PM, Shixiong(Ryan) Zhu <[hidden email]> wrote:

On Fri, Jan 12, 2018 at 11:45 AM, Xin Lu <[hidden email]> wrote:
seems like someone should investigate what caused the build time to go up an hour and if it's expected or not.  

On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun <[hidden email]> wrote:
Hi, All and Shane.

Can we increase the build time for `branch-2.3` during 2.3 RC period?

There are two known test issues, but the Jenkins on branch-2.3 with hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether the branch is healthy or not.




Reply | Threaded
Open this post in threaded view
|

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

shane knapp
this doesn't have anything to do w/the git timeouts...  those will timeout the build 10 mins after starting (and failing on the initial fetch call).

On Wed, Jan 17, 2018 at 9:51 PM, Sameer Agarwal <[hidden email]> wrote:
FYI, I ended up bumping the build timeouts from 255 to 275 minutes. All successful 2.3 (hadoop-2.7) builds last week were already taking 245-250 mins and had started timing out earlier today (towards the very end; while making consistent progress throughout). Increasing the timeout resolves the issue.

NB: This might be either due to additional tests that were recently added or due to the git delays that Shane reported; we haven't investigated the root cause yet.

On 12 January 2018 at 16:37, Dongjoon Hyun <[hidden email]> wrote:
For this issue, during SPARK-23028, Shane shared that the server limit is already higher.

1. Xiao Li increased the timeout of Spark test script for `master` branch first in the following commit.

[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT

2. Marco Gaido reports a flaky test suite and it turns out that the test suite hangs in SPARK-23055

3. Sameer Agarwal swiftly reverts it.

Thank you all!

Let's wait and see the dashboard.

Bests,
Dongjoon.



On Fri, Jan 12, 2018 at 3:22 PM, Shixiong(Ryan) Zhu <[hidden email]> wrote:

On Fri, Jan 12, 2018 at 11:45 AM, Xin Lu <[hidden email]> wrote:
seems like someone should investigate what caused the build time to go up an hour and if it's expected or not.  

On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun <[hidden email]> wrote:
Hi, All and Shane.

Can we increase the build time for `branch-2.3` during 2.3 RC period?

There are two known test issues, but the Jenkins on branch-2.3 with hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether the branch is healthy or not.