Build time limit in PR builder

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Build time limit in PR builder

Hyukjin Kwon
Hi all,

Seems we're hitting the time limit in PR builders (see https://github.com/apache/spark/pull/28627), in particular wen it's Maven build which takes more time compared to SBT in general.
Should we maybe increase the PR builder a bit more (10 ~ 20 mins?) to unblock these PRs and focus on cutting the build/testing time?

Reply | Threaded
Open this post in threaded view
|

Re: Build time limit in PR builder

Sean Owen-2
What else can we do, I suppose?

On Thu, May 28, 2020 at 3:13 AM Hyukjin Kwon <[hidden email]> wrote:
Hi all,

Seems we're hitting the time limit in PR builders (see https://github.com/apache/spark/pull/28627), in particular wen it's Maven build which takes more time compared to SBT in general.
Should we maybe increase the PR builder a bit more (10 ~ 20 mins?) to unblock these PRs and focus on cutting the build/testing time?

Reply | Threaded
Open this post in threaded view
|

Re: Build time limit in PR builder

shane knapp ☠
On Thu, May 28, 2020 at 7:16 AM Sean Owen <[hidden email]> wrote:
What else can we do, I suppose?

there,s not much else we can do.  i'll add 30m to the timeout.

shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Reply | Threaded
Open this post in threaded view
|

Re: Build time limit in PR builder

Hyukjin Kwon
I remember we were able to cut down pretty considerably in the past. For example, I investigated (https://github.com/apache/spark/pull/21822#issuecomment-407295739)
and fixed some before at, like https://github.com/apache/spark/pull/23111. Maybe we could skim again to reduce the build/testing time ..

But I agree with increasing it at this moment for now to unblock PRs. Don't forget to fix here too https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201 :-).

2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <[hidden email]>님이 작성:
On Thu, May 28, 2020 at 7:16 AM Sean Owen <[hidden email]> wrote:
What else can we do, I suppose?

there,s not much else we can do.  i'll add 30m to the timeout.

shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Reply | Threaded
Open this post in threaded view
|

Re: Build time limit in PR builder

shane knapp ☠
i'll get a PR put together now.  

On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <[hidden email]> wrote:
I remember we were able to cut down pretty considerably in the past. For example, I investigated (https://github.com/apache/spark/pull/21822#issuecomment-407295739)
and fixed some before at, like https://github.com/apache/spark/pull/23111. Maybe we could skim again to reduce the build/testing time ..

But I agree with increasing it at this moment for now to unblock PRs. Don't forget to fix here too https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201 :-).

2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <[hidden email]>님이 작성:
On Thu, May 28, 2020 at 7:16 AM Sean Owen <[hidden email]> wrote:
What else can we do, I suppose?

there,s not much else we can do.  i'll add 30m to the timeout.

shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Reply | Threaded
Open this post in threaded view
|

Re: Build time limit in PR builder

shane knapp ☠

On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <[hidden email]> wrote:
i'll get a PR put together now.  

On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <[hidden email]> wrote:
I remember we were able to cut down pretty considerably in the past. For example, I investigated (https://github.com/apache/spark/pull/21822#issuecomment-407295739)
and fixed some before at, like https://github.com/apache/spark/pull/23111. Maybe we could skim again to reduce the build/testing time ..

But I agree with increasing it at this moment for now to unblock PRs. Don't forget to fix here too https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201 :-).

2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <[hidden email]>님이 작성:
On Thu, May 28, 2020 at 7:16 AM Sean Owen <[hidden email]> wrote:
What else can we do, I suppose?

there,s not much else we can do.  i'll add 30m to the timeout.

shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
Reply | Threaded
Open this post in threaded view
|

Re: Build time limit in PR builder

Kousuke Saruta-2
Thanks all. It's very helpful!

- Kousuke

On 2020/05/29 3:31, shane knapp ☠ wrote:

On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <[hidden email]> wrote:
i'll get a PR put together now.  

On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <[hidden email]> wrote:
I remember we were able to cut down pretty considerably in the past. For example, I investigated (https://github.com/apache/spark/pull/21822#issuecomment-407295739)
and fixed some before at, like https://github.com/apache/spark/pull/23111. Maybe we could skim again to reduce the build/testing time ..

But I agree with increasing it at this moment for now to unblock PRs. Don't forget to fix here too https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201 :-).

2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <[hidden email]>님이 작성:
On Thu, May 28, 2020 at 7:16 AM Sean Owen <[hidden email]> wrote:
What else can we do, I suppose?

there,s not much else we can do.  i'll add 30m to the timeout.

shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu

Reply | Threaded
Open this post in threaded view
|

Re: Build time limit in PR builder

shane knapp ☠
the timer is set to 500m now in master, 3.0 and 2.4.

On Thu, May 28, 2020 at 12:32 PM Kousuke Saruta <[hidden email]> wrote:
Thanks all. It's very helpful!

- Kousuke

On 2020/05/29 3:31, shane knapp ☠ wrote:

On Thu, May 28, 2020 at 11:20 AM shane knapp ☠ <[hidden email]> wrote:
i'll get a PR put together now.  

On Thu, May 28, 2020 at 8:26 AM Hyukjin Kwon <[hidden email]> wrote:
I remember we were able to cut down pretty considerably in the past. For example, I investigated (https://github.com/apache/spark/pull/21822#issuecomment-407295739)
and fixed some before at, like https://github.com/apache/spark/pull/23111. Maybe we could skim again to reduce the build/testing time ..

But I agree with increasing it at this moment for now to unblock PRs. Don't forget to fix here too https://github.com/apache/spark/blob/master/dev/run-tests-jenkins.py#L201 :-).

2020년 5월 29일 (금) 오전 12:14, shane knapp ☠ <[hidden email]>님이 작성:
On Thu, May 28, 2020 at 7:16 AM Sean Owen <[hidden email]> wrote:
What else can we do, I suppose?

there,s not much else we can do.  i'll add 30m to the timeout.

shane
--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu


--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu



--
Shane Knapp
Computer Guy / Voice of Reason
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu