[VOTE] [SPIP] SPARK-18085: Better History Server scalability

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[VOTE] [SPIP] SPARK-18085: Better History Server scalability

Marcelo Vanzin
Hey all,

Following the SPIP process, I'm putting this SPIP up for a vote. It's
been open for comments as an SPIP for about 3 weeks now, and had been
open without the SPIP label for about 9 months before that. There has
been no new feedback since it was tagged as an SPIP, so I'm assuming
all the people who looked at it are OK with the current proposal.

The vote will be up for the next 72 hours. Please reply with your vote:

+1: Yeah, let's go forward and implement the SPIP.
+0: Don't really care.
-1: I don't think this is a good idea because of the following
technical reasons.

Thanks!

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Marcelo Vanzin
Adding my own +1 (binding).

On Mon, Jul 31, 2017 at 10:27 AM, Marcelo Vanzin <[hidden email]> wrote:

> Hey all,
>
> Following the SPIP process, I'm putting this SPIP up for a vote. It's
> been open for comments as an SPIP for about 3 weeks now, and had been
> open without the SPIP label for about 9 months before that. There has
> been no new feedback since it was tagged as an SPIP, so I'm assuming
> all the people who looked at it are OK with the current proposal.
>
> The vote will be up for the next 72 hours. Please reply with your vote:
>
> +1: Yeah, let's go forward and implement the SPIP.
> +0: Don't really care.
> -1: I don't think this is a good idea because of the following
> technical reasons.
>
> Thanks!
>
> --
> Marcelo



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Sean Owen
In reply to this post by Marcelo Vanzin
(Direct link to design doc, linked from JIRA)
https://issues.apache.org/jira/browse/SPARK-18085

I know Marcelo has looked closely at this issue for a long while and trust his judgment about what needs to be fixed, and how. I know he has a good view into practical pain points like this from customer deployments.

I read the whole doc. Although I have not followed the SHS issues closely, the recounting of issues makes sense to me, as well as the stated goals.

There is a considerable amount of change proposed here, but it's broken down into milestones. The change doesn't seem excessive given what needs to be improved -- it does need a rearchitecting.

Now, some minor comments

+1 to reusing a 'database' library already used by Spark.
Is 'spark-ui' too broad? doesn't sound like this module would actually house all the UIs. spark-shs-ui or something?
Good that this can be implemented in parallel to the existing mechanism for the initial milestones.
So the "Mx" milestones are essentially required, in your view? and the "SMx" are optional and stand-alone? 


So yes I have no concerns, trust the analysis, think it's a real problem that will take a sustained effort. +1


On Mon, Jul 31, 2017 at 6:28 PM Marcelo Vanzin <[hidden email]> wrote:
Hey all,

Following the SPIP process, I'm putting this SPIP up for a vote. It's
been open for comments as an SPIP for about 3 weeks now, and had been
open without the SPIP label for about 9 months before that. There has
been no new feedback since it was tagged as an SPIP, so I'm assuming
all the people who looked at it are OK with the current proposal.

The vote will be up for the next 72 hours. Please reply with your vote:

+1: Yeah, let's go forward and implement the SPIP.
+0: Don't really care.
-1: I don't think this is a good idea because of the following
technical reasons.

Thanks!

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Denis Bolshakov
+1

Absolutely agree on SPARK-18085.

On 1 August 2017 at 12:18, Sean Owen <[hidden email]> wrote:
(Direct link to design doc, linked from JIRA)
https://issues.apache.org/jira/browse/SPARK-18085

I know Marcelo has looked closely at this issue for a long while and trust his judgment about what needs to be fixed, and how. I know he has a good view into practical pain points like this from customer deployments.

I read the whole doc. Although I have not followed the SHS issues closely, the recounting of issues makes sense to me, as well as the stated goals.

There is a considerable amount of change proposed here, but it's broken down into milestones. The change doesn't seem excessive given what needs to be improved -- it does need a rearchitecting.

Now, some minor comments

+1 to reusing a 'database' library already used by Spark.
Is 'spark-ui' too broad? doesn't sound like this module would actually house all the UIs. spark-shs-ui or something?
Good that this can be implemented in parallel to the existing mechanism for the initial milestones.
So the "Mx" milestones are essentially required, in your view? and the "SMx" are optional and stand-alone? 


So yes I have no concerns, trust the analysis, think it's a real problem that will take a sustained effort. +1


On Mon, Jul 31, 2017 at 6:28 PM Marcelo Vanzin <[hidden email]> wrote:
Hey all,

Following the SPIP process, I'm putting this SPIP up for a vote. It's
been open for comments as an SPIP for about 3 weeks now, and had been
open without the SPIP label for about 9 months before that. There has
been no new feedback since it was tagged as an SPIP, so I'm assuming
all the people who looked at it are OK with the current proposal.

The vote will be up for the next 72 hours. Please reply with your vote:

+1: Yeah, let's go forward and implement the SPIP.
+0: Don't really care.
-1: I don't think this is a good idea because of the following
technical reasons.

Thanks!

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]




--
//with Best Regards
--Denis Bolshakov
e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Hyukjin Kwon
+1

Although I am not used to this code path,  I read the proposal few times and it makes sense to me.

On 1 Aug 2017 7:01 pm, "Denis Bolshakov" <[hidden email]> wrote:
+1

Absolutely agree on SPARK-18085.

On 1 August 2017 at 12:18, Sean Owen <[hidden email]> wrote:
(Direct link to design doc, linked from JIRA)
https://issues.apache.org/jira/browse/SPARK-18085

I know Marcelo has looked closely at this issue for a long while and trust his judgment about what needs to be fixed, and how. I know he has a good view into practical pain points like this from customer deployments.

I read the whole doc. Although I have not followed the SHS issues closely, the recounting of issues makes sense to me, as well as the stated goals.

There is a considerable amount of change proposed here, but it's broken down into milestones. The change doesn't seem excessive given what needs to be improved -- it does need a rearchitecting.

Now, some minor comments

+1 to reusing a 'database' library already used by Spark.
Is 'spark-ui' too broad? doesn't sound like this module would actually house all the UIs. spark-shs-ui or something?
Good that this can be implemented in parallel to the existing mechanism for the initial milestones.
So the "Mx" milestones are essentially required, in your view? and the "SMx" are optional and stand-alone? 


So yes I have no concerns, trust the analysis, think it's a real problem that will take a sustained effort. +1


On Mon, Jul 31, 2017 at 6:28 PM Marcelo Vanzin <[hidden email]> wrote:
Hey all,

Following the SPIP process, I'm putting this SPIP up for a vote. It's
been open for comments as an SPIP for about 3 weeks now, and had been
open without the SPIP label for about 9 months before that. There has
been no new feedback since it was tagged as an SPIP, so I'm assuming
all the people who looked at it are OK with the current proposal.

The vote will be up for the next 72 hours. Please reply with your vote:

+1: Yeah, let's go forward and implement the SPIP.
+0: Don't really care.
-1: I don't think this is a good idea because of the following
technical reasons.

Thanks!

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]




--
//with Best Regards
--Denis Bolshakov
e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Tom Graves-2
In reply to this post by Marcelo Vanzin
+1. 


Tom



On Monday, July 31, 2017, 12:28:02 PM CDT, Marcelo Vanzin <[hidden email]> wrote:


Hey all,

Following the SPIP process, I'm putting this SPIP up for a vote. It's
been open for comments as an SPIP for about 3 weeks now, and had been
open without the SPIP label for about 9 months before that. There has
been no new feedback since it was tagged as an SPIP, so I'm assuming
all the people who looked at it are OK with the current proposal.

The vote will be up for the next 72 hours. Please reply with your vote:

+1: Yeah, let's go forward and implement the SPIP.
+0: Don't really care.
-1: I don't think this is a good idea because of the following
technical reasons.

Thanks!

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Ryan Blue
+1 (non-binding)

On Tue, Aug 1, 2017 at 6:48 AM, Tom Graves <[hidden email]> wrote:
+1. 


Tom



On Monday, July 31, 2017, 12:28:02 PM CDT, Marcelo Vanzin <[hidden email]> wrote:


Hey all,

Following the SPIP process, I'm putting this SPIP up for a vote. It's
been open for comments as an SPIP for about 3 weeks now, and had been
open without the SPIP label for about 9 months before that. There has
been no new feedback since it was tagged as an SPIP, so I'm assuming
all the people who looked at it are OK with the current proposal.

The vote will be up for the next 72 hours. Please reply with your vote:

+1: Yeah, let's go forward and implement the SPIP.
+0: Don't really care.
-1: I don't think this is a good idea because of the following
technical reasons.

Thanks!

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]




--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Dong Joon Hyun

+1 (non-binding)

 

Dongjoon.

 

From: Ryan Blue <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Tuesday, August 1, 2017 at 9:06 AM
To: Tom Graves <[hidden email]>
Cc: Marcelo Vanzin <[hidden email]>, "[hidden email]" <[hidden email]>
Subject: Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

 

+1 (non-binding)

 

On Tue, Aug 1, 2017 at 6:48 AM, Tom Graves <[hidden email]> wrote:

+1. 

 

 

Tom

 

 

 

On Monday, July 31, 2017, 12:28:02 PM CDT, Marcelo Vanzin <[hidden email]> wrote:

 

 

Hey all,

 

Following the SPIP process, I'm putting this SPIP up for a vote. It's

been open for comments as an SPIP for about 3 weeks now, and had been

open without the SPIP label for about 9 months before that. There has

been no new feedback since it was tagged as an SPIP, so I'm assuming

all the people who looked at it are OK with the current proposal.

 

The vote will be up for the next 72 hours. Please reply with your vote:

 

+1: Yeah, let's go forward and implement the SPIP.

+0: Don't really care.

-1: I don't think this is a good idea because of the following

technical reasons.

 

Thanks!

 

--

Marcelo

 

---------------------------------------------------------------------

To unsubscribe e-mail: [hidden email]

 



 

--

Ryan Blue

Software Engineer

Netflix

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Marcelo Vanzin
In reply to this post by Sean Owen
Thanks all for the comments. Just a clarification:

On Tue, Aug 1, 2017 at 2:18 AM, Sean Owen <[hidden email]> wrote:
> Is 'spark-ui' too broad? doesn't sound like this module would actually house
> all the UIs. spark-shs-ui or something?
> Good that this can be implemented in parallel to the existing mechanism for
> the initial milestones.
> So the "Mx" milestones are essentially required, in your view? and the "SMx"
> are optional and stand-alone?

The spec attached to the bug was written before I had a chance to work
on most of the implementation, so it doesn't accurately reflect the
current status of the code. The major changes are all there, but I
changed some details when I ran into issues during implementation that
I thought would end up being blockers. The two main changes from the
spec are:

- I gave up separating the UI into a separate module for now. It would
probably break someone's workflow, and it made it really hard to track
UI changes made concurrently. With the current approach I know I'm not
missing anything that was added to the UI while I was writing that
code.

- While doing some perf testing I decided to make some optional
milestones required; mainly, having an in-memory implementation, and
having that be the default. This also keeps the current behavior as
the default, and people have to opt-in to using the new code.

- In spite of calling it out as a non-goal, I found that at least the
current stage page became really slow with a LevelDB backend with
large jobs (it's already slow now, but it got worse), so I included
SPARK-20657 in the things I think are needed for the first
implementation.

That should be all.

--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Marcelo Vanzin
In reply to this post by Marcelo Vanzin
This vote passes with 3 binding +1 votes, 5 non-binding votes, and no -1 votes.

Thanks all!

+1 votes (binding):
Tom Graves
Sean Owen
Marcelo Vanzin

+1 votes (non-binding):
Ryan Blue
Denis Bolshakov
Dong Joon Hyun
Hyukjin Kwon
Ashutosh Pathak


On Mon, Jul 31, 2017 at 10:27 AM, Marcelo Vanzin <[hidden email]> wrote:

> Hey all,
>
> Following the SPIP process, I'm putting this SPIP up for a vote. It's
> been open for comments as an SPIP for about 3 weeks now, and had been
> open without the SPIP label for about 9 months before that. There has
> been no new feedback since it was tagged as an SPIP, so I'm assuming
> all the people who looked at it are OK with the current proposal.
>
> The vote will be up for the next 72 hours. Please reply with your vote:
>
> +1: Yeah, let's go forward and implement the SPIP.
> +0: Don't really care.
> -1: I don't think this is a good idea because of the following
> technical reasons.
>
> Thanks!
>
> --
> Marcelo



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

rxin
A late +1 too. 

On Thu, Aug 3, 2017 at 1:37 PM Marcelo Vanzin <[hidden email]> wrote:
This vote passes with 3 binding +1 votes, 5 non-binding votes, and no -1 votes.

Thanks all!

+1 votes (binding):
Tom Graves
Sean Owen
Marcelo Vanzin

+1 votes (non-binding):
Ryan Blue
Denis Bolshakov
Dong Joon Hyun
Hyukjin Kwon
Ashutosh Pathak


On Mon, Jul 31, 2017 at 10:27 AM, Marcelo Vanzin <[hidden email]> wrote:
> Hey all,
>
> Following the SPIP process, I'm putting this SPIP up for a vote. It's
> been open for comments as an SPIP for about 3 weeks now, and had been
> open without the SPIP label for about 9 months before that. There has
> been no new feedback since it was tagged as an SPIP, so I'm assuming
> all the people who looked at it are OK with the current proposal.
>
> The vote will be up for the next 72 hours. Please reply with your vote:
>
> +1: Yeah, let's go forward and implement the SPIP.
> +0: Don't really care.
> -1: I don't think this is a good idea because of the following
> technical reasons.
>
> Thanks!
>
> --
> Marcelo



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Loading...