Timeline for Spark 2.3

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Timeline for Spark 2.3

Michael Armbrust
According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.

1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing.
3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.

Thoughts?

Michael
Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Justin Miller
That sounds fine to me. I’m hoping that this ticket can make it into Spark 2.3: https://issues.apache.org/jira/browse/SPARK-18016

It’s causing some pretty considerable problems when we alter the columns to be nullable, but we are OK for now without that.

Best,
Justin

On Nov 9, 2017, at 4:54 PM, Michael Armbrust <[hidden email]> wrote:

According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.

1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing.
3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.

Thoughts?

Michael

Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Anirudh Ramanathan-2
This would help the community on the Kubernetes effort quite a bit - giving us additional time for reviews and testing for the 2.3 release.

On Thu, Nov 9, 2017 at 3:56 PM, Justin Miller <[hidden email]> wrote:
That sounds fine to me. I’m hoping that this ticket can make it into Spark 2.3: https://issues.apache.org/jira/browse/SPARK-18016

It’s causing some pretty considerable problems when we alter the columns to be nullable, but we are OK for now without that.

Best,
Justin

On Nov 9, 2017, at 4:54 PM, Michael Armbrust <[hidden email]> wrote:

According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.

1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing.
3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.

Thoughts?

Michael


Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Matei Zaharia
Administrator
I’m also +1 on extending this to get Kubernetes and other features in.

Matei

> On Nov 9, 2017, at 4:04 PM, Anirudh Ramanathan <[hidden email]> wrote:
>
> This would help the community on the Kubernetes effort quite a bit - giving us additional time for reviews and testing for the 2.3 release.
>
> On Thu, Nov 9, 2017 at 3:56 PM, Justin Miller <[hidden email]> wrote:
> That sounds fine to me. I’m hoping that this ticket can make it into Spark 2.3: https://issues.apache.org/jira/browse/SPARK-18016
>
> It’s causing some pretty considerable problems when we alter the columns to be nullable, but we are OK for now without that.
>
> Best,
> Justin
>
>> On Nov 9, 2017, at 4:54 PM, Michael Armbrust <[hidden email]> wrote:
>>
>> According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.
>>
>> 1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
>> 2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing.
>> 3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.
>>
>> Thoughts?
>>
>> Michael
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Erik Erlandson-2
+1 on extending the deadline. It will significantly improve the logistics for upstreaming the Kubernetes back-end.  Also agreed, on the general realities of reduced bandwidth over the Nov-Dec holiday season.
Erik

On Thu, Nov 9, 2017 at 6:03 PM, Matei Zaharia <[hidden email]> wrote:
I’m also +1 on extending this to get Kubernetes and other features in.

Matei

> On Nov 9, 2017, at 4:04 PM, Anirudh Ramanathan <[hidden email]> wrote:
>
> This would help the community on the Kubernetes effort quite a bit - giving us additional time for reviews and testing for the 2.3 release.
>
> On Thu, Nov 9, 2017 at 3:56 PM, Justin Miller <[hidden email]> wrote:
> That sounds fine to me. I’m hoping that this ticket can make it into Spark 2.3: https://issues.apache.org/jira/browse/SPARK-18016
>
> It’s causing some pretty considerable problems when we alter the columns to be nullable, but we are OK for now without that.
>
> Best,
> Justin
>
>> On Nov 9, 2017, at 4:54 PM, Michael Armbrust <[hidden email]> wrote:
>>
>> According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.
>>
>> 1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
>> 2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing.
>> 3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.
>>
>> Thoughts?
>>
>> Michael
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Nick Pentreath
+1 I think that’s practical 

On Fri, 10 Nov 2017 at 03:13, Erik Erlandson <[hidden email]> wrote:
+1 on extending the deadline. It will significantly improve the logistics for upstreaming the Kubernetes back-end.  Also agreed, on the general realities of reduced bandwidth over the Nov-Dec holiday season.
Erik

On Thu, Nov 9, 2017 at 6:03 PM, Matei Zaharia <[hidden email]> wrote:
I’m also +1 on extending this to get Kubernetes and other features in.

Matei

> On Nov 9, 2017, at 4:04 PM, Anirudh Ramanathan <[hidden email]> wrote:
>
> This would help the community on the Kubernetes effort quite a bit - giving us additional time for reviews and testing for the 2.3 release.
>
> On Thu, Nov 9, 2017 at 3:56 PM, Justin Miller <[hidden email]> wrote:
> That sounds fine to me. I’m hoping that this ticket can make it into Spark 2.3: https://issues.apache.org/jira/browse/SPARK-18016
>
> It’s causing some pretty considerable problems when we alter the columns to be nullable, but we are OK for now without that.
>
> Best,
> Justin
>
>> On Nov 9, 2017, at 4:54 PM, Michael Armbrust <[hidden email]> wrote:
>>
>> According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.
>>
>> 1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
>> 2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing.
>> 3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.
>>
>> Thoughts?
>>
>> Michael
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Marco Gaido
I would love too to have SPARK-18016. I think it would help a lot of users.

2017-11-10 5:58 GMT+01:00 Nick Pentreath <[hidden email]>:
+1 I think that’s practical 

On Fri, 10 Nov 2017 at 03:13, Erik Erlandson <[hidden email]> wrote:
+1 on extending the deadline. It will significantly improve the logistics for upstreaming the Kubernetes back-end.  Also agreed, on the general realities of reduced bandwidth over the Nov-Dec holiday season.
Erik

On Thu, Nov 9, 2017 at 6:03 PM, Matei Zaharia <[hidden email]> wrote:
I’m also +1 on extending this to get Kubernetes and other features in.

Matei

> On Nov 9, 2017, at 4:04 PM, Anirudh Ramanathan <[hidden email]> wrote:
>
> This would help the community on the Kubernetes effort quite a bit - giving us additional time for reviews and testing for the 2.3 release.
>
> On Thu, Nov 9, 2017 at 3:56 PM, Justin Miller <[hidden email]> wrote:
> That sounds fine to me. I’m hoping that this ticket can make it into Spark 2.3: https://issues.apache.org/jira/browse/SPARK-18016
>
> It’s causing some pretty considerable problems when we alter the columns to be nullable, but we are OK for now without that.
>
> Best,
> Justin
>
>> On Nov 9, 2017, at 4:54 PM, Michael Armbrust <[hidden email]> wrote:
>>
>> According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.
>>
>> 1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
>> 2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing.
>> 3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.
>>
>> Thoughts?
>>
>> Michael
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Sean Owen
In reply to this post by Michael Armbrust
The original timeline was just +6 months from last planned release, so there was nothing too magic about it. That was pushed from +4 . The only risk here is that an extra month becomes 2, 3, and so users aren't getting the other 1000 fixes. But no particular problem with moving it back. 

On Thu, Nov 9, 2017, 5:54 PM Michael Armbrust <[hidden email]> wrote:
According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.

1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing
3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.

Thoughts?

Michael
Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Sameer Agarwal
Sounds good to me too. In addition to what has already been pointed out about the Spark History Server and the Kubernetes support, this would also give us enough time to further polish the new data source v2 API and the vectorized UDF API to iron out any kinks.

I'd like to volunteer to serve as the release manager for Spark 2.3. In terms of bandwidth, I will be available during this holiday season as I have no vacation planed during the Dec-Jan timeframe. And I'm fairly familiar with most of the major efforts targeted for the 2.3 release.

Thanks,
Sameer

On Fri, Nov 10, 2017 at 2:07 AM, Sean Owen <[hidden email]> wrote:
The original timeline was just +6 months from last planned release, so there was nothing too magic about it. That was pushed from +4 . The only risk here is that an extra month becomes 2, 3, and so users aren't getting the other 1000 fixes. But no particular problem with moving it back. 

On Thu, Nov 9, 2017, 5:54 PM Michael Armbrust <[hidden email]> wrote:
According to the timeline posted on the website, we are nearing branch cut for Spark 2.3.  I'd like to propose pushing this out towards mid to late December for a couple of reasons and would like to hear what people think.

1. I've done release management during the Thanksgiving / Christmas time before and in my experience, we don't actually get a lot of testing during this time due to vacations and other commitments. I think beginning the RC process in early January would give us the best coverage in the shortest amount of time.
2. There are several large initiatives in progress that given a little more time would leave us with a much more exciting 2.3 release. Specifically, the work on the history server, Kubernetes and continuous processing
3. Given the actual release date of Spark 2.2, I think we'll still get Spark 2.3 out roughly 6 months after.

Thoughts?

Michael



--
Sameer Agarwal
Software Engineer | Databricks Inc.
Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 2.3

Dong Jiang
Hi,

What is the process to request an issue/fix to be included in the next
release? Is there a place to vote for features?
I am interested in https://issues.apache.org/jira/browse/SPARK-13127, to see
if we can get Spark upgrade parquet to 1.9.0, which addresses the
https://issues.apache.org/jira/browse/PARQUET-686.
Can we include the fix in Spark 2.3 release?

Thanks,

Dong



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]