[Vote] SPIP: Continuous Processing Mode for Structured Streaming

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

[Vote] SPIP: Continuous Processing Mode for Structured Streaming

rxin
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.


Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

debasish83
+1

Is there any design doc related to API/internal changes ? Will CP be the default in structured streaming or it's a mode in conjunction with exisiting behavior.

Thanks.
Deb

On Nov 1, 2017 8:37 AM, "Reynold Xin" <[hidden email]> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.



Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

rxin
Most of those should be answered by the attached design sketch in the JIRA ticket. 

On Wed, Nov 1, 2017 at 5:29 PM Debasish Das <[hidden email]> wrote:
+1

Is there any design doc related to API/internal changes ? Will CP be the default in structured streaming or it's a mode in conjunction with exisiting behavior.

Thanks.
Deb

On Nov 1, 2017 8:37 AM, "Reynold Xin" <[hidden email]> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.



Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Cody Koeninger-2
Was there any answer to my question around the effect of changes to
the sink api regarding access to underlying offsets?

On Wed, Nov 1, 2017 at 11:32 AM, Reynold Xin <[hidden email]> wrote:

> Most of those should be answered by the attached design sketch in the JIRA
> ticket.
>
> On Wed, Nov 1, 2017 at 5:29 PM Debasish Das <[hidden email]>
> wrote:
>>
>> +1
>>
>> Is there any design doc related to API/internal changes ? Will CP be the
>> default in structured streaming or it's a mode in conjunction with exisiting
>> behavior.
>>
>> Thanks.
>> Deb
>>
>> On Nov 1, 2017 8:37 AM, "Reynold Xin" <[hidden email]> wrote:
>>
>> Earlier I sent out a discussion thread for CP in Structured Streaming:
>>
>> https://issues.apache.org/jira/browse/SPARK-20928
>>
>> It is meant to be a very small, surgical change to Structured Streaming to
>> enable ultra-low latency. This is great timing because we are also designing
>> and implementing data source API v2. If designed properly, we can have the
>> same data source API working for both streaming and batch.
>>
>>
>> Following the SPIP process, I'm putting this SPIP up for a vote.
>>
>> +1: Let's go ahead and design / implement the SPIP.
>> +0: Don't really care.
>> -1: I do not think this is a good idea for the following reasons.
>>
>>
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

rxin
I just replied.


On Wed, Nov 1, 2017 at 5:50 PM, Cody Koeninger <[hidden email]> wrote:
Was there any answer to my question around the effect of changes to
the sink api regarding access to underlying offsets?

On Wed, Nov 1, 2017 at 11:32 AM, Reynold Xin <[hidden email]> wrote:
> Most of those should be answered by the attached design sketch in the JIRA
> ticket.
>
> On Wed, Nov 1, 2017 at 5:29 PM Debasish Das <[hidden email]>
> wrote:
>>
>> +1
>>
>> Is there any design doc related to API/internal changes ? Will CP be the
>> default in structured streaming or it's a mode in conjunction with exisiting
>> behavior.
>>
>> Thanks.
>> Deb
>>
>> On Nov 1, 2017 8:37 AM, "Reynold Xin" <[hidden email]> wrote:
>>
>> Earlier I sent out a discussion thread for CP in Structured Streaming:
>>
>> https://issues.apache.org/jira/browse/SPARK-20928
>>
>> It is meant to be a very small, surgical change to Structured Streaming to
>> enable ultra-low latency. This is great timing because we are also designing
>> and implementing data source API v2. If designed properly, we can have the
>> same data source API working for both streaming and batch.
>>
>>
>> Following the SPIP process, I'm putting this SPIP up for a vote.
>>
>> +1: Let's go ahead and design / implement the SPIP.
>> +0: Don't really care.
>> -1: I do not think this is a good idea for the following reasons.
>>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Noman Khan
In reply to this post by rxin
+1
for ultra-low latency

Thanks & Regards
Noman

From: Reynold Xin
Sent: Wednesday, 1 November, 21:07
Subject: [Vote] SPIP: Continuous Processing Mode for Structured Streaming
Earlier I sent out a discussion thread for CP in Structured Streaming:

It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.




Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Sean Owen
In reply to this post by rxin
+0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.

On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.


Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

cloud0fan
+1.

I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.

On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
+0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.


On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.



Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Matei Zaharia
Administrator
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Weichen Xu
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

vaquarkhan
+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu <[hidden email]> wrote:
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]





--
Regards,
Vaquar Khan
+1 -224-436-0783
Greater Chicago
Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Burak Yavuz-2
+1

On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan <[hidden email]> wrote:
+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu <[hidden email]> wrote:
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]





--
Regards,
Vaquar Khan
<a href="tel:(224)%20436-0783" value="+12244360783" target="_blank">+1 -224-436-0783
Greater Chicago

Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Xiao Li
+1

2017-11-04 11:00 GMT-07:00 Burak Yavuz <[hidden email]>:
+1

On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan <[hidden email]> wrote:
+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu <[hidden email]> wrote:
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]





--
Regards,
Vaquar Khan
<a href="tel:(224)%20436-0783" value="+12244360783" target="_blank">+1 -224-436-0783
Greater Chicago


Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Tom Graves-2
In reply to this post by debasish83
+1 for the idea and feature, but I think the design is definitely lacking detail on the internal changes needed and how the execution pieces work and the communication.  Are you planning on posting more of those details or were you just planning on discussing in PR?

Tom

On Wednesday, November 1, 2017, 11:29:21 AM CDT, Debasish Das <[hidden email]> wrote:


+1

Is there any design doc related to API/internal changes ? Will CP be the default in structured streaming or it's a mode in conjunction with exisiting behavior.

Thanks.
Deb

On Nov 1, 2017 8:37 AM, "Reynold Xin" <[hidden email]> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.



Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

rxin
Thanks Tom. I'd imagine more details belong either in a full design doc, or a PR description. Might make sense to do an additional design doc, if there is enough delta from the current sketch doc.


On Mon, Nov 6, 2017 at 7:29 AM, Tom Graves <[hidden email]> wrote:
+1 for the idea and feature, but I think the design is definitely lacking detail on the internal changes needed and how the execution pieces work and the communication.  Are you planning on posting more of those details or were you just planning on discussing in PR?

Tom

On Wednesday, November 1, 2017, 11:29:21 AM CDT, Debasish Das <[hidden email]> wrote:


+1

Is there any design doc related to API/internal changes ? Will CP be the default in structured streaming or it's a mode in conjunction with exisiting behavior.

Thanks.
Deb

On Nov 1, 2017 8:37 AM, "Reynold Xin" <[hidden email]> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.




Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Michael Armbrust
In reply to this post by Xiao Li
+1

On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li <[hidden email]> wrote:
+1

2017-11-04 11:00 GMT-07:00 Burak Yavuz <[hidden email]>:
+1

On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan <[hidden email]> wrote:
+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu <[hidden email]> wrote:
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]





--
Regards,
Vaquar Khan
<a href="tel:(224)%20436-0783" value="+12244360783" target="_blank">+1 -224-436-0783
Greater Chicago



Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Joseph Bradley
+1

On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust <[hidden email]> wrote:
+1

On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li <[hidden email]> wrote:
+1

2017-11-04 11:00 GMT-07:00 Burak Yavuz <[hidden email]>:
+1

On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan <[hidden email]> wrote:
+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu <[hidden email]> wrote:
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]





--
Regards,
Vaquar Khan
<a href="tel:(224)%20436-0783" value="+12244360783" target="_blank">+1 -224-436-0783
Greater Chicago






--

Joseph Bradley

Software Engineer - Machine Learning

Databricks, Inc.

http://databricks.com

Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Shixiong(Ryan) Zhu
+1

On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley <[hidden email]> wrote:
+1

On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust <[hidden email]> wrote:
+1

On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li <[hidden email]> wrote:
+1

2017-11-04 11:00 GMT-07:00 Burak Yavuz <[hidden email]>:
+1

On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan <[hidden email]> wrote:
+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu <[hidden email]> wrote:
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]





--
Regards,
Vaquar Khan
<a href="tel:(224)%20436-0783" value="+12244360783" target="_blank">+1 -224-436-0783
Greater Chicago






--

Joseph Bradley

Software Engineer - Machine Learning

Databricks, Inc.

http://databricks.com


Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

rxin
In reply to this post by rxin
The vote has passed with the following +1s:

Reynold Xin*
Debasish Das
Noman Khan
Wenchen Fan*
Matei Zaharia*
Weichen Xu
Vaquar Khan
Burak Yavuz
Xiao Li
Tom Graves*
Michael Armbrust*
Joseph Bradley*
Shixiong Zhu*


And the following +0s:

Sean Owen*


Thanks for the feedback!


On Wed, Nov 1, 2017 at 8:37 AM, Reynold Xin <[hidden email]> wrote:
Earlier I sent out a discussion thread for CP in Structured Streaming:


It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.


Following the SPIP process, I'm putting this SPIP up for a vote.

+1: Let's go ahead and design / implement the SPIP.
+0: Don't really care.
-1: I do not think this is a good idea for the following reasons.



Reply | Threaded
Open this post in threaded view
|

Re: [Vote] SPIP: Continuous Processing Mode for Structured Streaming

Saisai Shao
In reply to this post by Shixiong(Ryan) Zhu
+1, looking forward to more design details of this feature.

Thanks
Jerry

On Wed, Nov 8, 2017 at 6:40 AM, Shixiong(Ryan) Zhu <[hidden email]> wrote:
+1

On Tue, Nov 7, 2017 at 1:34 PM, Joseph Bradley <[hidden email]> wrote:
+1

On Mon, Nov 6, 2017 at 5:11 PM, Michael Armbrust <[hidden email]> wrote:
+1

On Sat, Nov 4, 2017 at 11:02 AM, Xiao Li <[hidden email]> wrote:
+1

2017-11-04 11:00 GMT-07:00 Burak Yavuz <[hidden email]>:
+1

On Fri, Nov 3, 2017 at 10:02 PM, vaquar khan <[hidden email]> wrote:
+1

On Fri, Nov 3, 2017 at 8:14 PM, Weichen Xu <[hidden email]> wrote:
+1.

On Sat, Nov 4, 2017 at 8:04 AM, Matei Zaharia <[hidden email]> wrote:
+1 from me too.

Matei

> On Nov 3, 2017, at 4:59 PM, Wenchen Fan <[hidden email]> wrote:
>
> +1.
>
> I think this architecture makes a lot of sense to let executors talk to source/sink directly, and bring very low latency.
>
> On Thu, Nov 2, 2017 at 9:01 AM, Sean Owen <[hidden email]> wrote:
> +0 simply because I don't feel I know enough to have an opinion. I have no reason to doubt the change though, from a skim through the doc.
>
>
> On Wed, Nov 1, 2017 at 3:37 PM Reynold Xin <[hidden email]> wrote:
> Earlier I sent out a discussion thread for CP in Structured Streaming:
>
> https://issues.apache.org/jira/browse/SPARK-20928
>
> It is meant to be a very small, surgical change to Structured Streaming to enable ultra-low latency. This is great timing because we are also designing and implementing data source API v2. If designed properly, we can have the same data source API working for both streaming and batch.
>
>
> Following the SPIP process, I'm putting this SPIP up for a vote.
>
> +1: Let's go ahead and design / implement the SPIP.
> +0: Don't really care.
> -1: I do not think this is a good idea for the following reasons.
>
>
>


---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]





--
Regards,
Vaquar Khan
<a href="tel:(224)%20436-0783" value="+12244360783" target="_blank">+1 -224-436-0783
Greater Chicago






--

Joseph Bradley

Software Engineer - Machine Learning

Databricks, Inc.

http://databricks.com