the uniqueSource in StreamExecution, where is it be changed please?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

the uniqueSource in StreamExecution, where is it be changed please?

萝卜丝炒饭
Hi all,

These days I am learning the code about the StreamExecution.
In the method constructNextBatch(about line 365), I found the value of latestOffsets changed but I can not find where the s.getOffset of uniqueSource is  changed.
here is the code link:

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala

Would you like help understand it please?

Thanks.
Robin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: the uniqueSource in StreamExecution, where is it be changed please?

Liang-Chi Hsieh

Not sure if you are looking for how the returned value of `getOffset` changes.

I think it depends on how the actual `Source` classes implement it. For example, in `FileStreamSource`, you can see `getOffset` is updated by finding new files in the source.

Different source has different way to get its offset.


萝卜丝炒饭 wrote
Hi all,


These days I am learning the code about the StreamExecution.
In the method constructNextBatch(about line 365), I found the value of latestOffsets changed but I can not find where the s.getOffset of uniqueSource is  changed.
here is the code link:


https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala



Would you like help understand it please?


Thanks.
Robin
Liang-Chi Hsieh | @viirya
Spark Technology Center
http://www.spark.tc/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: the uniqueSource in StreamExecution, where is it be changedplease?

萝卜丝炒饭
Hi Hsieh,


I see.
Thanks for your reply.


---Original---
From: "Liang-Chi Hsieh"<[hidden email]>
Date: 2017/8/5 17:14:16
To: "dev"<[hidden email]>;
Subject: Re: the uniqueSource in StreamExecution, where is it be changedplease?


Not sure if you are looking for how the returned value of `getOffset`
changes.

I think it depends on how the actual `Source` classes implement it. For
example, in `FileStreamSource`, you can see `getOffset` is updated by
finding new files in the source.

Different source has different way to get its offset.



萝卜丝炒饭 wrote

> Hi all,
>
>
> These days I am learning the code about the StreamExecution.
> In the method constructNextBatch(about line 365), I found the value of
> latestOffsets changed but I can not find where the s.getOffset of
> uniqueSource is  changed.
> here is the code link:
>
>
> https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
>
>
>
> Would you like help understand it please?
>
>
> Thanks.
> Robin





-----
Liang-Chi Hsieh | @viirya
Spark Technology Center
http://www.spark.tc/
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/the-uniqueSource-in-StreamExecution-where-is-it-be-changed-please-tp22084p22087.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]
Loading...