[DISCUSS] SPIP: Native support of session window

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] SPIP: Native support of session window

Jungtaek Lim
Hi all,

I would like to initiate discussion thread to discuss "Native support of session window".
Origin issue is filed to SPARK-10816 [1] but I can file another one for representing SPIP if necessary. WIP but working PR is available as well, so we can even test it directly or see the difference if some of us feel more convenient to go through the source code instead of doc.

I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs link [2] for who feel convenient to comment in doc. 

Please let me know if we would like to see also technical design for this. I avoid to go too deep on SPIP doc so anyone could review and see the benefit of adopting this.

Looking forward to hear your feedback. 

Thanks,
Jungtaek Lim (HeartSaVioR)



Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SPIP: Native support of session window

Yuanjian Li
Hi Jungtaek:

   We also meet this problem during migration of streaming application to Structure Streaming in Baidu practice, we solved this in our folk and just steady running in product.
   As the initial plan we are doing the code clean work and preparing give a SPIP in Oct, happy to see your proposal. Hope we share some spots together.
   Here’s the PR and doc: 

Thanks,
Yuanjian Li


在 2018年9月28日,06:22,Jungtaek Lim <[hidden email]> 写道:

Hi all,

I would like to initiate discussion thread to discuss "Native support of session window".
Origin issue is filed to SPARK-10816 [1] but I can file another one for representing SPIP if necessary. WIP but working PR is available as well, so we can even test it directly or see the difference if some of us feel more convenient to go through the source code instead of doc.

I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs link [2] for who feel convenient to comment in doc. 

Please let me know if we would like to see also technical design for this. I avoid to go too deep on SPIP doc so anyone could review and see the benefit of adopting this.

Looking forward to hear your feedback. 

Thanks,
Jungtaek Lim (HeartSaVioR)




Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SPIP: Native support of session window

Jungtaek Lim
Thanks for sharing your proposal as well as implementation. Looks like your proposal is more like focused to design details: I may be better to write one more for design details and share it as well. Stay tuned!

Btw, I'm trying out your patch to see whether it passes the tests I've added, and looks like it fails on below UT: 
Could you take a look at UT and see whether I'm missing here or the UT is correct?

(Actually most of UTs I've added fail but some UTs are for update mode, and the patch doesn't provide same experience with select only session window, so I'm pointing only one UT which is testing basic session window.)

-Jungtaek Lim (HeartSaVioR)

2018년 9월 28일 (금) 오후 9:22, Yuanjian Li <[hidden email]>님이 작성:
Hi Jungtaek:

   We also meet this problem during migration of streaming application to Structure Streaming in Baidu practice, we solved this in our folk and just steady running in product.
   As the initial plan we are doing the code clean work and preparing give a SPIP in Oct, happy to see your proposal. Hope we share some spots together.
   Here’s the PR and doc: 

Thanks,
Yuanjian Li


在 2018年9月28日,06:22,Jungtaek Lim <[hidden email]> 写道:

Hi all,

I would like to initiate discussion thread to discuss "Native support of session window".
Origin issue is filed to SPARK-10816 [1] but I can file another one for representing SPIP if necessary. WIP but working PR is available as well, so we can even test it directly or see the difference if some of us feel more convenient to go through the source code instead of doc.

I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs link [2] for who feel convenient to comment in doc. 

Please let me know if we would like to see also technical design for this. I avoid to go too deep on SPIP doc so anyone could review and see the benefit of adopting this.

Looking forward to hear your feedback. 

Thanks,
Jungtaek Lim (HeartSaVioR)




Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SPIP: Native support of session window

Jungtaek Lim
Btw, just wrote up detailed design doc on existing patch:

This doc is a wall of text, since I guess we already imagine how session window works (and I showed a simple example in SPIP doc), so try to avoid drawing something which would take non-trivial efforts. New classes are linked to the actual source code so that we can read the code directly whenever curious/wonders about something. 

Please let me know anytime if something is unclear and need elaboration.

-Jungtaek Lim (HeartSaVioR)

2018년 9월 28일 (금) 오후 10:18, Jungtaek Lim <[hidden email]>님이 작성:
Thanks for sharing your proposal as well as implementation. Looks like your proposal is more like focused to design details: I may be better to write one more for design details and share it as well. Stay tuned!

Btw, I'm trying out your patch to see whether it passes the tests I've added, and looks like it fails on below UT: 
Could you take a look at UT and see whether I'm missing here or the UT is correct?

(Actually most of UTs I've added fail but some UTs are for update mode, and the patch doesn't provide same experience with select only session window, so I'm pointing only one UT which is testing basic session window.)

-Jungtaek Lim (HeartSaVioR)

2018년 9월 28일 (금) 오후 9:22, Yuanjian Li <[hidden email]>님이 작성:
Hi Jungtaek:

   We also meet this problem during migration of streaming application to Structure Streaming in Baidu practice, we solved this in our folk and just steady running in product.
   As the initial plan we are doing the code clean work and preparing give a SPIP in Oct, happy to see your proposal. Hope we share some spots together.
   Here’s the PR and doc: 

Thanks,
Yuanjian Li


在 2018年9月28日,06:22,Jungtaek Lim <[hidden email]> 写道:

Hi all,

I would like to initiate discussion thread to discuss "Native support of session window".
Origin issue is filed to SPARK-10816 [1] but I can file another one for representing SPIP if necessary. WIP but working PR is available as well, so we can even test it directly or see the difference if some of us feel more convenient to go through the source code instead of doc.

I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs link [2] for who feel convenient to comment in doc. 

Please let me know if we would like to see also technical design for this. I avoid to go too deep on SPIP doc so anyone could review and see the benefit of adopting this.

Looking forward to hear your feedback. 

Thanks,
Jungtaek Lim (HeartSaVioR)




Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] SPIP: Native support of session window

Yuanjian Li
Cool, thanks! 
Sorry for the late reply, we'll check out the UT and your design doc ASAP when we back from National Day holiday.

Thanks,
Yuanjian Li

Jungtaek Lim <[hidden email]> 于2018年9月29日周六 上午5:21写道:
Btw, just wrote up detailed design doc on existing patch:

This doc is a wall of text, since I guess we already imagine how session window works (and I showed a simple example in SPIP doc), so try to avoid drawing something which would take non-trivial efforts. New classes are linked to the actual source code so that we can read the code directly whenever curious/wonders about something. 

Please let me know anytime if something is unclear and need elaboration.

-Jungtaek Lim (HeartSaVioR)

2018년 9월 28일 (금) 오후 10:18, Jungtaek Lim <[hidden email]>님이 작성:
Thanks for sharing your proposal as well as implementation. Looks like your proposal is more like focused to design details: I may be better to write one more for design details and share it as well. Stay tuned!

Btw, I'm trying out your patch to see whether it passes the tests I've added, and looks like it fails on below UT: 
Could you take a look at UT and see whether I'm missing here or the UT is correct?

(Actually most of UTs I've added fail but some UTs are for update mode, and the patch doesn't provide same experience with select only session window, so I'm pointing only one UT which is testing basic session window.)

-Jungtaek Lim (HeartSaVioR)

2018년 9월 28일 (금) 오후 9:22, Yuanjian Li <[hidden email]>님이 작성:
Hi Jungtaek:

   We also meet this problem during migration of streaming application to Structure Streaming in Baidu practice, we solved this in our folk and just steady running in product.
   As the initial plan we are doing the code clean work and preparing give a SPIP in Oct, happy to see your proposal. Hope we share some spots together.
   Here’s the PR and doc: 

Thanks,
Yuanjian Li


在 2018年9月28日,06:22,Jungtaek Lim <[hidden email]> 写道:

Hi all,

I would like to initiate discussion thread to discuss "Native support of session window".
Origin issue is filed to SPARK-10816 [1] but I can file another one for representing SPIP if necessary. WIP but working PR is available as well, so we can even test it directly or see the difference if some of us feel more convenient to go through the source code instead of doc.

I've attached PDF version of SPIP in SPARK-10816, but adding Google Docs link [2] for who feel convenient to comment in doc. 

Please let me know if we would like to see also technical design for this. I avoid to go too deep on SPIP doc so anyone could review and see the benefit of adopting this.

Looking forward to hear your feedback. 

Thanks,
Jungtaek Lim (HeartSaVioR)