SPIP: Support Kafka delegation token in Structured Streaming

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

SPIP: Support Kafka delegation token in Structured Streaming

Gabor Somogyi
Hi all,

I am writing this e-mail in order to discuss the delegation token support for kafka feature which is reported in SPARK-25501. I've prepared a SPIP for it. PR is on the way...

Looking forward to hear your feedback.

BR,
G

Reply | Threaded
Open this post in threaded view
|

Re: SPIP: Support Kafka delegation token in Structured Streaming

Jungtaek Lim
Hi Gabor,

Thanks for proposing the feature. I'm definitely interested to see this feature, but honestly I'm not familiar with how Spark deals with delegation token for HDFS and HBase. I'll try to review the doc in general, and try to learn it, and review again based on understanding.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 9월 27일 (목) 오후 8:58, Gabor Somogyi <[hidden email]>님이 작성:
Hi all,

I am writing this e-mail in order to discuss the delegation token support for kafka feature which is reported in SPARK-25501. I've prepared a SPIP for it. PR is on the way...

Looking forward to hear your feedback.

BR,
G

Reply | Threaded
Open this post in threaded view
|

Re: SPIP: Support Kafka delegation token in Structured Streaming

Saisai Shao
In reply to this post by Gabor Somogyi
I like this proposal. Since Kafka already provides delegation token mechanism, we can also leverage Spark's delegation token framework to add Kafka as a built-in support. 

BTW I think there's no much difference in support structured streaming and DStream, maybe we can set both as goal.

Thanks
Saisai

Gabor Somogyi <[hidden email]> 于2018年9月27日周四 下午7:58写道:
Hi all,

I am writing this e-mail in order to discuss the delegation token support for kafka feature which is reported in SPARK-25501. I've prepared a SPIP for it. PR is on the way...

Looking forward to hear your feedback.

BR,
G

Reply | Threaded
Open this post in threaded view
|

Re: SPIP: Support Kafka delegation token in Structured Streaming

Gabor Somogyi
In reply to this post by Jungtaek Lim
Hi Jungtaek,

Thanks for your comments, just reacted on them.

BR,
G


On Sat, Sep 29, 2018 at 2:50 PM Jungtaek Lim <[hidden email]> wrote:
Hi Gabor,

Thanks for proposing the feature. I'm definitely interested to see this feature, but honestly I'm not familiar with how Spark deals with delegation token for HDFS and HBase. I'll try to review the doc in general, and try to learn it, and review again based on understanding.

Thanks,
Jungtaek Lim (HeartSaVioR)

2018년 9월 27일 (목) 오후 8:58, Gabor Somogyi <[hidden email]>님이 작성:
Hi all,

I am writing this e-mail in order to discuss the delegation token support for kafka feature which is reported in SPARK-25501. I've prepared a SPIP for it. PR is on the way...

Looking forward to hear your feedback.

BR,
G

Reply | Threaded
Open this post in threaded view
|

Re: SPIP: Support Kafka delegation token in Structured Streaming

Gabor Somogyi
In reply to this post by Saisai Shao
Hi Saisai,

The reasons why I've originally set the goal only structured streaming is the following:
 * Haven't seen big interest in the DStream area for new features
 * Separate the concerns even if there is a need

All in all happy to port the feature to DStream if you think it worth and you can help me, because no major differences are there (at least considering this area).

BR,
G


On Sun, Sep 30, 2018 at 4:06 AM Saisai Shao <[hidden email]> wrote:
I like this proposal. Since Kafka already provides delegation token mechanism, we can also leverage Spark's delegation token framework to add Kafka as a built-in support. 

BTW I think there's no much difference in support structured streaming and DStream, maybe we can set both as goal.

Thanks
Saisai

Gabor Somogyi <[hidden email]> 于2018年9月27日周四 下午7:58写道:
Hi all,

I am writing this e-mail in order to discuss the delegation token support for kafka feature which is reported in SPARK-25501. I've prepared a SPIP for it. PR is on the way...

Looking forward to hear your feedback.

BR,
G