Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

angers.zhu
Hi all, 

I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. 

Hope for your suggestions and ideas.


Best Regards

--------- Forwarded Message ---------

From: [hidden email]
Date: 12/18/2019 22:29
Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Add spark-dev group access privilege to google.

On 12/18/2019 22:02[hidden email] wrote:
I couldn't access the doc, please give permission to the spark-dev group

On Wed, 18 Dec 2019 at 18:05, angers.zhu <[hidden email]> wrote:

With the development of Spark and Hive,in current sql/hive-thriftserver module, 

we need to do a lot of work to solve code conflicts for different built-in hive versions.

It's an annoying and unending work in current ways. And these issues have limited 

our ability and convenience to develop new features for Spark’s thrift server. 

    We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 

TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:

  1. Build new module spark-service as spark’s thrift server 

  2. Don't need as much reflection and inherited code as `hive-thriftser` modules

  3. Support all functions current `sql/hive-thriftserver` support

  4. Use all code maintained by spark itself, won’t depend on Hive

  5. Support origin functions use spark’s own way, won't limited by Hive's code

  6. Support running without hive metastore or with hive metastore

  7. Support user impersonation by Multi-tenant splited hive authentication and DFS authentication

  8. Support session hook for with spark’s own code

  9. Add a new jdbc driver spark-jdbc, with spark’s own connection url  “jdbc:spark:<host>:<port>/<db>”

  10. Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform


 

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Yuming Wang
I'm  +1 for this SPIP for these two reasons:

1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636.
2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger and bigger. We can't ensure that there will be no compatibility issues in the future. If thriftserver does not depend on Hive, it will be much easier to upgrade the built-in Hive in the future.

On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <[hidden email]> wrote:
Hi all, 

I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. 

Hope for your suggestions and ideas.


Best Regards

--------- Forwarded Message ---------

From: [hidden email]
Date: 12/18/2019 22:29
Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Add spark-dev group access privilege to google.

On 12/18/2019 22:02[hidden email] wrote:
I couldn't access the doc, please give permission to the spark-dev group

On Wed, 18 Dec 2019 at 18:05, angers.zhu <[hidden email]> wrote:

With the development of Spark and Hive,in current sql/hive-thriftserver module, 

we need to do a lot of work to solve code conflicts for different built-in hive versions.

It's an annoying and unending work in current ways. And these issues have limited 

our ability and convenience to develop new features for Spark’s thrift server. 

    We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 

TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:

  1. Build new module spark-service as spark’s thrift server 

  2. Don't need as much reflection and inherited code as `hive-thriftser` modules

  3. Support all functions current `sql/hive-thriftserver` support

  4. Use all code maintained by spark itself, won’t depend on Hive

  5. Support origin functions use spark’s own way, won't limited by Hive's code

  6. Support running without hive metastore or with hive metastore

  7. Support user impersonation by Multi-tenant splited hive authentication and DFS authentication

  8. Support session hook for with spark’s own code

  9. Add a new jdbc driver spark-jdbc, with spark’s own connection url  “jdbc:spark:<host>:<port>/<db>”

  10. Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform


 

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Reply | Threaded
Open this post in threaded view
|

Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

cloud0fan
+1 for the new thrift server to get rid of the Hive dependencies!

On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <[hidden email]> wrote:
I'm  +1 for this SPIP for these two reasons:

1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636.
2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger and bigger. We can't ensure that there will be no compatibility issues in the future. If thriftserver does not depend on Hive, it will be much easier to upgrade the built-in Hive in the future.

On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <[hidden email]> wrote:
Hi all, 

I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. 

Hope for your suggestions and ideas.


Best Regards

--------- Forwarded Message ---------

From: [hidden email]
Date: 12/18/2019 22:29
Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Add spark-dev group access privilege to google.

On 12/18/2019 22:02[hidden email] wrote:
I couldn't access the doc, please give permission to the spark-dev group

On Wed, 18 Dec 2019 at 18:05, angers.zhu <[hidden email]> wrote:

With the development of Spark and Hive,in current sql/hive-thriftserver module, 

we need to do a lot of work to solve code conflicts for different built-in hive versions.

It's an annoying and unending work in current ways. And these issues have limited 

our ability and convenience to develop new features for Spark’s thrift server. 

    We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 

TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:

  1. Build new module spark-service as spark’s thrift server 

  2. Don't need as much reflection and inherited code as `hive-thriftser` modules

  3. Support all functions current `sql/hive-thriftserver` support

  4. Use all code maintained by spark itself, won’t depend on Hive

  5. Support origin functions use spark’s own way, won't limited by Hive's code

  6. Support running without hive metastore or with hive metastore

  7. Support user impersonation by Multi-tenant splited hive authentication and DFS authentication

  8. Support session hook for with spark’s own code

  9. Add a new jdbc driver spark-jdbc, with spark’s own connection url  “jdbc:spark:<host>:<port>/<db>”

  10. Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform


 

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Reply | Threaded
Open this post in threaded view
|

Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Gengliang
+1

On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan <[hidden email]> wrote:
+1 for the new thrift server to get rid of the Hive dependencies!

On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <[hidden email]> wrote:
I'm  +1 for this SPIP for these two reasons:

1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636.
2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger and bigger. We can't ensure that there will be no compatibility issues in the future. If thriftserver does not depend on Hive, it will be much easier to upgrade the built-in Hive in the future.

On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <[hidden email]> wrote:
Hi all, 

I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. 

Hope for your suggestions and ideas.


Best Regards

--------- Forwarded Message ---------

From: [hidden email]
Date: 12/18/2019 22:29
Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Add spark-dev group access privilege to google.

On 12/18/2019 22:02[hidden email] wrote:
I couldn't access the doc, please give permission to the spark-dev group

On Wed, 18 Dec 2019 at 18:05, angers.zhu <[hidden email]> wrote:

With the development of Spark and Hive,in current sql/hive-thriftserver module, 

we need to do a lot of work to solve code conflicts for different built-in hive versions.

It's an annoying and unending work in current ways. And these issues have limited 

our ability and convenience to develop new features for Spark’s thrift server. 

    We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 

TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:

  1. Build new module spark-service as spark’s thrift server 

  2. Don't need as much reflection and inherited code as `hive-thriftser` modules

  3. Support all functions current `sql/hive-thriftserver` support

  4. Use all code maintained by spark itself, won’t depend on Hive

  5. Support origin functions use spark’s own way, won't limited by Hive's code

  6. Support running without hive metastore or with hive metastore

  7. Support user impersonation by Multi-tenant splited hive authentication and DFS authentication

  8. Support session hook for with spark’s own code

  9. Add a new jdbc driver spark-jdbc, with spark’s own connection url  “jdbc:spark:<host>:<port>/<db>”

  10. Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform


 

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Reply | Threaded
Open this post in threaded view
|

Re: Fw:Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

sandeep_katta
+1

On Mon, 30 Dec 2019 at 10:24, Gengliang <[hidden email]> wrote:
+1

On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan <[hidden email]> wrote:
+1 for the new thrift server to get rid of the Hive dependencies!

On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <[hidden email]> wrote:
I'm  +1 for this SPIP for these two reasons:

1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636.
2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger and bigger. We can't ensure that there will be no compatibility issues in the future. If thriftserver does not depend on Hive, it will be much easier to upgrade the built-in Hive in the future.

On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <[hidden email]> wrote:
Hi all, 

I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. 

Hope for your suggestions and ideas.


Best Regards

--------- Forwarded Message ---------

From: [hidden email]
Date: 12/18/2019 22:29
Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Add spark-dev group access privilege to google.

On 12/18/2019 22:02[hidden email] wrote:
I couldn't access the doc, please give permission to the spark-dev group

On Wed, 18 Dec 2019 at 18:05, angers.zhu <[hidden email]> wrote:

With the development of Spark and Hive,in current sql/hive-thriftserver module, 

we need to do a lot of work to solve code conflicts for different built-in hive versions.

It's an annoying and unending work in current ways. And these issues have limited 

our ability and convenience to develop new features for Spark’s thrift server. 

    We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 

TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:

  1. Build new module spark-service as spark’s thrift server 

  2. Don't need as much reflection and inherited code as `hive-thriftser` modules

  3. Support all functions current `sql/hive-thriftserver` support

  4. Use all code maintained by spark itself, won’t depend on Hive

  5. Support origin functions use spark’s own way, won't limited by Hive's code

  6. Support running without hive metastore or with hive metastore

  7. Support user impersonation by Multi-tenant splited hive authentication and DFS authentication

  8. Support session hook for with spark’s own code

  9. Add a new jdbc driver spark-jdbc, with spark’s own connection url  “jdbc:spark:<host>:<port>/<db>”

  10. Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform


 

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

Reply | Threaded
Open this post in threaded view
|

Re: Fw: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

angers.zhu
Hi all,
Blow is our working repo

Hope for your good suggestions.

On 12/30/2019 14:38[hidden email] wrote:
+1

On Mon, 30 Dec 2019 at 10:24, Gengliang <[hidden email]> wrote:
+1

On Sun, Dec 29, 2019 at 8:33 PM Wenchen Fan <[hidden email]> wrote:
+1 for the new thrift server to get rid of the Hive dependencies!

On Mon, Dec 23, 2019 at 7:55 PM Yuming Wang <[hidden email]> wrote:
I'm  +1 for this SPIP for these two reasons:

1. The current thriftserver has some issues that are not easy to solve, such as: SPARK-28636.
2. The difference between the version of ORC we are using and the built-in Hive is using is getting bigger and bigger. We can't ensure that there will be no compatibility issues in the future. If thriftserver does not depend on Hive, it will be much easier to upgrade the built-in Hive in the future.

On Sat, Dec 21, 2019 at 9:28 PM angers.zhu <[hidden email]> wrote:
Hi all, 

I have complete a Design doc about how to use and config this new thrift server, and some design detail about change and impersonation. 

Hope for your suggestions and ideas.


Best Regards

--------- Forwarded Message ---------

From: [hidden email]
Date: 12/18/2019 22:29
Subject: Re: [VOTE][SPARK-29018][SPIP]:Build spark thrift server based on protocol v11

Add spark-dev group access privilege to google.

On 12/18/2019 22:02[hidden email] wrote:
I couldn't access the doc, please give permission to the spark-dev group

On Wed, 18 Dec 2019 at 18:05, angers.zhu <[hidden email]> wrote:

With the development of Spark and Hive,in current sql/hive-thriftserver module, 

we need to do a lot of work to solve code conflicts for different built-in hive versions.

It's an annoying and unending work in current ways. And these issues have limited 

our ability and convenience to develop new features for Spark’s thrift server. 

    We suppose to implement a new thrift server and JDBC driver based on Hive’s latest v11 

TCLService.thrift thrift protocol. Finally, the new thrift server have below feature:

  1. Build new module spark-service as spark’s thrift server 

  2. Don't need as much reflection and inherited code as `hive-thriftser` modules

  3. Support all functions current `sql/hive-thriftserver` support

  4. Use all code maintained by spark itself, won’t depend on Hive

  5. Support origin functions use spark’s own way, won't limited by Hive's code

  6. Support running without hive metastore or with hive metastore

  7. Support user impersonation by Multi-tenant splited hive authentication and DFS authentication

  8. Support session hook for with spark’s own code

  9. Add a new jdbc driver spark-jdbc, with spark’s own connection url  “jdbc:spark:<host>:<port>/<db>”

  10. Support both hive-jdbc and spark-jdbc client, then we can support most clients and BI platform


 

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don't think this is a good idea because ...

I'll start with my +1

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]