Question about Datasource V2

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about Datasource V2

colinma(马骏杰)

Hi devs,

 

I did some investigation on data source v2 feature and have some questions:

  1. View is not supported for now, eg, CREATE VIEW,  this feature is not finished or datasource v2 won’t support view?
  2. Column resolution are failed when use datasource v2, eg,  

select tbl.col from tbl   will failed with cannot resolve '`tbl.col`' given input columns

select col from tbl  is ok

  1. For “describe table”, when implement the datasource V2 table,  we can set the user properties and get it according to the interface in Table: Map<String, String> properties(). But for some properties, eg,  “Create time”, “Create by”, they are not user properties, should we also put them in properties?
  2. Is there a plan for datasource v2 and I’m interested in take some related tasks.

 

 

Best regards,

Colin Ma

Reply | Threaded
Open this post in threaded view
|

Re: Question about Datasource V2

cloud0fan
1. we plan to add view support in future releases.
2. can you open a JIRA ticket? This seems like a bug to me.
3. instead of defining a lot of fields in the table, we decide to use properties to keep all the extra information. We've defined some reserved properties like "comment", "location", which is not user properties and have special meanings. We can add "Create Time" as a reserved property as well, as this should be set by Spark, not users.
4. we plan to add view catalog, UDF catalog, bucket join, but we don't have an official roadmap yet.

On Fri, Jan 10, 2020 at 10:19 AM colinma(马骏杰) <[hidden email]> wrote:

Hi devs,

 

I did some investigation on data source v2 feature and have some questions:

  1. View is not supported for now, eg, CREATE VIEW,  this feature is not finished or datasource v2 won’t support view?
  2. Column resolution are failed when use datasource v2, eg,  

select tbl.col from tbl   will failed with cannot resolve '`tbl.col`' given input columns

select col from tbl  is ok

  1. For “describe table”, when implement the datasource V2 table,  we can set the user properties and get it according to the interface in Table: Map<String, String> properties(). But for some properties, eg,  “Create time”, “Create by”, they are not user properties, should we also put them in properties?
  2. Is there a plan for datasource v2 and I’m interested in take some related tasks.

 

 

Best regards,

Colin Ma

Reply | Threaded
Open this post in threaded view
|

Re: Question about Datasource V2(Internet mail)

colinma(马骏杰)

Thanks for clarification, Wenchen.

I’ll create JIRA for item 2 and do some investigation on UDF catalog.

 

From: Wenchen Fan <[hidden email]>
Date: Monday, January 13, 2020 at 4:17 PM
To: "colinma(
马骏杰)" <[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Subject: Re: Question about Datasource V2(Internet mail)

 

1. we plan to add view support in future releases.

2. can you open a JIRA ticket? This seems like a bug to me.

3. instead of defining a lot of fields in the table, we decide to use properties to keep all the extra information. We've defined some reserved properties like "comment", "location", which is not user properties and have special meanings. We can add "Create Time" as a reserved property as well, as this should be set by Spark, not users.

4. we plan to add view catalog, UDF catalog, bucket join, but we don't have an official roadmap yet.

 

On Fri, Jan 10, 2020 at 10:19 AM colinma(马骏杰) <[hidden email]> wrote:

Hi devs,

 

I did some investigation on data source v2 feature and have some questions:

  1. View is not supported for now, eg, CREATE VIEW,  this feature is not finished or datasource v2 won’t support view?
  2. Column resolution are failed when use datasource v2, eg,  

select tbl.col from tbl   will failed with cannot resolve '`tbl.col`' given input columns

select col from tbl  is ok

  1. For “describe table”, when implement the datasource V2 table,  we can set the user properties and get it according to the interface in Table: Map<String, String> properties(). But for some properties, eg,  “Create time”, “Create by”, they are not user properties, should we also put them in properties?
  2. Is there a plan for datasource v2 and I’m interested in take some related tasks.

 

 

Best regards,

Colin Ma