[VOTE] SPIP: Add FunctionCatalog

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

[VOTE] SPIP: Add FunctionCatalog

Ryan Blue-2

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Holden Karau
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

John Zhuge
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Chao Sun
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Dongjoon Hyun-2
+1 (binding)

Thank you, Ryan.

Bests,
Dongjoon.


On Mon, Mar 8, 2021 at 5:20 PM Chao Sun <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

RussS
+1 (for what it's worth)

Thanks for making such a robust proposal, i'm excited to see the new work coming from this

On Mar 8, 2021, at 11:44 PM, Dongjoon Hyun <[hidden email]> wrote:

+1 (binding)

Thank you, Ryan.

Bests,
Dongjoon.


On Mon, Mar 8, 2021 at 5:20 PM Chao Sun <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

cloud0fan
+1 (binding)

On Tue, Mar 9, 2021 at 1:47 PM Russell Spitzer <[hidden email]> wrote:
+1 (for what it's worth)

Thanks for making such a robust proposal, i'm excited to see the new work coming from this

On Mar 8, 2021, at 11:44 PM, Dongjoon Hyun <[hidden email]> wrote:

+1 (binding)

Thank you, Ryan.

Bests,
Dongjoon.


On Mon, Mar 8, 2021 at 5:20 PM Chao Sun <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Dr. Kent Yao
+1, looks great!

Kent Yao 
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark.

spark-authorizerA Spark SQL extension which provides SQL Standard Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
spark-func-extrasA library that brings excellent and useful functions from various modern database management systems to Apache Spark.




On 03/9/2021 17:10[hidden email] wrote:
+1 (binding)

On Tue, Mar 9, 2021 at 1:47 PM Russell Spitzer <[hidden email]> wrote:
+1 (for what it's worth)

Thanks for making such a robust proposal, i'm excited to see the new work coming from this

On Mar 8, 2021, at 11:44 PM, Dongjoon Hyun <[hidden email]> wrote:

+1 (binding)

Thank you, Ryan.

Bests,
Dongjoon.


On Mon, Mar 8, 2021 at 5:20 PM Chao Sun <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

huaxin gao
+1 (non-binding)

On Tue, Mar 9, 2021 at 1:12 AM Kent Yao <[hidden email]> wrote:
+1, looks great!

Kent Yao 
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark.

spark-authorizerA Spark SQL extension which provides SQL Standard Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
spark-func-extrasA library that brings excellent and useful functions from various modern database management systems to Apache Spark.




On 03/9/2021 17:10[hidden email] wrote:
+1 (binding)

On Tue, Mar 9, 2021 at 1:47 PM Russell Spitzer <[hidden email]> wrote:
+1 (for what it's worth)

Thanks for making such a robust proposal, i'm excited to see the new work coming from this

On Mar 8, 2021, at 11:44 PM, Dongjoon Hyun <[hidden email]> wrote:

+1 (binding)

Thank you, Ryan.

Bests,
Dongjoon.


On Mon, Mar 8, 2021 at 5:20 PM Chao Sun <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Erik Krogen
+1 from me (non-binding)

On Tue, Mar 9, 2021 at 9:27 AM huaxin gao <[hidden email]> wrote:
+1 (non-binding)

On Tue, Mar 9, 2021 at 1:12 AM Kent Yao <[hidden email]> wrote:
+1, looks great!

Kent Yao 
@ Data Science Center, Hangzhou Research Institute, NetEase Corp.
a spark enthusiast
kyuubiis a unified multi-tenant JDBC interface for large-scale data processing and analytics, built on top of Apache Spark.

spark-authorizerA Spark SQL extension which provides SQL Standard Authorization for Apache Spark.
spark-postgres A library for reading data from and transferring data to Postgres / Greenplum with Spark SQL and DataFrames, 10~100x faster.
spark-func-extrasA library that brings excellent and useful functions from various modern database management systems to Apache Spark.




On 03/9/2021 17:10[hidden email] wrote:
+1 (binding)

On Tue, Mar 9, 2021 at 1:47 PM Russell Spitzer <[hidden email]> wrote:
+1 (for what it's worth)

Thanks for making such a robust proposal, i'm excited to see the new work coming from this

On Mar 8, 2021, at 11:44 PM, Dongjoon Hyun <[hidden email]> wrote:

+1 (binding)

Thank you, Ryan.

Bests,
Dongjoon.


On Mon, Mar 8, 2021 at 5:20 PM Chao Sun <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 5:13 PM John Zhuge <[hidden email]> wrote:
+1 (non-binding)

On Mon, Mar 8, 2021 at 4:32 PM Holden Karau <[hidden email]> wrote:
+1 (binding)

On Mon, Mar 8, 2021 at 3:56 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue
--
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 


--
John Zhuge

--------------------------------------------------------------------- To unsubscribe e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Liang-Chi Hsieh

+1 (non-binding).

Thanks for the work!


Erik Krogen wrote
> +1 from me (non-binding)
>
> On Tue, Mar 9, 2021 at 9:27 AM huaxin gao &lt;

> huaxin.gao11@

> &gt; wrote:
>
>> +1 (non-binding)





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Jungtaek Lim-2
+1 (non-binding) Excellent description on SPIP doc! Thanks for the amazing effort!

On Wed, Mar 10, 2021 at 3:19 AM Liang-Chi Hsieh <[hidden email]> wrote:

+1 (non-binding).

Thanks for the work!


Erik Krogen wrote
> +1 from me (non-binding)
>
> On Tue, Mar 9, 2021 at 9:27 AM huaxin gao &lt;

> huaxin.gao11@

> &gt; wrote:
>
>> +1 (non-binding)





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Hyukjin Kwon
+1

2021년 3월 12일 (금) 오후 2:54, Jungtaek Lim <[hidden email]>님이 작성:
+1 (non-binding) Excellent description on SPIP doc! Thanks for the amazing effort!

On Wed, Mar 10, 2021 at 3:19 AM Liang-Chi Hsieh <[hidden email]> wrote:

+1 (non-binding).

Thanks for the work!


Erik Krogen wrote
> +1 from me (non-binding)
>
> On Tue, Mar 9, 2021 at 9:27 AM huaxin gao &lt;

> huaxin.gao11@

> &gt; wrote:
>
>> +1 (non-binding)





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Gengliang Wang-2
+1 (non-binding)

On Fri, Mar 12, 2021 at 3:00 PM Hyukjin Kwon <[hidden email]> wrote:
+1

2021년 3월 12일 (금) 오후 2:54, Jungtaek Lim <[hidden email]>님이 작성:
+1 (non-binding) Excellent description on SPIP doc! Thanks for the amazing effort!

On Wed, Mar 10, 2021 at 3:19 AM Liang-Chi Hsieh <[hidden email]> wrote:

+1 (non-binding).

Thanks for the work!


Erik Krogen wrote
> +1 from me (non-binding)
>
> On Tue, Mar 9, 2021 at 9:27 AM huaxin gao &lt;

> huaxin.gao11@

> &gt; wrote:
>
>> +1 (non-binding)





--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

kordex
+1 (for what it's worth). It will definitely help our efforts.

On Fri, Mar 12, 2021 at 12:14 PM Gengliang Wang <[hidden email]> wrote:

>
> +1 (non-binding)
>
> On Fri, Mar 12, 2021 at 3:00 PM Hyukjin Kwon <[hidden email]> wrote:
>>
>> +1
>>
>> 2021년 3월 12일 (금) 오후 2:54, Jungtaek Lim <[hidden email]>님이 작성:
>>>
>>> +1 (non-binding) Excellent description on SPIP doc! Thanks for the amazing effort!
>>>
>>> On Wed, Mar 10, 2021 at 3:19 AM Liang-Chi Hsieh <[hidden email]> wrote:
>>>>
>>>>
>>>> +1 (non-binding).
>>>>
>>>> Thanks for the work!
>>>>
>>>>
>>>> Erik Krogen wrote
>>>> > +1 from me (non-binding)
>>>> >
>>>> > On Tue, Mar 9, 2021 at 9:27 AM huaxin gao &lt;
>>>>
>>>> > huaxin.gao11@
>>>>
>>>> > &gt; wrote:
>>>> >
>>>> >> +1 (non-binding)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: [hidden email]
>>>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Takeshi Yamamuro
+1, too.

On Fri, Mar 12, 2021 at 8:51 PM kordex <[hidden email]> wrote:
+1 (for what it's worth). It will definitely help our efforts.

On Fri, Mar 12, 2021 at 12:14 PM Gengliang Wang <[hidden email]> wrote:
>
> +1 (non-binding)
>
> On Fri, Mar 12, 2021 at 3:00 PM Hyukjin Kwon <[hidden email]> wrote:
>>
>> +1
>>
>> 2021년 3월 12일 (금) 오후 2:54, Jungtaek Lim <[hidden email]>님이 작성:
>>>
>>> +1 (non-binding) Excellent description on SPIP doc! Thanks for the amazing effort!
>>>
>>> On Wed, Mar 10, 2021 at 3:19 AM Liang-Chi Hsieh <[hidden email]> wrote:
>>>>
>>>>
>>>> +1 (non-binding).
>>>>
>>>> Thanks for the work!
>>>>
>>>>
>>>> Erik Krogen wrote
>>>> > +1 from me (non-binding)
>>>> >
>>>> > On Tue, Mar 9, 2021 at 9:27 AM huaxin gao &lt;
>>>>
>>>> > huaxin.gao11@
>>>>
>>>> > &gt; wrote:
>>>> >
>>>> >> +1 (non-binding)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: [hidden email]
>>>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro
Reply | Threaded
Open this post in threaded view
|

Re: [VOTE] SPIP: Add FunctionCatalog

Ryan Blue
And a late +1 from me.

On Fri, Mar 12, 2021 at 5:46 AM Takeshi Yamamuro <[hidden email]> wrote:
+1, too.

On Fri, Mar 12, 2021 at 8:51 PM kordex <[hidden email]> wrote:
+1 (for what it's worth). It will definitely help our efforts.

On Fri, Mar 12, 2021 at 12:14 PM Gengliang Wang <[hidden email]> wrote:
>
> +1 (non-binding)
>
> On Fri, Mar 12, 2021 at 3:00 PM Hyukjin Kwon <[hidden email]> wrote:
>>
>> +1
>>
>> 2021년 3월 12일 (금) 오후 2:54, Jungtaek Lim <[hidden email]>님이 작성:
>>>
>>> +1 (non-binding) Excellent description on SPIP doc! Thanks for the amazing effort!
>>>
>>> On Wed, Mar 10, 2021 at 3:19 AM Liang-Chi Hsieh <[hidden email]> wrote:
>>>>
>>>>
>>>> +1 (non-binding).
>>>>
>>>> Thanks for the work!
>>>>
>>>>
>>>> Erik Krogen wrote
>>>> > +1 from me (non-binding)
>>>> >
>>>> > On Tue, Mar 9, 2021 at 9:27 AM huaxin gao &lt;
>>>>
>>>> > huaxin.gao11@
>>>>
>>>> > &gt; wrote:
>>>> >
>>>> >> +1 (non-binding)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: [hidden email]
>>>>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
---
Takeshi Yamamuro


--
Ryan Blue
Software Engineer
Netflix
Reply | Threaded
Open this post in threaded view
|

[RESULT] [VOTE] SPIP: Add FunctionCatalog

Ryan Blue-2
In reply to this post by Ryan Blue-2
This SPIP is adopted with the following +1 votes and no -1 or +0 votes:

Holden Karau*
John Zhuge
Chao Sun
Dongjoon Hyun*
Russell Spitzer
DB Tsai*
Wenchen Fan*
Kent Yao
Huaxin Gao
Liang-Chi Hsieh
Jungtaek Lim
Hyukjin Kwon*
Gengliang Wang
kordex
Takeshi Yamamuro
Ryan Blue

* = binding

On Mon, Mar 8, 2021 at 3:55 PM Ryan Blue <[hidden email]> wrote:

Hi everyone, I’d like to start a vote for the FunctionCatalog design proposal (SPIP).

The proposal is to add a FunctionCatalog interface that can be used to load and list functions for Spark to call. There are interfaces for scalar and aggregate functions.

In the discussion we’ve come to consensus and I’ve updated the design doc to match how functions will be called:

In addition to produceResult(InternalRow), which is optional, functions can define produceResult methods with arguments that are Spark’s internal data types, like UTF8String. Spark will prefer these methods when calling the UDF using codgen.

I’ve also updated the AggregateFunction interface and merged it with the partial aggregate interface because Spark doesn’t support non-partial aggregates.

The full SPIP doc is here: https://docs.google.com/document/d/1PLBieHIlxZjmoUB0ERF-VozCRJ0xw2j3qKvUNWpWA2U/edit#heading=h.82w8qxfl2uwl

Please vote on the SPIP in the next 72 hours. Once it is approved, I’ll do a final update of the PR and we can merge the API.

[ ] +1: Accept the proposal as an official SPIP
[ ] +0
[ ] -1: I don’t think this is a good idea because …

--
Ryan Blue


--
Ryan Blue