Feature request: Java-specific transform method in Dataset

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Feature request: Java-specific transform method in Dataset

Ismael Carnales
Hi,
 it would be nice to have an easier way to use the Dataset transform method from Java than implementing a Function1 from Scala.

I've made a simple implentation here: 


Should I open a JIRA?

Ismael Carnales
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: Java-specific transform method in Dataset

Sean Owen-2
Don't Java 8 lambdas let you do this pretty immediately? Can you give an example here of what you want to do and how you are trying to do it?

On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <[hidden email]> wrote:
Hi,
 it would be nice to have an easier way to use the Dataset transform method from Java than implementing a Function1 from Scala.

I've made a simple implentation here: 


Should I open a JIRA?

Ismael Carnales
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: Java-specific transform method in Dataset

Ismael Carnales
No, because Function1 from Scala is not a functional interface.
You can see a simple example of what I'm trying to accomplish In the unit test here:


On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <[hidden email]> wrote:
Don't Java 8 lambdas let you do this pretty immediately? Can you give an example here of what you want to do and how you are trying to do it?

On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <[hidden email]> wrote:
Hi,
 it would be nice to have an easier way to use the Dataset transform method from Java than implementing a Function1 from Scala.

I've made a simple implentation here: 


Should I open a JIRA?

Ismael Carnales
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: Java-specific transform method in Dataset

Sean Owen-2
I see, transform() doesn't have the same overload that other methods do in order to support Java 8 lambdas as you'd expect. One option is to introduce something like MapFunction for transform and introduce an overload.

I think transform() isn't used much at all, so maybe why it wasn't Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it might. I think it could be OK to add the overload to match how map works.

On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <[hidden email]> wrote:
No, because Function1 from Scala is not a functional interface.
You can see a simple example of what I'm trying to accomplish In the unit test here:


On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <[hidden email]> wrote:
Don't Java 8 lambdas let you do this pretty immediately? Can you give an example here of what you want to do and how you are trying to do it?

On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <[hidden email]> wrote:
Hi,
 it would be nice to have an easier way to use the Dataset transform method from Java than implementing a Function1 from Scala.

I've made a simple implentation here: 


Should I open a JIRA?

Ismael Carnales
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: Java-specific transform method in Dataset

rxin
This wouldn’t be a problem with Scala 2.12 right?

On Sun, Jul 1, 2018 at 12:23 PM Sean Owen <[hidden email]> wrote:
I see, transform() doesn't have the same overload that other methods do in order to support Java 8 lambdas as you'd expect. One option is to introduce something like MapFunction for transform and introduce an overload.

I think transform() isn't used much at all, so maybe why it wasn't Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it might. I think it could be OK to add the overload to match how map works.

On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <[hidden email]> wrote:
No, because Function1 from Scala is not a functional interface.
You can see a simple example of what I'm trying to accomplish In the unit test here:


On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <[hidden email]> wrote:
Don't Java 8 lambdas let you do this pretty immediately? Can you give an example here of what you want to do and how you are trying to do it?

On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <[hidden email]> wrote:
Hi,
 it would be nice to have an easier way to use the Dataset transform method from Java than implementing a Function1 from Scala.

I've made a simple implentation here: 


Should I open a JIRA?

Ismael Carnales
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: Java-specific transform method in Dataset

Sean Owen-2
It's true, that is one of the issues to be solved by the 2.12-compatible build, because it otherwise introduces an overload ambiguity for Java 8 lambdas. But for that reason I think the current transform() method would start working with lambdas. That would only help 2.12 builds; maybe that's an OK solution?

On Sun, Jul 1, 2018, 2:36 PM Reynold Xin <[hidden email]> wrote:
This wouldn’t be a problem with Scala 2.12 right?

On Sun, Jul 1, 2018 at 12:23 PM Sean Owen <[hidden email]> wrote:
I see, transform() doesn't have the same overload that other methods do in order to support Java 8 lambdas as you'd expect. One option is to introduce something like MapFunction for transform and introduce an overload.

I think transform() isn't used much at all, so maybe why it wasn't Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it might. I think it could be OK to add the overload to match how map works.

On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <[hidden email]> wrote:
No, because Function1 from Scala is not a functional interface.
You can see a simple example of what I'm trying to accomplish In the unit test here:


On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <[hidden email]> wrote:
Don't Java 8 lambdas let you do this pretty immediately? Can you give an example here of what you want to do and how you are trying to do it?

On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <[hidden email]> wrote:
Hi,
 it would be nice to have an easier way to use the Dataset transform method from Java than implementing a Function1 from Scala.

I've made a simple implentation here: 


Should I open a JIRA?

Ismael Carnales
Reply | Threaded
Open this post in threaded view
|

Re: Feature request: Java-specific transform method in Dataset

kant kodali
I am not affiliated with Flink or Spark but I do think some of the thoughts here makes sense 

On Sun, Jul 1, 2018 at 4:12 PM, Sean Owen <[hidden email]> wrote:
It's true, that is one of the issues to be solved by the 2.12-compatible build, because it otherwise introduces an overload ambiguity for Java 8 lambdas. But for that reason I think the current transform() method would start working with lambdas. That would only help 2.12 builds; maybe that's an OK solution?


On Sun, Jul 1, 2018, 2:36 PM Reynold Xin <[hidden email]> wrote:
This wouldn’t be a problem with Scala 2.12 right?

On Sun, Jul 1, 2018 at 12:23 PM Sean Owen <[hidden email]> wrote:
I see, transform() doesn't have the same overload that other methods do in order to support Java 8 lambdas as you'd expect. One option is to introduce something like MapFunction for transform and introduce an overload.

I think transform() isn't used much at all, so maybe why it wasn't Java-fied. Before Java 8 it wouldn't have made much sense in Java. Now it might. I think it could be OK to add the overload to match how map works.

On Sun, Jul 1, 2018 at 1:33 PM Ismael Carnales <[hidden email]> wrote:
No, because Function1 from Scala is not a functional interface.
You can see a simple example of what I'm trying to accomplish In the unit test here:


On Sun, Jul 1, 2018 at 2:48 PM Sean Owen <[hidden email]> wrote:
Don't Java 8 lambdas let you do this pretty immediately? Can you give an example here of what you want to do and how you are trying to do it?

On Sun, Jul 1, 2018, 12:42 PM Ismael Carnales <[hidden email]> wrote:
Hi,
 it would be nice to have an easier way to use the Dataset transform method from Java than implementing a Function1 from Scala.

I've made a simple implentation here: 


Should I open a JIRA?

Ismael Carnales