Error in custom encoder

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Error in custom encoder

asm0dey
Hello everyone,

I'm trying to implement new encoder, which will be able to encode Kotlin
data classes.

They can be generics, so I'm calculating their types in compiletime.

My trouble is I have following code generated:

/* 197 */       final InternalRow deserializetoobject_value_9 = false ?
/* 198 */       null : new kotlin.Pair(deserializetoobject_argValue_1,
deserializetoobject_mutableStateArray_1[0]);

It's obvious that InternalRow can't be assigned with kotlin.Pair

And therefore errors like this:
20/03/19 16:31:18 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java',
Line 195, Column 74: Assignment conversion not possible from type
"kotlin.Pair" to type "org.apache.spark.sql.catalyst.InternalRow"


Here [1] I put gist with more code: mine and generated and also stacktace

I'm building my encoder against current spark snapshot

Thank you,
Pasha

1. https://gist.github.com/asm0dey/14d5b8aa289403251a7110e5522cff38

0x87075DBDFED351B5.asc (7K) Download Attachment
signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Error in custom encoder

asm0dey
Nevermind, solved it. It was incorrect Literal creation with StructType
instead of ObjectType



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]