Timestamp interoperability design doc available for review

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Timestamp interoperability design doc available for review

Zoltan Ivanfi
Dear Spark Community,

Based on earlier feedback from the Spark community, we would like to suggest a short-term fix for the timestamp interoperability problem[1] between different SQL-on-Hadoop engines. I created a design document[2] and would like to ask you to review it and let me know of any concerns and/or suggestions you may have.

[1] https://issues.apache.org/jira/browse/SPARK-12297
[2] https://docs.google.com/document/d/1XmyVjr3eOJiNFjVeSnmjIU60Hq-XiZB03pgi3r1razM/edit

Thanks,

Zoltan
Reply | Threaded
Open this post in threaded view
|

Re: Timestamp interoperability design doc available for review

Imran Rashid-3
I've posted a design doc on SPARK-12297, which builds on what Zoltan posted here earlier.  It addresses the parquet issues and also considers current inconsistencies in timestamp behavior for spark across data formats and versions.  I believe this incorporates all of the prior concerns and feedback.  I should post a PR for it soon.


thanks,
Imran

On Wed, Aug 16, 2017 at 12:33 PM, Zoltan Ivanfi <[hidden email]> wrote:
Dear Spark Community,

Based on earlier feedback from the Spark community, we would like to suggest a short-term fix for the timestamp interoperability problem[1] between different SQL-on-Hadoop engines. I created a design document[2] and would like to ask you to review it and let me know of any concerns and/or suggestions you may have.

[1] https://issues.apache.org/jira/browse/SPARK-12297
[2] https://docs.google.com/document/d/1XmyVjr3eOJiNFjVeSnmjIU60Hq-XiZB03pgi3r1razM/edit

Thanks,

Zoltan