Timeline for Spark 3.0

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Timeline for Spark 3.0

Long, Andrew-2

Hey Friends,

 

Is there a timeline for spark 3.0 in terms of the first RC and final release?

 

Cheers Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 3.0

Sean Owen-2
That's a good question. Although we had penciled in 'middle of the
year' I don't think we're in sight of a QA phase just yet, as I
believe some key items are still in progress. I'm thinking of the Hive
update, and DS v2 work (?).

I'm also curious to hear what broad TODOs people see for 3.0? we
probably need to start focusing them down and/or deferring some parts,
in order to get a 3.0 release out anytime soon. Is that feasible? I
have a guess that it won't end up being released this quarter, but,
just a guess.

On Fri, Jun 28, 2019 at 5:14 PM Long, Andrew
<[hidden email]> wrote:

>
> Hey Friends,
>
>
>
> Is there a timeline for spark 3.0 in terms of the first RC and final release?
>
>
>
> Cheers Andrew

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Timeline for Spark 3.0

Ryan Blue
I think that with a few reviews, we could have a minimally working DSv2 SQL API ready in a couple weeks. We have PRs ready for the last of the major items, like ALTER TABLE, INSERT INTO, REPLACE TABLE, and a v2 catalog interface for the session catalog. Mainly, we're just waiting for final reviews on those and then I would be comfortable moving to QA for 3.0.

The built-in file sources unfortunately have some work to do to work with SQL, so I'm not including those in that statement. That would also not include all of the non-essential SQL statements, like SHOW TABLES or CREATE TABLE LIKE. Most of the non-essential SQL could be done fairly quickly, if anyone has time!

On Fri, Jun 28, 2019 at 4:53 PM Sean Owen <[hidden email]> wrote:
That's a good question. Although we had penciled in 'middle of the
year' I don't think we're in sight of a QA phase just yet, as I
believe some key items are still in progress. I'm thinking of the Hive
update, and DS v2 work (?).

I'm also curious to hear what broad TODOs people see for 3.0? we
probably need to start focusing them down and/or deferring some parts,
in order to get a 3.0 release out anytime soon. Is that feasible? I
have a guess that it won't end up being released this quarter, but,
just a guess.

On Fri, Jun 28, 2019 at 5:14 PM Long, Andrew
<[hidden email]> wrote:
>
> Hey Friends,
>
>
>
> Is there a timeline for spark 3.0 in terms of the first RC and final release?
>
>
>
> Cheers Andrew

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



--
Ryan Blue
Software Engineer
Netflix