spark-without-hive assembly for hive build/development purposes

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

spark-without-hive assembly for hive build/development purposes

kgyrtkirk
Hello,

Hive needs a spark assembly to execute the HoS tests.
Until now…this assembly have been downloaded from an S3 bucket - because
this is not the best solution available, it sometimes causes troubles
and inconveniences...

We had a discussion about improving this; and the best option would be
to download spark-without-hive assembly from the maven repository...but
that opens up a few questions:

1) which project should publish it: Hive or Spark?
2) what should be the group-id? this artifact is only needed for
Hive...but it contains Spark code! :) so its in some kind of grey zone...
3) how will we be able to get a spark-without-hive artifact for 2.0.0 -
since that version is already released?

for more details:
https://issues.apache.org/jira/browse/HIVE-14735

What do you guys think?

cheers,
Zoltan

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: spark-without-hive assembly for hive build/development purposes

rxin
Why do you need an assembly? Is there something preventing Hive from depending on normal jars like all other applications?

On Thu, Mar 16, 2017 at 3:42 PM, Zoltan Haindrich <[hidden email]> wrote:
Hello,

Hive needs a spark assembly to execute the HoS tests.
Until now…this assembly have been downloaded from an S3 bucket - because this is not the best solution available, it sometimes causes troubles and inconveniences...

We had a discussion about improving this; and the best option would be to download spark-without-hive assembly from the maven repository...but that opens up a few questions:

1) which project should publish it: Hive or Spark?
2) what should be the group-id? this artifact is only needed for Hive...but it contains Spark code! :) so its in some kind of grey zone...
3) how will we be able to get a spark-without-hive artifact for 2.0.0 - since that version is already released?

for more details:
https://issues.apache.org/jira/browse/HIVE-14735

What do you guys think?

cheers,
Zoltan

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: spark-without-hive assembly for hive build/development purposes

kgyrtkirk

Hive uses spark-submit to launch the spark jobs - that's why the assembly is needed.


On 03/16/2017 11:54 PM, Reynold Xin wrote:
Why do you need an assembly? Is there something preventing Hive from depending on normal jars like all other applications?

On Thu, Mar 16, 2017 at 3:42 PM, Zoltan Haindrich <[hidden email]> wrote:
Hello,

Hive needs a spark assembly to execute the HoS tests.
Until now…this assembly have been downloaded from an S3 bucket - because this is not the best solution available, it sometimes causes troubles and inconveniences...

We had a discussion about improving this; and the best option would be to download spark-without-hive assembly from the maven repository...but that opens up a few questions:

1) which project should publish it: Hive or Spark?
2) what should be the group-id? this artifact is only needed for Hive...but it contains Spark code! :) so its in some kind of grey zone...
3) how will we be able to get a spark-without-hive artifact for 2.0.0 - since that version is already released?

for more details:
https://issues.apache.org/jira/browse/HIVE-14735

What do you guys think?

cheers,
Zoltan

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: spark-without-hive assembly for hive build/development purposes

Marcelo Vanzin
In reply to this post by kgyrtkirk
The solution to this is being tracked in
https://issues.apache.org/jira/browse/HIVE-15302, although I haven't
seen activity in a while.

On Thu, Mar 16, 2017 at 3:42 PM, Zoltan Haindrich <[hidden email]> wrote:

> Hello,
>
> Hive needs a spark assembly to execute the HoS tests.
> Until now…this assembly have been downloaded from an S3 bucket - because
> this is not the best solution available, it sometimes causes troubles and
> inconveniences...
>
> We had a discussion about improving this; and the best option would be to
> download spark-without-hive assembly from the maven repository...but that
> opens up a few questions:
>
> 1) which project should publish it: Hive or Spark?
> 2) what should be the group-id? this artifact is only needed for Hive...but
> it contains Spark code! :) so its in some kind of grey zone...
> 3) how will we be able to get a spark-without-hive artifact for 2.0.0 -
> since that version is already released?
>
> for more details:
> https://issues.apache.org/jira/browse/HIVE-14735
>
> What do you guys think?
>
> cheers,
> Zoltan
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>



--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Loading...