Remove Flume support in 3.0.0?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Remove Flume support in 3.0.0?

Sean Owen-3
Marcelo makes an argument that Flume support should be removed in
3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598

I tend to agree. Is there an argument that it needs to be supported,
and can this move to Bahir if so?

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Flume support in 3.0.0?

Jörn Franke
I think it makes sense to remove it.
If it is not too much effort and the architecture of the flume source is not considered as too strange one may extract it as a separate project and put it on github in a dedicated non-supported repository. This would enable distributors and other companies to continue to use it with minor adaptions in case their architecture depends on it. Furthermore, if there is a growing interest then one could pick it up and create a clean connector based on the current Spark architecture to be available as a dedicated connector or again in later Spark versions.

That being said there are also „indirect“ ways to use Flume with Spark (eg via Kafka), so i believe people would not be affected so much by a removal.

(Non-Voting just my opinion)

> Am 10.10.2018 um 22:31 schrieb Sean Owen <[hidden email]>:
>
> Marcelo makes an argument that Flume support should be removed in
> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598
>
> I tend to agree. Is there an argument that it needs to be supported,
> and can this move to Bahir if so?
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Flume support in 3.0.0?

Marcelo Vanzin-2
In reply to this post by Sean Owen-3
BTW, although I did not file a bug for that, I think we should also
consider getting rid of the kafka-0.8 connector.

That would leave only kafka-0.10 as the single remaining dstream
connector in Spark, though. (If you ignore kinesis which we can't ship
in binary form or something like that?)
On Wed, Oct 10, 2018 at 1:32 PM Sean Owen <[hidden email]> wrote:

>
> Marcelo makes an argument that Flume support should be removed in
> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598
>
> I tend to agree. Is there an argument that it needs to be supported,
> and can this move to Bahir if so?
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


--
Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Flume support in 3.0.0?

Sean Owen-3
Yup was thinking the same. It is legacy too at this point. 

On Wed, Oct 10, 2018, 3:19 PM Marcelo Vanzin <[hidden email]> wrote:
BTW, although I did not file a bug for that, I think we should also
consider getting rid of the kafka-0.8 connector.

That would leave only kafka-0.10 as the single remaining dstream
connector in Spark, though. (If you ignore kinesis which we can't ship
in binary form or something like that?)
On Wed, Oct 10, 2018 at 1:32 PM Sean Owen <[hidden email]> wrote:
>
> Marcelo makes an argument that Flume support should be removed in
> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598
>
> I tend to agree. Is there an argument that it needs to be supported,
> and can this move to Bahir if so?
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>


--
Marcelo
Reply | Threaded
Open this post in threaded view
|

Re: Remove Flume support in 3.0.0?

Sean Owen-3
In reply to this post by Jörn Franke
Yep, that already exists as Bahir.
Also, would anyone object to declaring Flume support at least
deprecated in 2.4.0?
On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke <[hidden email]> wrote:

>
> I think it makes sense to remove it.
> If it is not too much effort and the architecture of the flume source is not considered as too strange one may extract it as a separate project and put it on github in a dedicated non-supported repository. This would enable distributors and other companies to continue to use it with minor adaptions in case their architecture depends on it. Furthermore, if there is a growing interest then one could pick it up and create a clean connector based on the current Spark architecture to be available as a dedicated connector or again in later Spark versions.
>
> That being said there are also „indirect“ ways to use Flume with Spark (eg via Kafka), so i believe people would not be affected so much by a removal.
>
> (Non-Voting just my opinion)
>
> > Am 10.10.2018 um 22:31 schrieb Sean Owen <[hidden email]>:
> >
> > Marcelo makes an argument that Flume support should be removed in
> > 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598
> >
> > I tend to agree. Is there an argument that it needs to be supported,
> > and can this move to Bahir if so?
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: [hidden email]
> >

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Flume support in 3.0.0?

rxin
Sounds like a good idea...

> On Oct 11, 2018, at 6:40 PM, Sean Owen <[hidden email]> wrote:
>
> Yep, that already exists as Bahir.
> Also, would anyone object to declaring Flume support at least
> deprecated in 2.4.0?
>> On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke <[hidden email]> wrote:
>>
>> I think it makes sense to remove it.
>> If it is not too much effort and the architecture of the flume source is not considered as too strange one may extract it as a separate project and put it on github in a dedicated non-supported repository. This would enable distributors and other companies to continue to use it with minor adaptions in case their architecture depends on it. Furthermore, if there is a growing interest then one could pick it up and create a clean connector based on the current Spark architecture to be available as a dedicated connector or again in later Spark versions.
>>
>> That being said there are also „indirect“ ways to use Flume with Spark (eg via Kafka), so i believe people would not be affected so much by a removal.
>>
>> (Non-Voting just my opinion)
>>
>>> Am 10.10.2018 um 22:31 schrieb Sean Owen <[hidden email]>:
>>>
>>> Marcelo makes an argument that Flume support should be removed in
>>> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598
>>>
>>> I tend to agree. Is there an argument that it needs to be supported,
>>> and can this move to Bahir if so?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Flume support in 3.0.0?

cloud0fan
Note that, it was deprecated in 2.3.0 already: https://spark.apache.org/docs/2.3.0/streaming-flume-integration.html

On Fri, Oct 12, 2018 at 12:46 AM Reynold Xin <[hidden email]> wrote:
Sounds like a good idea...

> On Oct 11, 2018, at 6:40 PM, Sean Owen <[hidden email]> wrote:
>
> Yep, that already exists as Bahir.
> Also, would anyone object to declaring Flume support at least
> deprecated in 2.4.0?
>> On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke <[hidden email]> wrote:
>>
>> I think it makes sense to remove it.
>> If it is not too much effort and the architecture of the flume source is not considered as too strange one may extract it as a separate project and put it on github in a dedicated non-supported repository. This would enable distributors and other companies to continue to use it with minor adaptions in case their architecture depends on it. Furthermore, if there is a growing interest then one could pick it up and create a clean connector based on the current Spark architecture to be available as a dedicated connector or again in later Spark versions.
>>
>> That being said there are also „indirect“ ways to use Flume with Spark (eg via Kafka), so i believe people would not be affected so much by a removal.
>>
>> (Non-Voting just my opinion)
>>
>>> Am 10.10.2018 um 22:31 schrieb Sean Owen <[hidden email]>:
>>>
>>> Marcelo makes an argument that Flume support should be removed in
>>> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598
>>>
>>> I tend to agree. Is there an argument that it needs to be supported,
>>> and can this move to Bahir if so?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Remove Flume support in 3.0.0?

Hyukjin Kwon
Yea, I thought we are already going to remove this out. +1 for removing it anyway.

2018년 10월 12일 (금) 오전 1:44, Wenchen Fan <[hidden email]>님이 작성:
Note that, it was deprecated in 2.3.0 already: https://spark.apache.org/docs/2.3.0/streaming-flume-integration.html

On Fri, Oct 12, 2018 at 12:46 AM Reynold Xin <[hidden email]> wrote:
Sounds like a good idea...

> On Oct 11, 2018, at 6:40 PM, Sean Owen <[hidden email]> wrote:
>
> Yep, that already exists as Bahir.
> Also, would anyone object to declaring Flume support at least
> deprecated in 2.4.0?
>> On Wed, Oct 10, 2018 at 2:29 PM Jörn Franke <[hidden email]> wrote:
>>
>> I think it makes sense to remove it.
>> If it is not too much effort and the architecture of the flume source is not considered as too strange one may extract it as a separate project and put it on github in a dedicated non-supported repository. This would enable distributors and other companies to continue to use it with minor adaptions in case their architecture depends on it. Furthermore, if there is a growing interest then one could pick it up and create a clean connector based on the current Spark architecture to be available as a dedicated connector or again in later Spark versions.
>>
>> That being said there are also „indirect“ ways to use Flume with Spark (eg via Kafka), so i believe people would not be affected so much by a removal.
>>
>> (Non-Voting just my opinion)
>>
>>> Am 10.10.2018 um 22:31 schrieb Sean Owen <[hidden email]>:
>>>
>>> Marcelo makes an argument that Flume support should be removed in
>>> 3.0.0 at https://issues.apache.org/jira/browse/SPARK-25598
>>>
>>> I tend to agree. Is there an argument that it needs to be supported,
>>> and can this move to Bahir if so?
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: [hidden email]
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: [hidden email]
>

---------------------------------------------------------------------
To unsubscribe e-mail: [hidden email]