flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Lawson <jlaw...@opensourceconnections.com>
Subject Re: Flume 1.7.0 Release Schedule
Date Sat, 11 Jun 2016 03:25:10 GMT
So 1.7 would support Kafka 0.9. The issues are pretty minimal around the
sink in master. Have you tried it? As for other means flume feels the
purest in not manipulating the data. If you are shipping logs, Logstash
(probably a bad word here :) is an option as it supports Kafka 0.9.

You could do something more hacky like piping into the Kafka console
producer too.

I know (elastic's) beats supports Kafka 0.9 in master but don't think it is
released either. Remember Kafka 0.9 came out in Oct or Dec so it's pretty
durn new compared to 0.8 which had over a year to get momentum. Let's not
even talk about 0.10 ;)

maybe this helps?

On Jun 10, 2016 10:34 PM, "Jason J. W. Williams" <jasonjwwilliams@gmail.com>


Flume (relay)----WAN (Avro SSL)---->Flume (concentrator)--->Kafka.

The remote Flume is talking to a "concentrator" flume in the Kafka
datacenter, so that the data is transferred via SSL over the WAN, since the
Kafka sink doesn't support SSL in 1.6.0. Otherwise, we could push from the
remote Flume directly into Kafka over the WAN.


On Fri, Jun 10, 2016 at 8:29 PM, Joe Lawson <
jlawson@opensourceconnections.com> wrote:

> Do you mean that you are replicating topics over WAN using Flume as a
> shipping agent? Sorry if I misunderstood the scenario.
> On Jun 10, 2016 9:58 PM, "Jason J. W. Williams" <jasonjwwilliams@gmail.com>
> wrote:
>> Thanks Joe that helps. What do you see folks using instead of Flume?
>> We're using largely a WAN relay for Kafka for remote sites that don't
>> warrant a local cluster.
>> -J
>> On Fri, Jun 10, 2016 at 6:17 PM, Joe Lawson <
>> jlawson@opensourceconnections.com> wrote:
>>> Flume seems to have lost some stream. It's improving but it feels like
>>> the priorities have become somewhat introspective so I'd say build from
>>> master is you want the latest.
>>> Don't hold your breath for 1.7. just an opinion. No bearing on reality.
>>> -Joe
>>> On Jun 10, 2016 12:45 PM, "Jason J. W. Williams" <
>>> jasonjwwilliams@gmail.com> wrote:
>>>> Is there an estimated release date for Flume 1.7.0? Having TLS support
>>>> in the Kafka sink would allow us to simplify our topology a lot.=20
>>>> -J

View raw message