flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Williams <jasonjwwilli...@gmail.com>
Subject Re: Flume 1.7.0 Release Schedule
Date Sat, 11 Jun 2016 05:06:58 GMT
Hey Joe,

I may try 1.7 if you think it's pretty stable in master. Flume as a relay actually solves
our problem pretty well, in that if the Kafka DC becomes unavailable for any reason, the Flume
file channel will queue up the entries reliably and replay when the link comes back.


Sent via iPhone

> On Jun 10, 2016, at 21:25, Joe Lawson <jlawson@opensourceconnections.com> wrote:
> So 1.7 would support Kafka 0.9. The issues are pretty minimal around the sink in master.
Have you tried it? As for other means flume feels the purest in not manipulating the data.
If you are shipping logs, Logstash (probably a bad word here :) is an option as it supports
Kafka 0.9.
> You could do something more hacky like piping into the Kafka console producer too.
> I know (elastic's) beats supports Kafka 0.9 in master but don't think it is released
either. Remember Kafka 0.9 came out in Oct or Dec so it's pretty durn new compared to 0.8
which had over a year to get momentum. Let's not even talk about 0.10 ;)
> maybe this helps?
> -Joe
> On Jun 10, 2016 10:34 PM, "Jason J. W. Williams" <jasonjwwilliams@gmail.com> wrote:
> Yeah. 
> Flume (relay)----WAN (Avro SSL)---->Flume (concentrator)--->Kafka.
> The remote Flume is talking to a "concentrator" flume in the Kafka datacenter, so that
the data is transferred via SSL over the WAN, since the Kafka sink doesn't support SSL in
1.6.0. Otherwise, we could push from the remote Flume directly into Kafka over the WAN.
> -J
>> On Fri, Jun 10, 2016 at 8:29 PM, Joe Lawson <jlawson@opensourceconnections.com>
>> Do you mean that you are replicating topics over WAN using Flume as a shipping agent?
Sorry if I misunderstood the scenario.
>>> On Jun 10, 2016 9:58 PM, "Jason J. W. Williams" <jasonjwwilliams@gmail.com>
>>> Thanks Joe that helps. What do you see folks using instead of Flume? We're using
largely a WAN relay for Kafka for remote sites that don't warrant a local cluster.
>>> -J
>>>> On Fri, Jun 10, 2016 at 6:17 PM, Joe Lawson <jlawson@opensourceconnections.com>
>>>> Flume seems to have lost some stream. It's improving but it feels like the
priorities have become somewhat introspective so I'd say build from master is you want the
>>>> Don't hold your breath for 1.7. just an opinion. No bearing on reality.
>>>> -Joe
>>>>> On Jun 10, 2016 12:45 PM, "Jason J. W. Williams" <jasonjwwilliams@gmail.com>
>>>>> Is there an estimated release date for Flume 1.7.0? Having TLS support
in the Kafka sink would allow us to simplify our topology a lot.=20
>>>>> -J

View raw message