flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: New Flafka component - "kafka consumer" channel
Date Fri, 28 Aug 2015 00:53:56 GMT
My understanding is that the Kafka channel expects "Flume Event" objects
to be stored in the Kafka topic.
Isn't that right ?

On 8/27/15 5:47 PM, "Hari Shreedharan" <hshreedharan@cloudera.com> wrote:

>So one of the things that the already existing Kafka channel can do is to
>run without a source. Does this outperform that as well? I have already
>seen people use it this way.
>On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <roshan@hortonworks.com>
>> Wanted to give a heads-up on this idea I have been working on Š
>> Using Flume as a Kafka producer or consumer has been gaining popularity
>> thanks to the Flafka components that were recently introduced.
>> For the use case of Flume as a Kafka consumer, it appears we can
>> the compromise between Mem channel (which is fast but can lose data) and
>>  File channel (which is slow but won't lose data) and get the best of
>> worlds.
>> I have a prototype of this idea  for a "Kafka Consumer" channel.  It is
>> designed to enable the use of Flume as a really light weight and very
>> Kafka consumer without the data loss potential of mem channel.  My
>> measurements indicate it easily outperforms memory channel.
>> Additional info here  Š
>> https://github.com/roshannaik/kafka-consumer-channel
>> I think the same idea could be applied for "Kafka producer channel".
>> -roshan

View raw message