flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: New Flafka component - "kafka consumer" channel
Date Fri, 28 Aug 2015 00:53:56 GMT
My understanding is that the Kafka channel expects "Flume Event" objects
to be stored in the Kafka topic.
Isn't that right ?
-roshan


On 8/27/15 5:47 PM, "Hari Shreedharan" <hshreedharan@cloudera.com> wrote:

>So one of the things that the already existing Kafka channel can do is to
>run without a source. Does this outperform that as well? I have already
>seen people use it this way.
>
>
>Thanks,
>Hari
>
>On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <roshan@hortonworks.com>
>wrote:
>
>> Wanted to give a heads-up on this idea I have been working on Š
>>
>> Using Flume as a Kafka producer or consumer has been gaining popularity
>> thanks to the Flafka components that were recently introduced.
>>
>> For the use case of Flume as a Kafka consumer, it appears we can
>>sidestep
>> the compromise between Mem channel (which is fast but can lose data) and
>>  File channel (which is slow but won't lose data) and get the best of
>>both
>> worlds.
>>
>> I have a prototype of this idea  for a "Kafka Consumer" channel.  It is
>> designed to enable the use of Flume as a really light weight and very
>>fast
>> Kafka consumer without the data loss potential of mem channel.  My
>> measurements indicate it easily outperforms memory channel.
>>
>> Additional info here  Š
>> https://github.com/roshannaik/kafka-consumer-channel
>>
>> I think the same idea could be applied for "Kafka producer channel".
>>
>> -roshan
>>


Mime
View raw message