flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject New Flafka component - "kafka consumer" channel
Date Thu, 27 Aug 2015 23:11:49 GMT
Wanted to give a heads-up on this idea I have been working on ...

Using Flume as a Kafka producer or consumer has been gaining popularity thanks to the Flafka
components that were recently introduced.

For the use case of Flume as a Kafka consumer, it appears we can sidestep the compromise between
Mem channel (which is fast but can lose data) and  File channel (which is slow but won't lose
data) and get the best of both worlds.

I have a prototype of this idea  for a "Kafka Consumer" channel.  It is designed to enable
the use of Flume as a really light weight and very fast Kafka consumer without the data loss
potential of mem channel.  My measurements indicate it easily outperforms memory channel.

Additional info here  ...
https://github.com/roshannaik/kafka-consumer-channel

I think the same idea could be applied for "Kafka producer channel".

-roshan

Mime
View raw message