flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: New Flafka component - "kafka consumer" channel
Date Fri, 28 Aug 2015 00:47:03 GMT
So one of the things that the already existing Kafka channel can do is to
run without a source. Does this outperform that as well? I have already
seen people use it this way.


Thanks,
Hari

On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <roshan@hortonworks.com> wrote:

> Wanted to give a heads-up on this idea I have been working on …
>
> Using Flume as a Kafka producer or consumer has been gaining popularity
> thanks to the Flafka components that were recently introduced.
>
> For the use case of Flume as a Kafka consumer, it appears we can sidestep
> the compromise between Mem channel (which is fast but can lose data) and
>  File channel (which is slow but won't lose data) and get the best of both
> worlds.
>
> I have a prototype of this idea  for a "Kafka Consumer" channel.  It is
> designed to enable the use of Flume as a really light weight and very fast
> Kafka consumer without the data loss potential of mem channel.  My
> measurements indicate it easily outperforms memory channel.
>
> Additional info here  …
> https://github.com/roshannaik/kafka-consumer-channel
>
> I think the same idea could be applied for "Kafka producer channel".
>
> -roshan
>

Mime
View raw message