flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: New Flafka component - "kafka consumer" channel
Date Fri, 28 Aug 2015 00:56:33 GMT
Nope. You can put anything you want, just set parseAsFlumeEvent to false
and the channel won't attempt to convert it into a Flume event. It just
stashes the whole thing into the body of the returned event.


Thanks,
Hari

On Thu, Aug 27, 2015 at 5:53 PM, Roshan Naik <roshan@hortonworks.com> wrote:

> My understanding is that the Kafka channel expects "Flume Event" objects
> to be stored in the Kafka topic.
> Isn't that right ?
> -roshan
>
>
> On 8/27/15 5:47 PM, "Hari Shreedharan" <hshreedharan@cloudera.com> wrote:
>
> >So one of the things that the already existing Kafka channel can do is to
> >run without a source. Does this outperform that as well? I have already
> >seen people use it this way.
> >
> >
> >Thanks,
> >Hari
> >
> >On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <roshan@hortonworks.com>
> >wrote:
> >
> >> Wanted to give a heads-up on this idea I have been working on Š
> >>
> >> Using Flume as a Kafka producer or consumer has been gaining popularity
> >> thanks to the Flafka components that were recently introduced.
> >>
> >> For the use case of Flume as a Kafka consumer, it appears we can
> >>sidestep
> >> the compromise between Mem channel (which is fast but can lose data) and
> >>  File channel (which is slow but won't lose data) and get the best of
> >>both
> >> worlds.
> >>
> >> I have a prototype of this idea  for a "Kafka Consumer" channel.  It is
> >> designed to enable the use of Flume as a really light weight and very
> >>fast
> >> Kafka consumer without the data loss potential of mem channel.  My
> >> measurements indicate it easily outperforms memory channel.
> >>
> >> Additional info here  Š
> >> https://github.com/roshannaik/kafka-consumer-channel
> >>
> >> I think the same idea could be applied for "Kafka producer channel".
> >>
> >> -roshan
> >>
>
>

Mime
View raw message