flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: New Flafka component - "kafka consumer" channel
Date Fri, 28 Aug 2015 21:56:14 GMT
OK that's really good to know. We won't need an additional component if it
can function that way. Also in that case I would expect it to be quite
fast.

Will try to get some numbers next week. Glad I only spent a couple
evenings on that prototype.

-roshan


On 8/27/15 5:56 PM, "Hari Shreedharan" <hshreedharan@cloudera.com> wrote:

>Nope. You can put anything you want, just set parseAsFlumeEvent to false
>and the channel won't attempt to convert it into a Flume event. It just
>stashes the whole thing into the body of the returned event.
>
>
>Thanks,
>Hari
>
>On Thu, Aug 27, 2015 at 5:53 PM, Roshan Naik <roshan@hortonworks.com>
>wrote:
>
>> My understanding is that the Kafka channel expects "Flume Event" objects
>> to be stored in the Kafka topic.
>> Isn't that right ?
>> -roshan
>>
>>
>> On 8/27/15 5:47 PM, "Hari Shreedharan" <hshreedharan@cloudera.com>
>>wrote:
>>
>> >So one of the things that the already existing Kafka channel can do is
>>to
>> >run without a source. Does this outperform that as well? I have already
>> >seen people use it this way.
>> >
>> >
>> >Thanks,
>> >Hari
>> >
>> >On Thu, Aug 27, 2015 at 4:11 PM, Roshan Naik <roshan@hortonworks.com>
>> >wrote:
>> >
>> >> Wanted to give a heads-up on this idea I have been working on Š
>> >>
>> >> Using Flume as a Kafka producer or consumer has been gaining
>>popularity
>> >> thanks to the Flafka components that were recently introduced.
>> >>
>> >> For the use case of Flume as a Kafka consumer, it appears we can
>> >>sidestep
>> >> the compromise between Mem channel (which is fast but can lose data)
>>and
>> >>  File channel (which is slow but won't lose data) and get the best of
>> >>both
>> >> worlds.
>> >>
>> >> I have a prototype of this idea  for a "Kafka Consumer" channel.  It
>>is
>> >> designed to enable the use of Flume as a really light weight and very
>> >>fast
>> >> Kafka consumer without the data loss potential of mem channel.  My
>> >> measurements indicate it easily outperforms memory channel.
>> >>
>> >> Additional info here  Š
>> >> https://github.com/roshannaik/kafka-consumer-channel
>> >>
>> >> I think the same idea could be applied for "Kafka producer channel".
>> >>
>> >> -roshan
>> >>
>>
>>


Mime
View raw message