flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AnilKumar B <akumarb2...@gmail.com>
Subject Re: flume fail over mechanism
Date Wed, 15 Jan 2014 17:01:40 GMT
Thanks Ehsan.

Thanks & Regards,
B Anil Kumar.


On Wed, Jan 15, 2014 at 3:33 PM, Ehsan Haq <ehsan.haq@klarna.com> wrote:

> Hi,
> 1) If the source and sinks provided by the community are good enough for
> you then don't invent yours. I think there are alot of work already done,
> you can try those before writing your own source/sink.
>
> 2) For reliability you should be using file bases channel, which you
> already plan to. For message failures I guess you need to handle it
> yourself. You probably need to write some code there in order to catch the
> failing message and routing them to some other channel, a more lenient one
> like file or for the case of hbase you could write the whole failing
> message in a different table.
> You will also need to deal with duplicate messages if you are planning to
> use a reliable channel (like file based channel).
>
> /Ehsan
>
>
> On Wed, Jan 15, 2014 at 10:43 AM, AnilKumar B <akumarb2010@gmail.com>wrote:
>
>> I am planning to use file based channels.
>>
>> Thanks & Regards,
>> B Anil Kumar.
>>
>>
>> On Wed, Jan 15, 2014 at 3:12 PM, AnilKumar B <akumarb2010@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> In our pipeline we are thinking of using flume, our data source can be
>>> either filer or hbase or it can be couchbase also and sink is either
>>> filer(down stream's) or another hbase cluster(down stream's).
>>>
>>> So I need some help in following.
>>> 1) To handle multiple sources and sinks, do I need to write custom flume
>>> sink and source? or I should use community's respective source and sinks?
>>>
>>> 2) For us, we cannot miss any data, Is there any mechanism in flume to
>>> handle failed messages, I mean suppose flume failed write the records into
>>> hbase, how exactly it will takes care? Or should I maintain state of each
>>> record and based on it's state I am thinking of handling failed messages,
>>> Is that correct way? I am trying to use zookeeper for state management. So
>>> just want to know, whether my approach is correct or not.
>>>
>>>
>>> Thanks & Regards,
>>> B Anil Kumar.
>>>
>>
>>
>
>
> --
> *Muhammad Ehsan ul Haque*
> Klarna AB
> Norra Stationsgatan 61
> SE-113 43 Stockholm
>
> Tel: +46 (0)8- 120 120 00
> Fax: +46 (0)8- 120 120 99
> Web: www.klarna.com
>

Mime
View raw message