flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "samnik60 ." <monishs...@gmail.com>
Subject Re: Flume NG High Availability
Date Wed, 02 Mar 2016 19:51:44 GMT
Thanks Gonzalo,

In my other use case, i want to achive the same HA i have mentioned before
( active/active or active/stand by )  using spool directory source, so
going by your response , if i use a spool directory as source i cannot have
active/active or active/stand by HA , since active/active will result in
race condition when two source try to process from same directory.


On Wed, Mar 2, 2016 at 1:17 PM, Gonzalo Herreros <gherreros@gmail.com>

> Jms queues guarantee only one of the clients will get each message.
> Unless you build it yourself, Flume doesn't have active/pasive. HA is
> achieved by having multiple agents running the same configuration.
> On 2 March 2016 at 18:12, samnik60 . <monishsvce@gmail.com> wrote:
>> Hi guys,
>> I have the following queries about flume ng High Availability,
>> - Is it possible to have active/active or active/standby flume agent? If
>> so kindly point me to a document to refer, i am unable to find it in flume
>> documentation.
>> The Use Case i am trying to address,
>> I  want to run a flume agent in a node which will take events from a jms
>> queue and persist it in hdfs. If the flume agent goes down , i want another
>> flume agent running in another node to take over this responsiblity. (
>> active/standby)
>> I dont mind having both flume agents trying to process events from same
>> jms queue , provided a event is taken by only one flume agent without race
>> condition which i feel is not practical ( active/active).
>> Any information on this is greatly appreciated.
>> Thanks,
>> Sam.

View raw message