flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From أنس الليثي <dev.fano...@gmail.com>
Subject Re: How to configure multiple flume agents for failover
Date Thu, 31 Mar 2016 08:38:13 GMT
yes, you are correct.

Maybe I used the wrong term in the title and the correct term is high
availability not failover.


On 31 March 2016 at 11:35, Asim Zafir <asim.zafir@gmail.com> wrote:

> If I understand it right you want to have multiple instances of flume-ng
> (flume agents) to run on the source - such as webserver for example? but do
> not want all of them active at once? Is your goal is to achieve HA over
> flume agents  or something else?
> On Thu, Mar 31, 2016 at 1:21 AM, أنس الليثي <dev.fanooos@gmail.com> wrote:
>> Dears
>> We are planning to use flume in consuming data from Gnip. We already
>> configured a flume agent that uses a customized source to read from gnip
>> and write the streamed data to HDSF.
>> This agent now is considered a single point of failure in the
>> architecture.
>> If this agent goes down for any reason, no data will be read and the
>> system will not be updated.
>> Is there a way to configure multiple agents doing the same job but only
>> one is working and the others in something like standby mode. If the
>> working agent is down for any reason, one of the other standby agents works
>> automatically.
>> We can achieve this using a monitoring tool like nagios  that monitor the
>> status of the agent and take an action based on that status. If the agent
>> is down for 3 subsequent checks, start another agent or restart the same
>> agent.
>> Is there anything like this from flume side? something like a master
>> flume agent monitoring the other agents and so on ?
>> Best regards

Anas Rabei
Senior Software Developer

View raw message