flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From أنس الليثي <dev.fano...@gmail.com>
Subject How to configure multiple flume agents for failover
Date Thu, 31 Mar 2016 08:21:23 GMT

We are planning to use flume in consuming data from Gnip. We already
configured a flume agent that uses a customized source to read from gnip
and write the streamed data to HDSF.

This agent now is considered a single point of failure in the architecture.
If this agent goes down for any reason, no data will be read and the system
will not be updated.

Is there a way to configure multiple agents doing the same job but only one
is working and the others in something like standby mode. If the working
agent is down for any reason, one of the other standby agents works

We can achieve this using a monitoring tool like nagios  that monitor the
status of the agent and take an action based on that status. If the agent
is down for 3 subsequent checks, start another agent or restart the same

Is there anything like this from flume side? something like a master flume
agent monitoring the other agents and so on ?

Best regards

View raw message