flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asim Zafir <asim.za...@gmail.com>
Subject Re: How to configure multiple flume agents for failover
Date Thu, 31 Mar 2016 08:35:31 GMT
If I understand it right you want to have multiple instances of flume-ng
(flume agents) to run on the source - such as webserver for example? but do
not want all of them active at once? Is your goal is to achieve HA over
flume agents  or something else?

On Thu, Mar 31, 2016 at 1:21 AM, أنس الليثي <dev.fanooos@gmail.com> wrote:

> Dears
> We are planning to use flume in consuming data from Gnip. We already
> configured a flume agent that uses a customized source to read from gnip
> and write the streamed data to HDSF.
> This agent now is considered a single point of failure in the
> architecture.
> If this agent goes down for any reason, no data will be read and the
> system will not be updated.
> Is there a way to configure multiple agents doing the same job but only
> one is working and the others in something like standby mode. If the
> working agent is down for any reason, one of the other standby agents works
> automatically.
> We can achieve this using a monitoring tool like nagios  that monitor the
> status of the agent and take an action based on that status. If the agent
> is down for 3 subsequent checks, start another agent or restart the same
> agent.
> Is there anything like this from flume side? something like a master flume
> agent monitoring the other agents and so on ?
> Best regards

View raw message