flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@gmail.com>
Subject Re: Restarts without data loss
Date Mon, 09 Jul 2012 06:46:14 GMT
Simple solution:

Two configs on different ports, iptables with transparent forwarding to both ports. Block
the first one, all events will be redirected to the other port. Wait 5 minutes, the mem channel
should be clear now. Do you changes, start the new config, redirect the traffic to these port
and change the other config.


On Jul 9, 2012, at 8:29 AM, Arvind Prabhakar wrote:

> Hi,
> On Sun, Jul 8, 2012 at 11:18 PM, Senthilvel Rangaswamy <senthilvel@gmail.com
>> wrote:
>> We are using Flume 1.2.0 with memory channel. When we rollout new
>> configs/decorators
>> we may need to restart flume at which point any events in memory channel
>> is gone. Any
>> ways to avoid this ?
> One way to address this would be to make sure that the upstream sink or
> client can be routed to a different agent when necessary. That way when you
> do want to restart the file channel, you would first route all the traffic
> elsewhere, drain the channel and then do the shutdown as necessary. Once
> the system is back up, you could route the traffic back to this agent.
> I am sure that there are multiple other ways of doing this.
> Regards,
> Arvind Prabhakar
>> Thanks,
>> --
>> ..Senthil
>> "If there's anything more important than my ego around, I want it
>> caught and shot now."
>>                                                    - Douglas Adams.

Alexander Alten-Lorenz
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

View raw message