flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@gmail.com>
Subject Re: Restarts without data loss
Date Mon, 09 Jul 2012 06:46:14 GMT
Simple solution:

Two configs on different ports, iptables with transparent forwarding to both ports. Block
the first one, all events will be redirected to the other port. Wait 5 minutes, the mem channel
should be clear now. Do you changes, start the new config, redirect the traffic to these port
and change the other config.

cheers,
 alex 


On Jul 9, 2012, at 8:29 AM, Arvind Prabhakar wrote:

> Hi,
> 
> On Sun, Jul 8, 2012 at 11:18 PM, Senthilvel Rangaswamy <senthilvel@gmail.com
>> wrote:
> 
>> We are using Flume 1.2.0 with memory channel. When we rollout new
>> configs/decorators
>> we may need to restart flume at which point any events in memory channel
>> is gone. Any
>> ways to avoid this ?
>> 
> 
> One way to address this would be to make sure that the upstream sink or
> client can be routed to a different agent when necessary. That way when you
> do want to restart the file channel, you would first route all the traffic
> elsewhere, drain the channel and then do the shutdown as necessary. Once
> the system is back up, you could route the traffic back to this agent.
> 
> I am sure that there are multiple other ways of doing this.
> 
> Regards,
> Arvind Prabhakar
> 
> 
> 
>> 
>> Thanks,
>> --
>> ..Senthil
>> 
>> "If there's anything more important than my ego around, I want it
>> caught and shot now."
>>                                                    - Douglas Adams.
>> 
>> 


--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message