flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Senthilvel Rangaswamy <senthil...@gmail.com>
Subject Re: Restarts without data loss
Date Mon, 09 Jul 2012 06:54:31 GMT
The source is a tail -F. So, I am not sure if this solution will work.

On Sun, Jul 8, 2012 at 11:46 PM, alo alt <wget.null@gmail.com> wrote:

> Simple solution:
>
> Two configs on different ports, iptables with transparent forwarding to
> both ports. Block the first one, all events will be redirected to the other
> port. Wait 5 minutes, the mem channel should be clear now. Do you changes,
> start the new config, redirect the traffic to these port and change the
> other config.
>
> cheers,
>  alex
>
>
> On Jul 9, 2012, at 8:29 AM, Arvind Prabhakar wrote:
>
> > Hi,
> >
> > On Sun, Jul 8, 2012 at 11:18 PM, Senthilvel Rangaswamy <
> senthilvel@gmail.com
> >> wrote:
> >
> >> We are using Flume 1.2.0 with memory channel. When we rollout new
> >> configs/decorators
> >> we may need to restart flume at which point any events in memory channel
> >> is gone. Any
> >> ways to avoid this ?
> >>
> >
> > One way to address this would be to make sure that the upstream sink or
> > client can be routed to a different agent when necessary. That way when
> you
> > do want to restart the file channel, you would first route all the
> traffic
> > elsewhere, drain the channel and then do the shutdown as necessary. Once
> > the system is back up, you could route the traffic back to this agent.
> >
> > I am sure that there are multiple other ways of doing this.
> >
> > Regards,
> > Arvind Prabhakar
> >
> >
> >
> >>
> >> Thanks,
> >> --
> >> ..Senthil
> >>
> >> "If there's anything more important than my ego around, I want it
> >> caught and shot now."
> >>                                                    - Douglas Adams.
> >>
> >>
>
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>


-- 
..Senthil

"If there's anything more important than my ego around, I want it
 caught and shot now."
                                                    - Douglas Adams.

Mime
View raw message