flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shengjie Min <kelvin....@gmail.com>
Subject Re: flume write ahead log is finished
Date Tue, 04 Dec 2012 21:02:15 GMT
>Generally you should be fine to bring down the file channel at any
>point. That is why we have the WAL.

I understand WAL is there in case you restart your flume agent. But in my
case I am switching from one flume agent to another type, let's say from
tailsource to AMQP agent. So before I shutdown tailsource I really want to
make sure WAL is all finished. How do you export the channel size in this
case?


On 4 December 2012 17:36, Brock Noland <brock@cloudera.com> wrote:

> Generally you should be fine to bring down the file channel at any
> point. That is why we have the WAL.
>
> If you wanted to be sure the channel was empty, we do export the
> channel size (depth).
>
> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <kelvin.msj@gmail.com>
> wrote:
> > Hi,
> >
> > When it comes to maintenance outage, flume needs to be shutdown and
> offline
> > for a while. We want to make sure flume's WAL is completed before we shut
> > down the process. Is there anyway or any metrics exposed can indicate
> WAL is
> > fully processed so that it's safe to bring it down gracefully?
> >
> > Of coz, the assumption here is that, the source of flume is not writing
> any
> > new data, flume only has a certain size backlog needs to be cleaned up.
> >
> >
> > --
> > All the best,
> > Shengjie Min
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>



-- 
All the best,
Shengjie Min

Mime
View raw message