flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: flume write ahead log is finished
Date Tue, 04 Dec 2012 21:08:35 GMT
You probably want the JSon reporting:
http://flume.apache.org/FlumeUserGuide.html#monitoring

Brock

On Tue, Dec 4, 2012 at 3:02 PM, Shengjie Min <kelvin.msj@gmail.com> wrote:
>>Generally you should be fine to bring down the file channel at any
>>point. That is why we have the WAL.
>
> I understand WAL is there in case you restart your flume agent. But in my
> case I am switching from one flume agent to another type, let's say from
> tailsource to AMQP agent. So before I shutdown tailsource I really want to
> make sure WAL is all finished. How do you export the channel size in this
> case?
>
>
> On 4 December 2012 17:36, Brock Noland <brock@cloudera.com> wrote:
>>
>> Generally you should be fine to bring down the file channel at any
>> point. That is why we have the WAL.
>>
>> If you wanted to be sure the channel was empty, we do export the
>> channel size (depth).
>>
>> On Tue, Dec 4, 2012 at 11:26 AM, Shengjie Min <kelvin.msj@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > When it comes to maintenance outage, flume needs to be shutdown and
>> > offline
>> > for a while. We want to make sure flume's WAL is completed before we
>> > shut
>> > down the process. Is there anyway or any metrics exposed can indicate
>> > WAL is
>> > fully processed so that it's safe to bring it down gracefully?
>> >
>> > Of coz, the assumption here is that, the source of flume is not writing
>> > any
>> > new data, flume only has a certain size backlog needs to be cleaned up.
>> >
>> >
>> > --
>> > All the best,
>> > Shengjie Min
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>
>
>
>
> --
> All the best,
> Shengjie Min
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message