flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Madhu Gmail <madhu.munag...@gmail.com>
Subject Re: Data in File-channel data folder
Date Thu, 11 Apr 2013 20:11:50 GMT
Thank you Hari.

This is very useful information.

Let me try that scenario where I will send more data to create log-3.

Thanks

Madhu  Munagala


On Apr 11, 2013, at 2:57 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:

> The File Channel deletes the files from which all events have been removed at the time
of a checkpoint. But, the channel will keep 2 log files per data directory even if all its
events are taken out. Once all events from log-1, log-2 are taken out and then events get
written to log-3, log-1 will be deleted at the next checkpoint. Unless a log-3 is created,
both log-1 and log-2 will not be deleted.
> 
> 
> 
> Hari
> 
> -- 
> Hari Shreedharan
> 
> On Thursday, April 11, 2013 at 12:13 PM, Madhu Gmail wrote:
> 
>> Hello,
>> 
>> I have not heard from anyone. so just want make sure I have explained the issue correctly.
>> 
>> I think this is a common problem for everyone who uses it flume.
>> 
>> when flume sink consumes the log event from file channel, what will happen to the
data that is committed to local disk under data directory.
>> 
>> will it grow indefinitely like log-1, log-2, log-3.....and so on ???
>> 
>> do I have to write script to remove the data from data directory ??
>> 
>> 
>> 
>> Madhu Munagala
>> (214)679-2872
>> 
>> On Apr 11, 2013, at 11:52 AM, Madhu Gmail <madhu.munagala@gmail.com> wrote:
>> 
>>> Hello,
>>> 
>>> How to clean up the data in file channel data folder. After the log events are
processed by the sink, I still see the log-1 and log-2 shows 1.6GB and 1.2GB.
>>> 
>>> once the log events are processed by the sink, the channel should not have any
data in data directory under file-channel ....??
>>> 
>>> 
>>> Madhu Munagala
>>> (214)679-2872
> 

Mime
View raw message