flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@googlemail.com>
Subject Re: Tail Rolling Files
Date Thu, 23 Feb 2012 23:15:48 GMT
Hi,

pls send the config for your sink.
You create one file in a directory, and after that file has reached 1GB you reset that? 

best,
Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

On Feb 23, 2012, at 10:23 AM, Cameron Gandevia wrote:

> Hi
> 
> We use the TailSource in flume to tail our log files. Whenever the log files reach 1gb
in size they are rolled. We are experiencing a problem where flume cannot keep up with the
log write rate and eventually missing an entire file because it rolls over before flume notices.
I think this happens because flume doesn't detect the roll until it finishes tailing the current
file. I am think a simple solution to start would be to include a timestamp in the log files
and use a tailDir so that every file is seen. My only concern is we will end up with more
and more tail threads as the writing gets further and further ahead.
> 
> Just wondering if anyone else experienced and dealt with this problem?
> 
> Thanks
> 
> 


Mime
View raw message