flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@googlemail.com>
Subject Re: Tail Rolling Files
Date Thu, 23 Feb 2012 22:13:43 GMT

pls send the config for your sink.
You create one file in a directory, and after that file has reached 1GB you reset that? 


Alexander Lorenz

On Feb 23, 2012, at 10:23 AM, Cameron Gandevia wrote:

> Hi
> We use the TailSource in flume to tail our log files. Whenever the log files reach 1gb
in size they are rolled. We are experiencing a problem where flume cannot keep up with the
log write rate and eventually missing an entire file because it rolls over before flume notices.
I think this happens because flume doesn't detect the roll until it finishes tailing the current
file. I am think a simple solution to start would be to include a timestamp in the log files
and use a tailDir so that every file is seen. My only concern is we will end up with more
and more tail threads as the writing gets further and further ahead.
> Just wondering if anyone else experienced and dealt with this problem?
> Thanks

View raw message