flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cameron Gandevia <cgande...@gmail.com>
Subject Tail Rolling Files
Date Thu, 23 Feb 2012 18:23:48 GMT

We use the TailSource in flume to tail our log files. Whenever the log
files reach 1gb in size they are rolled. We are experiencing a problem
where flume cannot keep up with the log write rate and eventually missing
an entire file because it rolls over before flume notices. I think this
happens because flume doesn't detect the roll until it finishes tailing the
current file. I am think a simple solution to start would be to include a
timestamp in the log files and use a tailDir so that every file is seen. My
only concern is we will end up with more and more tail threads as the
writing gets further and further ahead.

Just wondering if anyone else experienced and dealt with this problem?


View raw message