flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew McNair <andrew.mcn...@gmail.com>
Subject using flume to manage all log rotation
Date Thu, 10 Nov 2011 00:10:11 GMT

I'm interested in using flume to manage all log rotation in my deployment.
Currently flume is tailing a logfile. Every night log4j creates a gz backup
of the previous day's logfile. There's a cron job to delete older gz backup

The problems with this approach are:
- Log4j's (apache-log4j-extras's) RollingFileAppender isn't super robust. A
few days ago it locked up all threads while it gzipped the logfiles
- I need a strategy to handle the flume agent going down and missing events
because log4j has rotated the file flume is tailing
- Once Flume has delivered the content of the logfile I don't need the gz
backups, so they're just wasting space

What I'd really like is to have flume handle rolling over the original
files, and deleting backup files once flume is fairly confident it's
delivered the contents to the source. Has anybody setup something like
this? Any pointers would be much appreciated.


View raw message