flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjie Lai <mjla...@gmail.com>
Subject Re: using flume to manage all log rotation
Date Thu, 10 Nov 2011 19:27:40 GMT

+1 logrotate.

Off the topic a little bit:

@Otis. Can sematext add flume project to search-hadoop.com? 
search-hadoop's an awesome tool. So later on, we can point to it for 
previous flume email threads.


On 11/09/2011 07:46 PM, Otis Gospodnetic wrote:
> Hi Andrew,
> We use logrotate to rotate, gzip, and purge old log files.
> We also use tail -F from Flume because Flume's own tail has its issues
> (discussed a lot on the old Flume ML 6 or more months ago)
> Otis
> ----
> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
> Lucene ecosystem search :: http://search-lucene.com/
>     ------------------------------------------------------------------------
>     *From:* Andrew McNair <andrew.mcnair@gmail.com>
>     *To:* flume-user@incubator.apache.org
>     *Sent:* Wednesday, November 9, 2011 7:10 PM
>     *Subject:* using flume to manage all log rotation
>     Hi,
>     I'm interested in using flume to manage all log rotation in my
>     deployment. Currently flume is tailing a logfile. Every night log4j
>     creates a gz backup of the previous day's logfile. There's a cron
>     job to delete older gz backup files.
>     The problems with this approach are:
>     - Log4j's (apache-log4j-extras's) RollingFileAppender isn't super
>     robust. A few days ago it locked up all threads while it gzipped the
>     logfiles
>     - I need a strategy to handle the flume agent going down and missing
>     events because log4j has rotated the file flume is tailing
>     - Once Flume has delivered the content of the logfile I don't need
>     the gz backups, so they're just wasting space
>     What I'd really like is to have flume handle rolling over the
>     original files, and deleting backup files once flume is fairly
>     confident it's delivered the contents to the source. Has anybody
>     setup something like this? Any pointers would be much appreciated.
>     Thanks,
>     Andrew

View raw message