flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: using flume to manage all log rotation
Date Fri, 11 Nov 2011 21:44:32 GMT
Hi Mingjie - thanks for the kind words about search-hadoop.com! :)

Yes, Flume is on the list of projects to add!

Tell us if you are missing any features.

Otis
----

Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/


>________________________________
>From: Mingjie Lai <mjlai09@gmail.com>
>To: flume-user@incubator.apache.org
>Sent: Thursday, November 10, 2011 2:27 PM
>Subject: Re: using flume to manage all log rotation
>
>
>+1 logrotate.
>
>Off the topic a little bit:
>
>@Otis. Can sematext add flume project to search-hadoop.com? 
>search-hadoop's an awesome tool. So later on, we can point to it for 
>previous flume email threads.
>
>Thanks,
>Mingjie
>
>On 11/09/2011 07:46 PM, Otis Gospodnetic wrote:
>> Hi Andrew,
>>
>> We use logrotate to rotate, gzip, and purge old log files.
>> We also use tail -F from Flume because Flume's own tail has its issues
>> (discussed a lot on the old Flume ML 6 or more months ago)
>>
>> Otis
>> ----
>> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
>> Lucene ecosystem search :: http://search-lucene.com/
>>
>>     ------------------------------------------------------------------------
>>     *From:* Andrew McNair <andrew.mcnair@gmail.com>
>>     *To:* flume-user@incubator.apache.org
>>     *Sent:* Wednesday, November 9, 2011 7:10 PM
>>     *Subject:* using flume to manage all log rotation
>>
>>     Hi,
>>
>>     I'm interested in using flume to manage all log rotation in my
>>     deployment. Currently flume is tailing a logfile. Every night log4j
>>     creates a gz backup of the previous day's logfile. There's a cron
>>     job to delete older gz backup files.
>>
>>     The problems with this approach are:
>>     - Log4j's (apache-log4j-extras's) RollingFileAppender isn't super
>>     robust. A few days ago it locked up all threads while it gzipped the
>>     logfiles
>>     - I need a strategy to handle the flume agent going down and missing
>>     events because log4j has rotated the file flume is tailing
>>     - Once Flume has delivered the content of the logfile I don't need
>>     the gz backups, so they're just wasting space
>>
>>     What I'd really like is to have flume handle rolling over the
>>     original files, and deleting backup files once flume is fairly
>>     confident it's delivered the contents to the source. Has anybody
>>     setup something like this? Any pointers would be much appreciated.
>>
>>     Thanks,
>>     Andrew
>>
>>
>
>
>
Mime
View raw message