flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chalcy Raja <Chalcy.R...@careerbuilder.com>
Subject RE: flume agent Error
Date Wed, 14 Mar 2012 12:46:13 GMT
Thank you! Alex.  

I have set up taildir to switch to end ( true) of file.  Does this mean when the agent starts
again, we loose data or it will catch up from where left off?

How do I roll the tailed files after they are written to hdfs?

My config:
agent1: tailDir( "/mylogdir/", ".*\\.log", true, 0) | agentE2EChain("collector1:35853", "collector2:35853");
collector1: collectorSource( 35853 ) | collectorSink( "hdfs://mycluster/%Y-%m-%d-%H/", "logfragment-"
,12000); 

P.S. Looks like we may have to change to syslog.


Thanks,
Chalcy

-----Original Message-----
From: alo alt [mailto:wget.null@googlemail.com] 
Sent: Wednesday, March 14, 2012 3:57 AM
To: flume-user@incubator.apache.org
Subject: Re: flume agent Error

Hi,

You could the agent config reload over the CLI the configuration. 

Let me say some notes:
I know that tail and tailDir are the coolest features in flume. But, you have to notice, flume
uses a tail, which means that act like a tail -f in a console. The file will be larger, the
memory will be higher used. The tail process will be restarted, the marker get lose and it
will start again from the beginning of the file (or end, if use use that switch). So, the
loosing of threads in a larger setup is a cause of that.

You can prevent it if you roll the tailed files after they are written into HDFS. Or do not
use tail, use syslog or avro instead. Or, when you use tailDir spilt the directories to catch
only small numbers of files.

best,
 Alex 

--
Alexander Lorenz
http://mapredit.blogspot.com

On Mar 12, 2012, at 4:38 PM, Chalcy Raja wrote:

> Hi,
> 
> I am using one flume agent to taildir a few directories.  One agent thread went into
an ERROR state.  I see a file stuck in error state.  I know starting the agent again can resolve
the issue, but would like to know the cause of the error state.
> 
> 1.How would I find the cause of the error state?  I checked the log on the agent and
could not find any error log.
> 2. How to restart only that thread without having to restart the agent?
> 
> Any answer is appreciated.
> 
> Thanks,
> Chalcy
> 



Mime
View raw message