flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@googlemail.com>
Subject Re: flume agent Error
Date Wed, 14 Mar 2012 07:57:25 GMT

You could the agent config reload over the CLI the configuration. 

Let me say some notes:
I know that tail and tailDir are the coolest features in flume. But, you have to notice, flume
uses a tail, which means that act like a tail -f in a console. The file will be larger, the
memory will be higher used. The tail process will be restarted, the marker get lose and it
will start again from the beginning of the file (or end, if use use that switch). So, the
loosing of threads in a larger setup is a cause of that.

You can prevent it if you roll the tailed files after they are written into HDFS. Or do not
use tail, use syslog or avro instead. Or, when you use tailDir spilt the directories to catch
only small numbers of files.


Alexander Lorenz

On Mar 12, 2012, at 4:38 PM, Chalcy Raja wrote:

> Hi,
> I am using one flume agent to taildir a few directories.  One agent thread went into
an ERROR state.  I see a file stuck in error state.  I know starting the agent again can resolve
the issue, but would like to know the cause of the error state.
> 1.How would I find the cause of the error state?  I checked the log on the agent and
could not find any error log.
> 2. How to restart only that thread without having to restart the agent?
> Any answer is appreciated.
> Thanks,
> Chalcy

View raw message