flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Everton" <...@iocaine.org>
Subject Re: Fwd: Is it possible to get the file name from flume agent
Date Wed, 27 Jul 2011 04:23:06 GMT
You can get the original file name (if you're using any of the tail
sources) from the tailSrcFile attribute. We use it like this:

collectorSink("file:///logs/flume/apps/%{application}/access/%Y-%m-%d/%H/",
"%{tailSrcFile}-%{host}-", 300000),

The second argument is the file name that will be used. If you're not
using the tail sources, you could add a custom attribute to your flow
definition that contains the file name.

As for parsing the file name and using that to create the folder names,
this thread
(https://groups.google.com/a/cloudera.org/group/flume-user/msg/9e78675c759223ed)
has an example of using the regex extractor to pull parts of the message
out to use in the collector file name.

Cheers,
Dan 

On Wed, 27 Jul 2011 12:03 +0800, "Junxian Yan" <junxian.yan@gmail.com>
wrote:
> Copy this question to new group
> 
> ---------- Forwarded message ----------
> From: Junxian Yan <junxian.yan@gmail.com>
> Date: Tue, Jul 26, 2011 at 2:59 PM
> Subject: Re: Is it possible to get the file name from flume agent
> To: Flume Users <flume-user@cloudera.org>
> 
> 
> Or maybe a more specific description for my question:
> Is it possible to get the original file name from flume agent output
> 
> R
> 
> 
> On Tue, Jul 26, 2011 at 2:58 PM, Junxian Yan <junxian.yan@gmail.com>
> wrote:
> 
> > Hi Guys
> >
> > I want to customize the collector output file name and the storing path by
> > the log file name.
> > Such as if the log name is a-b.log
> > I want the storing path is like /a/b/a-b.log
> >
> > Is it possible?
> >
> >
> >
> 

Mime
View raw message