flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjie Lai <mjla...@gmail.com>
Subject Re: Use metadata field for output bucketing
Date Sat, 05 Nov 2011 00:02:42 GMT

You can try escapedFormatDfs. Here is example:

$ bin/flume node_nowatch -n f1 -c 'f1: text("/tmp/aa.txt") | 
value("src", 123) collector(2000){escapedFormatDfs("file:///tmp", 
"aaaa-%{src}" )};'

$ ls /tmp
aaaa-123
...

Should also work for s3.

Mingjie

On 11/04/2011 03:17 PM, Shuang wrote:
> Hi, guys,
>    After reading the Flume User Guide, I thought this is possible, but
> would like to confirm with your guys. Currently, I have collectors
> configured as:
> collectorSink("s3n://mybucket/flume/%Y-%m-%d/\",\"log-%{host}-\", 3600000),
> and I have a field called "source" in Flume event's metadata table, and
> would like to use in the collectorSink path, something like this:
> collectorSink("s3n://mybucket/flume/%{metadata['source']}/%Y-%m-%d/\",\"log-%{host}-\",
> 3600000),
>
> I wonder what's the right syntax to refer to field in the metadata
> table. I search the user guide and couldn't find any example. Also I'd
> like to point out, this is kind of similar to how Scribe's message
> category is used.
>
> Shuang

Mime
View raw message