flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark <static.void....@gmail.com>
Subject Re: Transforming value
Date Tue, 09 Aug 2011 18:04:56 GMT
Thanks I figured I could use one of those decorators but I'm not sure of 
the syntax on how to apply when also using a regex to parse out a value 
to use for bucketing.

This is what I'm using:
  - exec multiconfig 'localhost: syslogTcp(10514) | { regex("\d+ (\w+)", 
1, "type") => collectorSink("file:///tmp/testing/%{type}", "" ) } ;'

And I have messages in the following format:
  - "12345 FOO this is my message"

How can I bucket on "foo" and save the following message "this is my 
message"? Right now, bucketing works but the whole string gets saved as 
the message.


On 8/9/11 10:38 AM, Eric Sammer wrote:
> Mark:
> The regex[1] (or regexAll) decorators should do what you want. They
> apply a given regex to the body of an event and put a capture in an
> attribute field. It's primarily for things such as extracting content
> that one can then bucket output by.
> If you need to do something fancier than this, you'll have to
> implement a decorator yourself (or file a JIRA to request an
> additional feature of an existing deco).
> [1] http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_flume_sink_decorator_catalog
> On Tue, Aug 9, 2011 at 10:08 AM, Mark<static.void.dev@gmail.com>  wrote:
>> Thread working?
>> On 8/9/11 7:56 AM, Mark wrote:
>>> I'm not quite sure if this is possible but is there a way to transform
>>> the message before saving into a collector source. I see that regex can
>>> be used to use dynamic values for the path/filename of the the sink but
>>> I can not find out how to apply this to the value of the message. For
>>> example, we would like to use regex to parse out a certain value from
>>> the message... similar to rsyslog message templates.
>>> Thanks

View raw message