flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark <static.void....@gmail.com>
Subject Re: Transforming value
Date Tue, 09 Aug 2011 18:28:47 GMT
Ok just wanted to check before writing something custom.

Do you forsee adding such ability? It would sure help those of use who 
are switching from other aggregation frameworks such a rsyslog or syslog-ng


On 8/9/11 11:25 AM, Eric Sammer wrote:
> Mark:
> The extractor is just that. In other words, it doesn't mutate the
> original message, just pulls bits out into attributes. Today, there's
> no way to change the message (more of a substitute than a simple
> capturing match) without writing a customer deco. Sorry.
> On Tue, Aug 9, 2011 at 11:04 AM, Mark<static.void.dev@gmail.com>  wrote:
>> Thanks I figured I could use one of those decorators but I'm not sure of the
>> syntax on how to apply when also using a regex to parse out a value to use
>> for bucketing.
>> This is what I'm using:
>>   - exec multiconfig 'localhost: syslogTcp(10514) | { regex("\d+ (\w+)", 1,
>> "type") =>  collectorSink("file:///tmp/testing/%{type}", "" ) } ;'
>> And I have messages in the following format:
>>   - "12345 FOO this is my message"
>> How can I bucket on "foo" and save the following message "this is my
>> message"? Right now, bucketing works but the whole string gets saved as the
>> message.
>> Thanks
>> On 8/9/11 10:38 AM, Eric Sammer wrote:
>>> Mark:
>>> The regex[1] (or regexAll) decorators should do what you want. They
>>> apply a given regex to the body of an event and put a capture in an
>>> attribute field. It's primarily for things such as extracting content
>>> that one can then bucket output by.
>>> If you need to do something fancier than this, you'll have to
>>> implement a decorator yourself (or file a JIRA to request an
>>> additional feature of an existing deco).
>>> [1]
>>> http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_flume_sink_decorator_catalog
>>> On Tue, Aug 9, 2011 at 10:08 AM, Mark<static.void.dev@gmail.com>    wrote:
>>>> Thread working?
>>>> On 8/9/11 7:56 AM, Mark wrote:
>>>>> I'm not quite sure if this is possible but is there a way to transform
>>>>> the message before saving into a collector source. I see that regex can
>>>>> be used to use dynamic values for the path/filename of the the sink but
>>>>> I can not find out how to apply this to the value of the message. For
>>>>> example, we would like to use regex to parse out a certain value from
>>>>> the message... similar to rsyslog message templates.
>>>>> Thanks

View raw message