flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From NerdyNick <nerdyn...@gmail.com>
Subject Re: filter out events
Date Wed, 27 Jul 2011 15:19:16 GMT
Seeing as this idea of filtering events comes up a lot do you think a
decorator built into Flume that takes a RegEx and if it matches it
doesn't send the event would solve most use-cases out there. If this
sounds like a good idea I'll throw a Jira task up to do it as well as
make notes about the limitations to make sure they get into the docs
about the decorator. Of course we got to wait tell the new Jira is
ready.

On Mon, Jul 25, 2011 at 4:41 PM, Jonathan Hsieh <jon@cloudera.com> wrote:
> [Please subscribe to new flume-user@incubator.apache.org list,
> bcc flume-user@cloudera.org, cc flume-user@incubator.apache.org]
> Lior,
> You could write a decorator plugin in that only propagates the message that
> meet a particular criteria and not send the others.
> If you used BE or DFO mode, you could put this at the agent or the
> collector.
> If you are using this with the e2e mode, this would need to go to the agent.
>  This is because you would need to make sure that this filter happens
> *before* it gets to the agentE2E.  If it is after, you will remove messages
> that are needed to calculate a proper checksum!
> Jon.
>
> On Tue, Jul 12, 2011 at 11:21 PM, Lior H <harel.lior@gmail.com> wrote:
>>
>> I am looking for a way to drop events from flowing down the flume
>> stream.
>> For example, when tailing a directory of log files from different
>> processes, how would one collect errors and warnings only?
>
>
>
> --
> // Jonathan Hsieh (shay)
> // Software Engineer, Cloudera
> // jon@cloudera.com
>
>



-- 
Nick Verbeck - NerdyNick
----------------------------------------------------
NerdyNick.com
Coloco.ubuntu-rocks.org

Mime
View raw message