flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hai Thai" <...@mailtrust.com>
Subject Re: Interceptors on a replicated source fan-out
Date Thu, 16 Jun 2016 16:06:29 GMT
Thank you both for your answers,

A follow up question which is related: when using and configuring the multiplexing channel
selector, is it possible to send one message to multiple channels? The goal is to send all
messages to one channel while a filtered subset of those messages also go to another channel.

The other option that could be considered is to extend the AbstractChannelSelector to accomplish


-----Original Message-----
From: "Jeong-shik Jang" <jsjangg@gmail.com>
Sent: Thursday, June 16, 2016 6:10am
To: user@flume.apache.org
Subject: Re: Interceptors on a replicated source fan-out

Hi Ha Thai,

I think implementing a sink wrapper which allows you to use interceptors in
sink side can be an option.
Actually, I implemented one and have been using it successfully in our
Possible performance gain by leveraging multi-threaded processing of sinks
is also a benefit of this approach, which was the motivation that drove me
to implement it.

If you want to try to implement by yourself, I can give you some idea; my
company has plan to make it open source; but it doesn't happen yet.


2016-06-16 17:53 GMT+09:00 Chris Horrocks <chris@hor.rocks>:

> Interceptors are configured per source, to achieve the result you describe
> it would be easier to apply the interceptor to the source of the receiving
> agent. The alternative would be to use the morphline interceptor and write
> some conditional logic, but I'm not sure if the current interceptor
> implementations would allow you to conditionally bind certain events to a
> specific channel. I suppose you could always write a custom interceptor to
> do it.
> --
> Chris Horrocks
> From: Hai Thai <hai@mailtrust.com> <hai@mailtrust.com>
> Reply: user@flume.apache.org <user@flume.apache.org>
> <user@flume.apache.org>
> Date: 15 June 2016 at 22:40:07
> To: user@flume.apache.org <user@flume.apache.org> <user@flume.apache.org>
> Subject:  Interceptors on a replicated source fan-out
> When doing a replicated fan-out of a source, is there any way to apply an
> i=
> nterceptor to just 1 destination channel of that fan out while having the
> o=
> ther channels unaffected?
> -H

View raw message