flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Crobak <joec...@gmail.com>
Subject Re: E2E mode with decorators
Date Tue, 16 Aug 2011 15:29:08 GMT
On Tue, Aug 16, 2011 at 10:59 AM, Felix Giguere Villegas <
felix.giguere@mate1inc.com> wrote:

> Maybe I'm missing something, but why don't you put your filtering Decorator
> on the agent/source instead?
> What's the point of sending those events all the way to the CollectorSink
> if they're going to be filtered out in the end? The only reason I can see is
> if the only place where you can easily determine which events to filter out
> IS at the end of the flow, but I can't think of a reason why that would be
> the case...
> Good question - this might be worth investigating.  The other reason I can
think of is if you're forwarding data from your collector to multiple sinks
-- e.g. both HDFS and Hbase, but perhaps filtering out some of the data for
one or the other (we're not doing this, and I guess it doesn't work in E2E
mode according to FLUME-165 anyway).

I simplified our situation in my example -- we're really doing a
light-weight ETL with an in-memory aggregation. So for each event that comes
in, 0 or more events might come out of that event -- it's not just
filtering, and it's not 1 event in, 1 event out.

> I don't know the answer to your specific question (and I'd be curious to
> find out as well), so I'm sorry if my comment doesn't help :) ...
> --
> Felix
> On Tue, Aug 16, 2011 at 10:52 AM, Joe Crobak <joecrow@gmail.com> wrote:
>> According to the Flume FAQ [1], Flume ack's events from the CollectorSink
>> in E2E mode.  If I have a Decorator running on the Collector that filters
>> out events (or transforms them or something), does that mean those events
>> won't get ACK'd and thus will delivery will be retried for them
>> indefinitely? IOW, is E2E mode unsupported in this situation -- or maybe is
>> there a way for me to ACK events that I want to filter from the Decorator
>> itself?
>> Thanks,
>> Joe
>> [1] https://github.com/cloudera/flume/wiki/FAQ

View raw message