flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Crobak <joec...@gmail.com>
Subject buffered sink decorator practices
Date Wed, 10 Aug 2011 14:46:33 GMT
We've written a simple sink decorator to do in-memory aggregations.
 Currently, we're using a roll sink to cause the aggregator decorator to be
closed/reopened ever 60 seconds.  Based upon the info in [1], by default the
close() operation has 30 seconds to complete.  We're seeing this fail in
some cases due to other bottlenecks. I'm hesitant to just up the timeout,
though, since long GCs or other events could cause the problem regardless of
the timeout.

With all this in mind, I have two questions.
1) Rollsink and BatchingDecorator seem to share a lot of similar logic to
run a background thread to flush events periodically. There seems to be a
lot of subtly in these implementations to avoid deadlocks.  Are either of
these suitable for subclassing? (I guess BatchingDecorator is closer to what
I'm looking for)... has anyone ever done this before?
2) It's possible for our sink decorator to generate more events than it
receives, so I am afraid it could become behind -- are there dangers in
using a threadpool to call append() from a decorator to forward events to
the collector?



View raw message