flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Crobak <joec...@gmail.com>
Subject Re: buffered sink decorator practices
Date Tue, 16 Aug 2011 20:05:58 GMT
On Tue, Aug 16, 2011 at 2:53 PM, Mingjie Lai <mjlai09@gmail.com> wrote:

> Joe.
> I have a similar case that I need to accumulate some data at a decorator
> and write to a sink in a batch.
> 1) is a reasonable choice. But I don't think deriving from
> BatchingDecorator be a good idea. For my case, there is little in common
> from the class. Why not borrowing the idea and implementing yours version?
> The biggest piece of code that I was looking to use was the TimeoutThread.
 I ended up just using that as is.

> For 2), I don't like the idea to call append() in a separate thread. (if I
> understand your solution correctly.)
> Can you elaborate? Is there a technical reason why not? AFAICT, this is
what RollSink does (although it does block other threads while it's
closing/reopening).  In our case, we're batching up a lot of data over a
minute, so I'd prefer not to make a single call of append() block to forward
all the data along.  As I mentioned, I'd be interested in doing this with
multiple threads, even.


View raw message