Joe.

 > Can you elaborate?

Sorry. Let me correct myself that calling super.append() in a separate 
thread should be fine. I was a little bit confused by your 2nd question 
``It's possible for our sink decorator to generate more events than it 
receives'', so made a wrong statement.

 > are there dangers in using a threadpool to call append() from a 
decorator to forward events to the collector

I'll be working on a new hbase sink decorator which use multi-threaded 
to improve the hbase sink write throughput. Your use case is quite 
similar. Looking forward to hearing more from you.

Thanks,
Mingjie


On 08/16/2011 01:05 PM, Joe Crobak wrote:
> On Tue, Aug 16, 2011 at 2:53 PM, Mingjie Lai <mjlai09@gmail.com
> <mailto:mjlai09@gmail.com>> wrote:
>
>     Joe.
>
>     I have a similar case that I need to accumulate some data at a
>     decorator and write to a sink in a batch.
>
>     1) is a reasonable choice. But I don't think deriving from
>     BatchingDecorator be a good idea. For my case, there is little in
>     common from the class. Why not borrowing the idea and implementing
>     yours version?
>
> The biggest piece of code that I was looking to use was the
> TimeoutThread.  I ended up just using that as is.
>
>     For 2), I don't like the idea to call append() in a separate thread.
>     (if I understand your solution correctly.)
>
> Can you elaborate? Is there a technical reason why not? AFAICT, this is
> what RollSink does (although it does block other threads while it's
> closing/reopening).  In our case, we're batching up a lot of data over a
> minute, so I'd prefer not to make a single call of append() block to
> forward all the data along.  As I mentioned, I'd be interested in doing
> this with multiple threads, even.
>
> Thanks,
> Joe