flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allan Feid <allanf...@gmail.com>
Subject multi-threaded elasticsearch sink
Date Wed, 19 Jun 2013 15:00:42 GMT
I'm not that great at Java at the moment, but it appears that the single
threaded nature of the elasticsearch sink has trouble keeping up with ~5k
events/second at 2k batch size. It looks like the HDFS sink has the ability
to run multiple threads that write to the HDFS. I can get some performance
increase by adding multiple ElasticSearch sinks to simulate parallelism,
but it would be great for the sink itself to support multiple threads.

Is there a sink example that should be used as a guide towards getting the
same features in the elasticsearch sink?

Thanks,
Allan

Mime
View raw message