flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sverre Bakke <sverre.ba...@gmail.com>
Subject Channel locks
Date Tue, 19 May 2015 12:35:21 GMT

When using a single source, channel and sink setup with Syslog, memory
channel and KafkaSink (from
https://github.com/thilinamb/flume-ng-kafka-sink) I am experiencing
performance issues.

After looking at the source code of the sink, it seems that the sink
will begin a transaction, then do everything (e.g. compress, send over
the network, wait for ack), and eventually close/commit the
transaction after everything is done.

Is there any way to increase the performance of this setup?

Earlier I have seen people proposing adding more sinks to get higher
throughput, but if every sink holds a lock on the channel until done
with processing, then I would assume that this only causes additional
performance issues? Are my understanding correct on how channel
locking works?


View raw message