flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sverre Bakke <sverre.ba...@gmail.com>
Subject Re: Channel locks
Date Wed, 20 May 2015 16:46:52 GMT
Hi,

Thanks for the response. Does that mean that transaction.begin() does not
acquire a lock that is kept until transaction.commit()?

If not, how can a sink slow down the source? I would think that the channel
would fill up and throw channelfullexception instead rather than slow down
the source?

Regards,
Sverre
On May 19, 2015 17:31, "Hari Shreedharan" <hshreedharan@cloudera.com> wrote:

> Each sink does not hold a lock on the channel. Channels can handle
> multiple sinks reading from them and in fact, that is encouraged.
>
> On Tuesday, May 19, 2015, Sverre Bakke <sverre.bakke@gmail.com> wrote:
>
>> Hi,
>>
>> When using a single source, channel and sink setup with Syslog, memory
>> channel and KafkaSink (from
>> https://github.com/thilinamb/flume-ng-kafka-sink) I am experiencing
>> performance issues.
>>
>> After looking at the source code of the sink, it seems that the sink
>> will begin a transaction, then do everything (e.g. compress, send over
>> the network, wait for ack), and eventually close/commit the
>> transaction after everything is done.
>>
>> Is there any way to increase the performance of this setup?
>>
>> Earlier I have seen people proposing adding more sinks to get higher
>> throughput, but if every sink holds a lock on the channel until done
>> with processing, then I would assume that this only causes additional
>> performance issues? Are my understanding correct on how channel
>> locking works?
>>
>> Regards,
>> Sverre
>>
>
>
> --
>
> Thanks,
> Hari
>
>

Mime
View raw message