flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Needham, Guy" <Guy.Need...@virginmedia.co.uk>
Subject RE: Channel locks
Date Tue, 19 May 2015 16:18:52 GMT
With multiple sinks reading from one channel, will each sink read each event, or will the events
be distributed between the sinks?

Guy Needham | Data Discovery
Virgin Media   | Technology and Transformation | Data
Bartley Wood Business Park, Hook, Hampshire RG27 9UP
D 01256 75 3362
I welcome VSRE emails. Learn more at http://vsre.info/
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: 19 May 2015 16:30
To: user@flume.apache.org
Subject: Re: Channel locks

Each sink does not hold a lock on the channel. Channels can handle multiple sinks reading
from them and in fact, that is encouraged.

On Tuesday, May 19, 2015, Sverre Bakke <sverre.bakke@gmail.com<mailto:sverre.bakke@gmail.com>>

When using a single source, channel and sink setup with Syslog, memory
channel and KafkaSink (from
https://github.com/thilinamb/flume-ng-kafka-sink) I am experiencing
performance issues.

After looking at the source code of the sink, it seems that the sink
will begin a transaction, then do everything (e.g. compress, send over
the network, wait for ack), and eventually close/commit the
transaction after everything is done.

Is there any way to increase the performance of this setup?

Earlier I have seen people proposing adding more sinks to get higher
throughput, but if every sink holds a lock on the channel until done
with processing, then I would assume that this only causes additional
performance issues? Are my understanding correct on how channel
locking works?




Save Paper - Do you really need to print this e-mail?

Visit www.virginmedia.com for more information, and more fun.

This email and any attachments are or may be confidential and legally privileged
and are sent solely for the attention of the addressee(s). If you have received this
email in error, please delete it from your system: its use, disclosure or copying is
unauthorised. Statements and opinions expressed in this email may not represent
those of Virgin Media. Any representations or commitments in this email are
subject to contract. 

Registered office: Media House, Bartley Wood Business Park, Hook, Hampshire, RG27 9UP
Registered in England and Wales with number 2591237
View raw message