flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Smaine Kahlouch <smaine.kahlo...@smartjog.com>
Subject Syslog TCP performances issue with filechannel
Date Wed, 04 Mar 2015 14:28:26 GMT
Hi all,

I'm currently doing benchmarks on flume.
We're planning to use flume with syslogtcp as source and filechannel in 
order to have avoid data loss.

The performances are quiet good when a memorychannel is used :
~*100 000events/sec* (event size = 600bytes)

But as soon as i switch to filechannel the performances drop drammatically:
~*300events/sec*

Despite this poor result, the behaviour is really strange because i have 
a heavy disk usage (all the disks), near 100%.

I use a tool provided by syslog-ng in order to generate syslog logs : 
loggen 
<http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/loggen.1.html>

ex : loggen -i -I 3000000 --size 600 --active-connections 200 
myflumehost 20515


Flume version : 1.5.2
Operating System : Centos 6

Please find my flume configuration enclosed. The filechannel is spread 
over 5 disks in order to improve performance.


Could you please help me to configure properly syslogtcp source with 
filechannel ?

Regards,

-- 
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com


Mime
View raw message