Hi Jeff,

Thank you for your answer.
Actually i don't think the issue is related to the syslog source.

I didn't change anything to the source configuration, the only difference between the 2 tests is the channel.
I don't understand why the disk load is so high whereas the performance is so low (300events/sec)

And using multiple ports forces to load balances on our sources servers over these ports.
Alternatively we can use iptables rules, i'll try that and let you know.

regards,
-- 
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com


On 03/04/15 16:20, Jeff Lord wrote:
Have you tried using the multiport syslog source?
We typically experience better perf with that one.

On Wednesday, March 4, 2015, Smaine Kahlouch <smaine.kahlouch@smartjog.com> wrote:
Hi all,

I'm currently doing benchmarks on flume.
We're planning to use flume with syslogtcp as source and filechannel in order to have avoid data loss.

The performances are quiet good when a memorychannel is used :
~100 000events/sec (event size = 600bytes)

But as soon as i switch to filechannel the performances drop drammatically:
~300events/sec

Despite this poor result, the behaviour is really strange because i have a heavy disk usage (all the disks), near 100%.

I use a tool provided by syslog-ng in order to generate syslog logs : loggen

ex : loggen -i -I 3000000 --size 600 --active-connections 200 myflumehost 20515


Flume version : 1.5.2
Operating System : Centos 6

Please find my flume configuration enclosed. The filechannel is spread over 5 disks in order to improve performance.


Could you please help me to configure properly syslogtcp source with filechannel ?

Regards,
-- 
Smaine Kahlouch - Engineer, Research & Engineering
Arkena | T: +33 1 5868 6196
27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France
arkena.com