flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Blade Liu <hafzc...@gmail.com>
Subject Re: Performance of Flume in production systems
Date Fri, 26 Sep 2014 03:53:07 GMT
Hi Asim and Jeff,

Thanks for your nice suggestions.  I found two excellent articles, one on
performance test and the other on deployment design/optimization in
production system.

Flume NG Performance Measurements

Log collection system architecture and design of Meituan.com(Chinese
version, strongly recommended that you can use Chrome translator for

I guess building a stable and efficient collection system is  challenging
and also fun.


2014-09-26 3:15 GMT+08:00 Jeff Lord <jlord@cloudera.com>:

> Whether or not flume can handle 20k eps will depend on several factors.
> The main ones being:
> 1. What is the avg size of event
> 2. What source will you be using
> With that said I have seen a single flume agent handle well over 20k eps
> using the multiport syslog source.
> Here is a link to a presentation given by Arvind Prabhakar on planning a
> flume deployment.
> http://goo.gl/FsfmmC
> -Jeff
> On Wed, Sep 24, 2014 at 10:53 PM, Blade Liu <hafzcdcn@gmail.com> wrote:
>> Hi,
>> I'm going to deploy Flume in production systems, but a little worried
>> about its performance in real-world environment. Could anyone tell me about
>> Flume's actual performance in production environment? say, if Flume can
>> deal with 20,000 events per second from a single source(and what about
>> 100-200 sources with one final HDFS sink).
>> In addition, to reach good performance of tens of thousands of events per
>> second, how many servers(agents) should be used?  More agents(and more
>> tiers), better performance?
>> Thanks very much for your suggestions.
>> Cheers,
>> Blade

View raw message