flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chalcy Raja <Chalcy.R...@careerbuilder.com>
Subject RE: Scale of a flume collector
Date Wed, 14 Mar 2012 00:35:31 GMT
Hi Kim,

How are you handling realtime collection?  I am okay with a few minutes delay, but the collector
is collecting only once every hour.  What configuration I should use to set the collector
to collect every minute?

Have you set up the flume yet?  How are you handling agent dying?

From: Kim, Jongkook  [jongkook.kim@citi.com]
Sent: Tuesday, February 14, 2012 2:25 PM
To: 'flume-user@incubator.apache.org'
Subject: Scale of a flume collector

Hi all.

I'm in the middle of hardware provisioning for flume-hbase-hadoop solution.
The plan is that flume agents collect and pass log data to collectors and the collectors write
data into hbase using sink.
The question is a flume collector's scale.

Flume agents:250
Data receiving ratio: 5.78MB/second
Data writing ratio: 17.9MB/second
Number of data nodes: 12

This system will be used to provide real-time use case, so there shouldn't be delay.
How many collectors required to handle this request?

Thanks in advance,

View raw message