flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: how to print the channel capacity
Date Wed, 15 May 2013 08:49:10 GMT
here is one example for the capacity defining flow
https://cwiki.apache.org/FLUME/flume-ng-performance-measurements.html


On Wed, May 15, 2013 at 2:16 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:

> sorry pressed enter too soon
>
> as for your question: how many events a flume agent can hold?
> sorry but I don't think there is any direct answer to that.... .I may be
> very well wrong there as I am myself pretty new with flume
>
> there was a JIRA for the capacity of file channels FLUME-1571
>
>
> On Wed, May 15, 2013 at 1:50 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>
>> for maximum performance on your data flow two things which will matter
>> most are: the channel and the transaction batch size.
>> when you say losing data, are you using memory channel? or file channel?
>>
>> Flume can batch events. The batch size is the maximum number of events
>> that a sink or client will attempt to take from a channel in a single
>> transaction.
>>
>>  What is the channel type
>> do you have a slow sink so the # events written out are less than # event
>> incoming to channels so over time it piles up
>>
>> others may point out more things.
>> Also your flume conf and if you are seeing any errors on flume then that
>> will help people to find out the problem
>>
>>
>> On Wed, May 15, 2013 at 11:07 AM, liuyongbo <liuyongbo@baidu.com> wrote:
>>
>>>  Hi:****
>>>
>>>          I’m using flume to pass log data to mongodb, but I find that
>>> some data lose when the pressure is in high level, so I want to know the
>>> max request that flume can hold and need to print the capacity.but I can
>>> not find the proper way to do this instead of change the source code. Any
>>> ideas?****
>>>
>>>          thanks****
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> Nitin Pawar
>



-- 
Nitin Pawar

Mime
View raw message