flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeong-shik Jang <jsj...@gruter.com>
Subject Re: FLume OG Choke Limit Not Working
Date Thu, 14 Mar 2013 14:15:49 GMT
Hi James,

According to source codes, depending on event body size, choke limit may 
not work accurately.
One scenario I can imagine for this case is:

  * Given choke limit 10 which means 1 KB bucket size and event body
    size 100KB,
  * when append() is called in choke deco, it may wait for 2 times (200
    milliseconds) as 100KB is larger than bucket size (1 KB)
  * but at third time it just ships the event to Sink no matter how
    large the event body is,
  * this can happen about 3 times within a seconds (every 100 * 3
    milliseconds).
  * This may result in a spike of more than 300KB (100KB * 3)/sec;
    2.4Mbit/sec.

So I am curious about the size of your event body.

-JS

On 3/13/13 7:35 PM, James Stewart wrote:
>
> Hello again,
>
> Further to my previous problem with the Choke decorator under Flume 
> OG, I've tried separating out my 'aggregator' into 3 logical nodes as 
> follows...
>
> OLD:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config aggregator.mydomain.com 'collectorSource(35853)' 
> 'batch(100, 1000) gzip choke("mychoke") 
> agentBESink("collector.mydomain.com", 35853)'
>
> NEW:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config batcher 'collectorSource(35853)' 'batch(100, 1000) 
> agentBESink("aggregator.mydomain.com", 35854)'
>
> exec config zipper 'collectorSource(35854)' 'gzip 
> agentBESink("aggregator.mydomain.com ", 35855)'
>
> exec config choker 'collectorSource(35855)' 'choke("mychoke") 
> agentBESink("collector.mydomain.com ", 35853)'
>
> Looking at the stats, it seems that the 'choker' decorator is having 
> no effect on the data passing through it. Even when I set 'mychoke' to 
> 10, (which equates to 80Kbit/sec) I still see regular spikes above 
> 1.5Mbit/sec.
>
> The Flume User Guide says:
>
> /"The choke decorator works as follows: when append() is called on the 
> sink to which the choke is attached, the append() call works normally 
> if the amount of data transferred (during a small duration of time) is 
> within the limit assigned to the choke-id corresponding to the choke. 
> If the limit has been exceeded, then append() is blocked for a small 
> duration of time."/
>
> Our traffic is quite bursty -- could this be a problem for the Choke 
> decorator?
>
> Any help is much appreciated as I've hit a bit of a wall.
>
> Cheers,
>
> James
>
> *From:*James Stewart [mailto:James.Stewart@Optiver.com.au]
> *Sent:* Tuesday, 12 March 2013 10:54 AM
> *To:* user@flume.apache.org
> *Subject:* FLume OG Choke Limit Not Working
>
> Hello all,
>
> I'm using flume OG (unable to upgrade to NG at this stage) and I am 
> having trouble with the choke decorator.
>
> I am aggregating the data flows from several logical nodes at a single 
> 'aggregator' logical node. The data flows should be batched, zipped, 
> choked and then sent on to another 'collector' logical node. I am 
> using the following config to achieve this:
>
> exec setChokeLimit aggregator.mydomain.com mychoke 150
>
> exec config aggregator.mydomain.com 'collectorSource(35853)' 
> 'batch(100, 1000) gzip choke("mychoke") 
> agentBESink("collector.mydomain.com", 35853)'
>
> The choke decorator should limit transfer to 150KB/sec, which equates 
> to 1.2Mbit/sec. However I am regularly recording Flume traffic spikes 
> of 5Mbit/sec and more.
>
> Can anybody suggest what I might be doing wrong? Is it ok to chain the 
> batch, gzip and choke decorators like this, or should they each be in 
> a separate logical node?
>
> Thanks,
>
> James
>
> ------------------------------------------------------------------------
>
> Information contained in this communication (including any 
> attachments) is confidential and may be privileged or subject to 
> copyright. If you have received this communication in error you are 
> not authorised to use the information in any way and Optiver requests 
> that you notify the sender by return email, destroy all copies and 
> delete the information from your system. Optiver does not represent, 
> warrant or guarantee that this communication is free from computer 
> viruses or other defects or that the integrity of this communication 
> has been maintained. Any views expressed in this communication are 
> those of the individual sender. Optiver does not accept liability for 
> any loss or damage caused directly or indirectly by this communication 
> or its use.
>
> Please consider the environment before printing this email.
>


-- 
Jeong-shik Jang / jsjang@gruter.com
Gruter, Inc., R&D Team Leader
www.gruter.com
Enjoy Connecting


Mime
View raw message