flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Fair <matt.f...@gmail.com>
Subject Re: multiple flume clients and memory
Date Thu, 26 Mar 2015 03:03:14 GMT
The machine that I have seen it both on my machine with 16 GB and 60 GB of
memory, when running about 40 clients and ~4k clients respectively using up
100% of memory.  If I run without the flume client I have no memory
problems, but when I insatiate a flume RPCClient, then I run into memory
problems.

Thanks,
Matt

On Wed, Mar 25, 2015 at 6:42 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:

> How much memory are you talking about? The RPC client will hold on to the
> batch of events you sent, plus some additional threading overhead. Under
> the hood, it uses a Netty client which should not really have a big memory
> footprint.
>
> Thanks,
> Hari
>
>
> On Wed, Mar 25, 2015 at 3:27 PM, Matt Fair <matt.fair@gmail.com> wrote:
>
>> I have an application that launches a bunch of processes (40+) on the
>> same machine, each one connects to flume using the default flume
>> RPCClient.  I however have noticed that each RPCClient takes up a decent
>> amount of memory, and when you create as many clients like I am, it adds up
>> to a lot of memory.  One thought I had to alleviate having to create all of
>> the clients was to create only a single RPCClient and then have my other
>> processes connect to it via a socket, but that seems a little redundant
>> since that is what the RPCClient is suppose to do anyways.  Have others
>> found themselves in this same situation?  Is there a way to handle memory
>> more efficiently or is there another RPCClient implementation that doesn't
>> take up as much memory?
>>
>> Thanks,
>> Matt
>>
>
>

Mime
View raw message