flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: multiple flume clients and memory
Date Wed, 25 Mar 2015 22:42:10 GMT
How much memory are you talking about? The RPC client will hold on to the batch of events you
sent, plus some additional threading overhead. Under the hood, it uses a Netty client which
should not really have a big memory footprint.

Thanks, Hari

On Wed, Mar 25, 2015 at 3:27 PM, Matt Fair <matt.fair@gmail.com> wrote:

> I have an application that launches a bunch of processes (40+) on the same
> machine, each one connects to flume using the default flume RPCClient.  I
> however have noticed that each RPCClient takes up a decent amount of
> memory, and when you create as many clients like I am, it adds up to a lot
> of memory.  One thought I had to alleviate having to create all of the
> clients was to create only a single RPCClient and then have my other
> processes connect to it via a socket, but that seems a little redundant
> since that is what the RPCClient is suppose to do anyways.  Have others
> found themselves in this same situation?  Is there a way to handle memory
> more efficiently or is there another RPCClient implementation that doesn't
> take up as much memory?
> Thanks,
> Matt
View raw message