flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Camp, Roy" <rc...@ebay.com>
Subject RE: File Channel Capacity issue
Date Sun, 25 Nov 2012 23:08:55 GMT
Ah okay, thanks!  I think I might just start a second instance of flume with an updated config
and then flip the application layer over.

Is there a maximum value that the FIleChannel capacity will accept?  Would setting this value
really high result in any performance impact?

Each of my collectors processes about 10MM events per day, so in the event the sink fails
during a weekend, I don’t want it to fill up and lose events.  I’m thinking of setting
the capacity to 100MM.



From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Friday, November 23, 2012 1:14 PM
To: user@flume.apache.org
Subject: Re: File Channel Capacity issue


The FileChannel actually uses a fixed size checkpoint file -- so it is not possible to set
it to unlimited size (the checkpoint file is mmap-ed to a fixed size buffer). To change the
capacity of the channel, the easiest way off the top of my head is:

* Shutdown the agent.
* Delete all files in the file channel's checkpoint directory. (not the data directories.
Also you might want to move them out, rather than delete to be safe)
* Change your configuration to increase the capacity of the channel.
* Restart the agent - this will cause full replay, so the agent might take sometime to start
up if there are a lot of events in the channel (to avoid this - shutdown the source before
shutting the agent down - so the sink can drain out the channel completely, wait for about
1-2 mins after the channel is empty so that the data files get deleted (this happens only
immediately after a checkpoint - you can verify this by making sure each data dir has only
2 files each), since all events have been sent out - so during restart the channel will be
quite empty, with very little to replay).

Hope this helps.


Hari Shreedharan

On Friday, November 23, 2012 at 11:58 AM, Camp, Roy wrote:

I am having issue with a slow sink (not flume related) but it is causing my file channel to
overflow.  When trying to increase it I get the following error but can’t seem to find the
setting it refers to in any of the documentation.  Additionally, is there a way to set the
file channel to unlimited?  It doesn’t seem to like 0.

java.lang.IllegalStateException: Channel closed [channel=collectorfile]. Due to java.lang.IllegalStateException:
Configured capacity is 100000000 but the  checkpoint file capacity is 1000000. See FileChannel
documentation on how to change a channels capacity.



View raw message