flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: The size of data folder continue grow
Date Tue, 26 Nov 2013 14:10:08 GMT
On Tue, Nov 26, 2013 at 7:41 AM, GuoWei <wei.guo@wbkit.com> wrote:
> Dear all,
> I use flume and custom base sink to put data to HBase. In flume, I use file channel.
> the file channel data put in the following folder.
> razor.channels.c_error.dataDirs = /var/lib/flume-ng/data/error
> But I see the channel data folder size continue grow until to 2gb.
> Does flume remove the data in channel data folder after flush to HBase? Or still store
in data folder after flush to HBase ?

By default the file channel will keep any log which has data in the
channel in addition to 2 logs for safety purposes.  If you have a
small channel which is empty and simply don't want to store that data,
then you can turn down the max file size for the file channel.

> Or the speed sink to HBase is too slower than source ?

You'd need look at the metrics to decide this.

> And how to speed up the sink to HBase ?

I suggest you first decide if that is actually a problem, but a common
issue with hbase is called "hot spotting" due to bad key design.

View raw message