flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Why used space of flie channel buffer directory increase?
Date Tue, 19 Mar 2013 18:32:23 GMT
It is possible for the directory size to increase even if no writes are going in to the channel.
If the channel size is non-zero and the sink is still writing events to HDFS, the takes get
written to disk as well (so we know what events in the files were removed when the channel/agent
restarts). Eventually the channel will clean up the files which have all events taken (though
it will keep at least 2 files per data directory, just to be safe). 

-- 
Hari Shreedharan


On Tuesday, March 19, 2013 at 10:32 AM, Alexander Alten-Lorenz wrote:

> Hey,
> 
> what says debug? Do you can gather logs and attach them?
> 
> - Alex
> 
> On Mar 19, 2013, at 5:27 PM, "Kenison, Matt" <Matt.Kenison@disney.com (mailto:Matt.Kenison@disney.com)>
wrote:
> 
> > Check the JMX counter first, to make sure you really are not sending new events.
If not, is it your checkpoint directory or data directory that is increasing in size?
> > 
> > 
> > From: Zhiwen Sun <pensz01@gmail.com (mailto:pensz01@gmail.com)>
> > Reply-To: "user@flume.apache.org (mailto:user@flume.apache.org)" <user@flume.apache.org
(mailto:user@flume.apache.org)>
> > Date: Tue, 19 Mar 2013 01:19:19 -0700
> > To: "user@flume.apache.org (mailto:user@flume.apache.org)" <user@flume.apache.org
(mailto:user@flume.apache.org)>
> > Subject: Why used space of flie channel buffer directory increase?
> > 
> > hi all:
> > 
> > I test flume-ng in my local machine. The data flow is :
> > 
> > tail -F file | nc 127.0.0.01 4444 > flume agent > hdfs
> > 
> > My configuration file is here :
> > 
> > > a1.sources = r1
> > > a1.channels = c2
> > > 
> > > a1.sources.r1.type = netcat
> > > a1.sources.r1.bind = 192.168.201.197
> > > a1.sources.r1.port = 44444
> > > a1.sources.r1.max-line-length = 1000000
> > > 
> > > a1.sinks.k1.type = logger
> > > 
> > > a1.channels.c1.type = memory
> > > a1.channels.c1.capacity = 10000
> > > a1.channels.c1.transactionCapacity = 10000
> > > 
> > > a1.channels.c2.type = file
> > > a1.sources.r1.channels = c2
> > > 
> > > a1.sources.r1.interceptors = i1
> > > a1.sources.r1.interceptors.i1.type = timestamp
> > > 
> > > a1.sinks = k2
> > > a1.sinks.k2.type = hdfs
> > > a1.sinks.k2.channel = c2 
> > > a1.sinks.k2.hdfs.path = hdfs://127.0.0.1:9000/flume/events/%Y-%m-%d
> > > a1.sinks.k2.hdfs.writeFormat = Text
> > > a1.sinks.k2.hdfs.rollInterval = 10
> > > a1.sinks.k2.hdfs.rollSize = 10000000
> > > a1.sinks.k2.hdfs.rollCount = 0
> > > 
> > > a1.sinks.k2.hdfs.filePrefix = app 
> > > a1.sinks.k2.hdfs.fileType = DataStream
> > > 
> > 
> > 
> > 
> > 
> > it seems that events were collected correctly.
> > 
> > But there is a problem boring me: Used space of file channel (~/.flume) has always
increased, even there is no new event.
> > 
> > Is my configuration wrong or other problem? 
> > 
> > thanks.
> > 
> > 
> > Best regards.
> > 
> > Zhiwen Sun 
> 
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
> 
> 



Mime
View raw message