flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 손희진 <heejin.s...@samsung.com>
Subject java.lang.OutOfMemoryError: Direct buffer memory
Date Mon, 02 Mar 2015 00:27:40 GMT
<HTML><HEAD><TITLE>Samsung Enterprise Portal mySingle</TITLE>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META content="text/html; charset=euc-kr" http-equiv=Content-Type>
<STYLE id=mysingle_style>P {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN-TOP: 5px
}
TD {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN-TOP: 5px
}
LI {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN-TOP: 5px
}
BODY {
	FONT-SIZE: 9pt; FONT-FAMILY: 굴림체, arial; MARGIN: 10px; LINE-HEIGHT: 1.4
}
</STYLE>

<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY>
<META name=GENERATOR content=ActiveSquare>
<META name=GENERATOR content=ActiveSquare><X-BODY>
<META name=GENERATOR content=ActiveSquare><X-BODY>
<P>Hi there,</P>
<P>I'm testing flume with thrift source, file channel, and HDFS sink. </P>
<P>Also there is a flume client which sends events using thrift, and the size of each
event is up to 30 Mbytes.</P>
<P>It works fine for a short period, but after a few minutes, following error occurs
at org.apache.flume.channel.ChannelProcessor.</P>
<P>
<TABLE class=DataTable style="POSITION: relative">
<TBODY data-bind="foreach: events">
<TR class=highlight data-bind="attr: {'data-offset': offset}, css: { highlight: highlighted
}" data-offset="35896438">
<TD class=nonMessageColumn data-bind="text: formattedTime">8:15:36.450 PM</TD>
<TD class="nonMessageColumn error" data-bind="text: loglevel, css: {warn: loglevel ===
'WARN', error: loglevel === 'ERROR', fatal: loglevel === 'FATAL'}">ERROR</TD>
<TD class=nonMessageColumn data-bind="text: source">org.apache.flume.channel.ChannelProcessor</TD>
<TD><PRE class=message data-bind="text: message">Error while writing to required
channel: FileChannel c1 { dataDirs: [/data/2/flumechannel/data] }
java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
	at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
	at sun.nio.ch.IOUtil.write(IOUtil.java:58)
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
	at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)
	at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)
	at org.apache.flume.channel.file.Log.put(Log.java:633)
	at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
	at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
	at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
	at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
	at org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)
	at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)
	at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
	at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
	at org.apache.thrift.server.Invocation.run(Invocation.java:18)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
</PRE></TD></TR></TBODY></TABLE></P>
<P>I increased direct memory size upto 2G byte, but it didn't work.</P>
<P>Here's my flume configuration.</P>
<P>&nbsp;</P>
<P>#source</P>
<P>tier1.sources.s1.type&nbsp;&nbsp;&nbsp;&nbsp; = thrift<BR>tier1.sources.s1.bind&nbsp;&nbsp;&nbsp;&nbsp;
= 0.0.0.0<BR>tier1.sources.s1.port&nbsp;&nbsp;&nbsp;&nbsp; = 30010<BR>tier1.sources.s1.channels
= c0 c1 memdefault<BR>tier1.sources.s1.selector.type = multiplexing<BR>tier1.sources.s1.selector.header
= category<BR>tier1.sources.s1.selector.mapping.Log4j =&nbsp;c0 <BR>tier1.sources.s1.selector.mapping.Data
= c1<BR>tier1.sources.s1.selector.default = memDefault</P>
<P>&nbsp;</P>
<P>#channel</P>
<P>tier1.channels.c1.type&nbsp;&nbsp; = memory<BR>tier1.channels.c1.checkpointDir=/data/2/flumechannel/checkpoint<BR>tier1.channels.c1.dataDirs=/data/2/flumechannel/data<BR>tier1.channels.c1.transactionCapacity
= 1<BR>tier1.channels.c1.maxFileSize = 500000000</P>
<P>&nbsp;</P>
<P>#sink</P>
<P>tier1.sinks.k1.type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
= hdfs<BR>tier1.sinks.k1.channel&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
c1<BR>tier1.sinks.k1.hdfs.path = /user/soul<BR>tier1.sinks.k1.hdfs.round = false<BR>tier1.sinks.k1.hdfs.fileType
= DataStream<BR>tier1.sinks.k1.hdfs.rollCount = 1<BR>tier1.sinks.k1.hdfs.batchSize
= 1<BR>tier1.sinks.k1.hdfs.retryInterval = 10<BR>tier1.sinks.k1.hdfs.proxyUser
= soul<BR>tier1.sinks.k1.hdfs.maxOpenFiles = 10<BR>tier1.sinks.k1.hdfs.idleTimeOut
= 900</P>
<P>&nbsp;</P>
<P>and java config. option: -Xmx2g -XX:MaxDirectMemorySize=2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
</P>
<P>&nbsp;</P>
<P>When I use memory channel instead of file channel, it works great.</P>
<P>I can't understand this phenomena.</P>
<P>The only clue that I have is that the exception&nbsp;always occurs after "org.apache.flume.channel.file.Log:
Roll end"</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>Hope your comments.</P>
<P>Thank you.</P>
<P>&nbsp;</P>
<P>&nbsp;</P></X-BODY></X-BODY></BODY></HTML><img
src='http://ext.samsung.net/mailcheck/SeenTimeChecker?do=61dd5ae2e408a716506e7d7427027ed9e7c83b76671f074a279babd5c747fd394603607fb4f10f2b93a068f28056b538f4bcdeced46ed5ee08cece8541bc14eacf878f9a26ce15a0'
border=0 width=0 height=0 style='display:none'>
Mime
View raw message