flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HEEJIN SOHN <heejin.s...@samsung.com>
Subject Re: Re: java.lang.OutOfMemoryError: Direct buffer memory
Date Mon, 02 Mar 2015 04:11:56 GMT
<HTML><HEAD><TITLE>Samsung Enterprise Portal mySingle</TITLE>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<STYLE id=mysingle_style>P {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
TD {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
LI {
	MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
BODY {
	FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN: 10px; LINE-HEIGHT: 1.4
}
</STYLE>

<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY>
<META name=GENERATOR content=ActiveSquare>
<P>Wan Yi //</P>
<P>Tried as you've suggested (maxFileSize = 1M), but only met the same exception.</P>
<P>&nbsp;</P>
<P>Hari Shreedharan//</P>
<P>All flume options are set using Cloudera manager. (CDH-5.2.0-1)</P>
<P>Also, possible event size is under 100MB.</P>
<P>&nbsp;</P>
<P>Here is the log pattern.</P>
<P>8:15:26.367 PM &nbsp;INFO &nbsp;org.apache.flume.sink.hdfs.BucketWriter Creating
/user/25068525641.jpg.27.1425035726356.tmp<BR>8:15:36.407 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.Log
Roll start /data/2/flumechannel/data<BR>8:15:36.407 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.LogFile
Opened /data/2/flumechannel/data/log-4<BR>8:15:36.447 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.LogFile
Closing /data/2/flumechannel/data/log-3<BR>8:15:36.447 PM &nbsp;INFO &nbsp;org.apache.flume.channel.file.Log
Roll end<BR>8:15:36.450 PM &nbsp;ERROR &nbsp;org.apache.flume.channel.ChannelProcessor
Error while writing to required channel: FileChannel c1 { dataDirs: [/data/2/flumechannel/data]
}<BR>java.lang.OutOfMemoryError: Direct buffer memory<BR>&nbsp;at java.nio.Bits.reserveMemory(Bits.java:658)<BR>&nbsp;at
java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)<BR>&nbsp;at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)<BR>&nbsp;at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)<BR>&nbsp;at
sun.nio.ch.IOUtil.write(IOUtil.java:58)<BR>&nbsp;at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)<BR>&nbsp;at org.apache.flume.channel.file.Log.put(Log.java:633)<BR>&nbsp;at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)<BR>&nbsp;at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)<BR>&nbsp;at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)<BR>&nbsp;at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)<BR>&nbsp;at
org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)<BR>&nbsp;at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)<BR>&nbsp;at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)<BR>&nbsp;at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)<BR>&nbsp;at
org.apache.thrift.server.Invocation.run(Invocation.java:18)<BR>&nbsp;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<BR>&nbsp;at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<BR>&nbsp;at
java.lang.Thread.run(Thread.java:745)</P>
<P><BR>8:15:36.451 PM &nbsp;ERROR &nbsp;org.apache.thrift.ProcessFunction
Internal error processing append java.lang.OutOfMemoryError: Direct buffer memory<BR>&nbsp;at
java.nio.Bits.reserveMemory(Bits.java:658)<BR>&nbsp;at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)<BR>&nbsp;at
java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)<BR>&nbsp;at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)<BR>&nbsp;at
sun.nio.ch.IOUtil.write(IOUtil.java:58)<BR>&nbsp;at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)<BR>&nbsp;at
org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)<BR>&nbsp;at org.apache.flume.channel.file.Log.put(Log.java:633)<BR>&nbsp;at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)<BR>&nbsp;at
org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)<BR>&nbsp;at
org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)<BR>&nbsp;at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)<BR>&nbsp;at
org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)<BR>&nbsp;at
org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)<BR>&nbsp;at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)<BR>&nbsp;at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)<BR>&nbsp;at
org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)<BR>&nbsp;at
org.apache.thrift.server.Invocation.run(Invocation.java:18)<BR>&nbsp;at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<BR>&nbsp;at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<BR>&nbsp;at
java.lang.Thread.run(Thread.java:745)<BR></P>
<P>&nbsp;</P>
<P>Thank you very much.</P>
<P>&nbsp;</P>
<P>Heejin Sohn.</P>
<P>&nbsp;</P>
<P>&nbsp;</P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Hari Shreedharan&lt;hshreedharan@cloudera.com&gt;</P>
<P><B>Date</B> : 2015-03-02 12:22 (GMT+09:00)</P>
<P><B>Title</B> : Re: java.lang.OutOfMemoryError: Direct buffer memory</P>
<P>&nbsp;</P>How are you passing the JVM options in. Is the event being written
more than 2g?&nbsp;<SPAN></SPAN><BR><BR>On Sunday, March 1, 2015,
Wan Yi(武汉_技术部_搜索与精准化_万毅) &lt;<A href="mailto:wanyi@yhd.com">wanyi@yhd.com</A>&gt;
wrote:<BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT:
#ccc 1px solid"><U></U><U></U>
<DIV lang=ZH-CN style="MARGIN: 7.5pt" link="blue" vlink="purple">
<DIV>
<P class=MsoNormal><SPAN lang=EN-US>tier1.channels.c1.maxFileSize = 500000000</SPAN><SPAN
lang=EN-US><U></U><U></U></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P class=MsoNormal><SPAN lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Calibri','sans-serif';
COLOR: #1f497d">try to change the maxFileSize to a small value, may be 1M or 10M<U></U><U></U></SPAN></P>
<P class=MsoNormal><SPAN lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Calibri','sans-serif';
COLOR: #1f497d"><U></U><U></U></SPAN>&nbsp;</P>
<DIV>
<P class=MsoNormal style="TEXT-ALIGN: justify; MARGIN: 0cm 0cm 0pt; TEXT-JUSTIFY: inter-ideograph"><B><SPAN
lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Verdana','sans-serif'; COLOR: #1f497d"><U></U><U></U></SPAN></B>&nbsp;</P></DIV>
<P class=MsoNormal><SPAN lang=EN-US style="FONT-SIZE: 10.5pt; FONT-FAMILY: 'Calibri','sans-serif';
COLOR: #1f497d">Wayne Wan<U></U><U></U></SPAN></P>
<DIV>
<DIV style="BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium
none; PADDING-BOTTOM: 0cm; PADDING-TOP: 3pt; PADDING-LEFT: 0cm; BORDER-LEFT: medium none;
PADDING-RIGHT: 0cm">
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><B><SPAN style="FONT-SIZE: 10pt;
FONT-FAMILY: 宋体">发件人<SPAN lang=EN-US>:</SPAN></SPAN></B><SPAN
lang=EN-US style="FONT-SIZE: 10pt; FONT-FAMILY: 宋体"> </SPAN><SPAN style="FONT-SIZE:
10pt; FONT-FAMILY: 'Batang','serif'">손희진</SPAN><SPAN lang=EN-US style="FONT-SIZE:
10pt; FONT-FAMILY: 宋体"> [mailto:<A href="javascript:_e(%7B%7D,'cvml','heejin.sohn@samsung.com');"
target=_blank>heejin.sohn@samsung.com</A>] <BR></SPAN><B><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: 宋体">发送时间<SPAN lang=EN-US>:</SPAN></SPAN></B><SPAN
lang=EN-US style="FONT-SIZE: 10pt; FONT-FAMILY: 宋体"> 2015</SPAN><SPAN style="FONT-SIZE:
10pt; FONT-FAMILY: 宋体">年<SPAN lang=EN-US>3</SPAN>月<SPAN lang=EN-US>2</SPAN>日<SPAN
lang=EN-US> 8:28<BR></SPAN><B>收件人<SPAN lang=EN-US>:</SPAN></B><SPAN
lang=EN-US> <A href="javascript:_e(%7B%7D,'cvml','user@flume.apache.org');" target=_blank>user@flume.apache.org</A><BR></SPAN><B>主题<SPAN
lang=EN-US>:</SPAN></B><SPAN lang=EN-US> java.lang.OutOfMemoryError:
Direct buffer memory<U></U><U></U></SPAN></SPAN></P></DIV></DIV>
<P class=MsoNormal><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>Hi there,<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>I'm testing flume with thrift source, file channel, and HDFS
sink. <U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>Also there is a flume client which sends events using thrift,
and the size of each event is up to 30 Mbytes.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>It works fine for a short period, but after a few minutes,
following error occurs at org.apache.flume.channel.ChannelProcessor.<U></U><U></U></SPAN></P>
<TABLE cellPadding=0 border=0>
<TBODY>
<TR>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt">
<P class=MsoNormal style="MARGIN-BOTTOM: 3.75pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm"><SPAN
lang=EN-US style="FONT-SIZE: 9pt; FONT-FAMILY: GulimChe">8:15:36.450 PM<U></U><U></U></SPAN></P></TD>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt">
<P class=MsoNormal style="MARGIN-BOTTOM: 3.75pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm"><SPAN
lang=EN-US style="FONT-SIZE: 9pt; FONT-FAMILY: GulimChe">ERROR<U></U><U></U></SPAN></P></TD>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt">
<P class=MsoNormal style="MARGIN-BOTTOM: 3.75pt; MARGIN-LEFT: 0cm; MARGIN-RIGHT: 0cm"><SPAN
lang=EN-US style="FONT-SIZE: 9pt; FONT-FAMILY: GulimChe">org.apache.flume.channel.ChannelProcessor<U></U><U></U></SPAN></P></TD>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT:
0.75pt"><PRE><SPAN lang=EN-US>Error while writing to required channel: FileChannel
c1 { dataDirs: [/data/2/flumechannel/data] }<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>java.lang.OutOfMemoryError: Direct buffer memory<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.Bits.reserveMemory(Bits.java:658)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:123)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.IOUtil.write(IOUtil.java:58)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.Log.put(Log.java:633)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.thrift.server.Invocation.run(Invocation.java:18)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<U></U><U></U></SPAN></PRE><PRE><SPAN
lang=EN-US>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at java.lang.Thread.run(Thread.java:745)<U></U><U></U></SPAN></PRE></TD></TR></TBODY></TABLE>
<P><SPAN lang=EN-US>I increased direct memory size upto 2G byte, but it didn't
work.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>Here's my flume configuration.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>#source<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>tier1.sources.s1.type&nbsp;&nbsp;&nbsp;&nbsp;
= thrift<BR>tier1.sources.s1.bind&nbsp;&nbsp;&nbsp;&nbsp; = 0.0.0.0<BR>tier1.sources.s1.port&nbsp;&nbsp;&nbsp;&nbsp;
= 30010<BR>tier1.sources.s1.channels = c0 c1 memdefault<BR>tier1.sources.s1.selector.type
= multiplexing<BR>tier1.sources.s1.selector.header = category<BR>tier1.sources.s1.selector.mapping.Log4j
=&nbsp;c0 <BR>tier1.sources.s1.selector.mapping.Data = c1<BR>tier1.sources.s1.selector.default
= memDefault<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>#channel<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>tier1.channels.c1.type&nbsp;&nbsp; = memory<BR>tier1.channels.c1.checkpointDir=/data/2/flumechannel/checkpoint<BR>tier1.channels.c1.dataDirs=/data/2/flumechannel/data<BR>tier1.channels.c1.transactionCapacity
= 1<BR>tier1.channels.c1.maxFileSize = 500000000<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>#sink<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>tier1.sinks.k1.type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
= hdfs<BR>tier1.sinks.k1.channel&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; =
c1<BR>tier1.sinks.k1.hdfs.path = /user/soul<BR>tier1.sinks.k1.hdfs.round = false<BR>tier1.sinks.k1.hdfs.fileType
= DataStream<BR>tier1.sinks.k1.hdfs.rollCount = 1<BR>tier1.sinks.k1.hdfs.batchSize
= 1<BR>tier1.sinks.k1.hdfs.retryInterval = 10<BR>tier1.sinks.k1.hdfs.proxyUser
= soul<BR>tier1.sinks.k1.hdfs.maxOpenFiles = 10<BR>tier1.sinks.k1.hdfs.idleTimeOut
= 900<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>and java config. option: -Xmx2g -XX:MaxDirectMemorySize=2g
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC <U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>When I use memory channel instead of file channel, it works
great.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>I can't understand this phenomena.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>The only clue that I have is that the exception&nbsp;always
occurs after "org.apache.flume.channel.file.Log: Roll end"<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US>Hope your comments.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US>Thank you.<U></U><U></U></SPAN></P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P><SPAN lang=EN-US><U></U><U></U></SPAN>&nbsp;</P>
<P class=MsoNormal style="MARGIN: 0cm 0cm 0pt"><SPAN lang=EN-US><IMG src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=61dd5ae2e408a716506e7d7427027ed9e7c83b76671f074a279babd5c747fd394603607fb4f10f2b93a068f28056b538f4bcdeced46ed5ee08cece8541bc14eacf878f9a26ce15a0"><U></U><U></U></SPAN></P></DIV></DIV></BLOCKQUOTE>
<P>&nbsp;</P><!--SP:heejin.sohn-->
<TABLE style="BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-COLLAPSE: collapse; BORDER-BOTTOM:
0px; BORDER-LEFT: 0px" borderColor=black cellSpacing=0 cellPadding=0 width=241 border=0>
<TBODY>
<TR>
<TD height=122 width=241 align=center>
<P align=left><FONT face=Verdana><SPAN style="FONT-SIZE: 10pt"><STRONG>&nbsp;
손 희 진&nbsp;</STRONG>책임</SPAN></FONT> 
<P align=left>&nbsp;&nbsp;메모리제조센터&nbsp;<FONT face=verdana><FONT
face=굴림체>메모리기술혁신팀&nbsp;</FONT></FONT></P>
<P align=left>&nbsp; 설비엔지니어링그룹</P>
<P align=left><FONT face=verdana>&nbsp;&nbsp;Mobile : 010-3430-5168</FONT></P>
<P style="MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px; LINE-HEIGHT: 100%" align=left><FONT
face=verdana>&nbsp; E-mail : heejin.sohn@samsung.com</FONT></P></TD></TR></TBODY></TABLE><!--heejin.sohn:EP-->
<P>&nbsp;</P>
<P>&nbsp;</P><!--SP:heejin.sohn-->
<TABLE style="BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-COLLAPSE: collapse; BORDER-BOTTOM:
0px; BORDER-LEFT: 0px" borderColor=black cellSpacing=0 cellPadding=0 width=241 border=0>
<TBODY>
<TR>
<TD height=122 width=241 align=center>
<P align=left><FONT face=Verdana><SPAN style="FONT-SIZE: 10pt"><STRONG>&nbsp;
손 희 진&nbsp;</STRONG>책임</SPAN></FONT> 
<P align=left>&nbsp;&nbsp;메모리제조센터&nbsp;<FONT face=verdana><FONT
face=굴림체>메모리기술혁신팀&nbsp;</FONT></FONT></P>
<P align=left>&nbsp; 설비엔지니어링그룹</P>
<P align=left><FONT face=verdana>&nbsp;&nbsp;Mobile : 010-3430-5168</FONT></P>
<P style="MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px; LINE-HEIGHT: 100%" align=left><FONT
face=verdana>&nbsp; E-mail : heejin.sohn@samsung.com</FONT></P></TD></TR></TBODY></TABLE><!--heejin.sohn:EP-->
<P>&nbsp;</P></BODY></HTML><img src='http://ext.samsung.net/mailcheck/SeenTimeChecker?do=61dd5ae2e408a7162486384e88cb346bf1952ec14b898c73279babd5c747fd394603607fb4f10f2b93a068f28056b538f4bcdeced46ed5ee08cece8541bc14eacf878f9a26ce15a0'
border=0 width=0 height=0 style='display:none'>
Mime
View raw message