Cloudera manager provides a resident memory usage graph which just increases monotonically. It seems that deallocation occurred from time to time, but the amount is negligible considering the allocation rate.

Direct buffer size was set up to 10g and this tendency was independent of buffer size.

Also, the thrift client which sends flume events stopped last night and continued to work after 10 hours. During that time, the memory usage of the flume agent has never changed (Garbage collection has never happened).

Anyway, it seems that the problem is caused by memory deallocation(or garbage collection) which never or barely occurred during the operation of file channel (or channel processor).

I doubted HW problem. However, this test was performed on other server and the result was the same.

 

 

Thanks for your comments.

Heejin Sohn

 

------- Original Message -------

Sender : Ashish<paliwalashish@gmail.com>

Date : 2015-03-02 13:53 (GMT+09:00)

Title : Re: Re: java.lang.OutOfMemoryError: Direct buffer memory

 

Can you monitor how much off-heap/direct buffers are allocated? Try to up the -XX:MaxDirectMemorySize=2g to a higher value and see if the Agents holds good. If the allocation rate is higher than deallocation rate, you would still end up with OOME. 

Let's try few setting and monitor how things go.





On Mon, Mar 2, 2015 at 9:41 AM, HEEJIN SOHN <heejin.sohn@samsung.com> wrote:

Wan Yi //

Tried as you've suggested (maxFileSize = 1M), but only met the same exception.

 

Hari Shreedharan//

All flume options are set using Cloudera manager. (CDH-5.2.0-1)

Also, possible event size is under 100MB.

 

Here is the log pattern.

8:15:26.367 PM  INFO  org.apache.flume.sink.hdfs.BucketWriter Creating /user/25068525641.jpg.27.1425035726356.tmp
8:15:36.407 PM  INFO  org.apache.flume.channel.file.Log Roll start /data/2/flumechannel/data
8:15:36.407 PM  INFO  org.apache.flume.channel.file.LogFile Opened /data/2/flumechannel/data/log-4
8:15:36.447 PM  INFO  org.apache.flume.channel.file.LogFile Closing /data/2/flumechannel/data/log-3
8:15:36.447 PM  INFO  org.apache.flume.channel.file.Log Roll end


8:15:36.450 PM  ERROR  org.apache.flume.channel.ChannelProcessor Error while writing to required channel: FileChannel c1 { dataDirs: [/data/2/flumechannel/data] }
java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:658)
 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
 at sun.nio.ch.IOUtil.write(IOUtil.java:58)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
 at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)
 at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)
 at org.apache.flume.channel.file.Log.put(Log.java:633)
 at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
 at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
 at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
 at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
 at org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)
 at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)
 at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)
 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
 at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
 at org.apache.thrift.server.Invocation.run(Invocation.java:18)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)


8:15:36.451 PM  ERROR  org.apache.thrift.ProcessFunction Internal error processing append java.lang.OutOfMemoryError: Direct buffer memory
 at java.nio.Bits.reserveMemory(Bits.java:658)
 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
 at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
 at sun.nio.ch.IOUtil.write(IOUtil.java:58)
 at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
 at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)
 at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)
 at org.apache.flume.channel.file.Log.put(Log.java:633)
 at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
 at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
 at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
 at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
 at org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)
 at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)
 at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)
 at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
 at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
 at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
 at org.apache.thrift.server.Invocation.run(Invocation.java:18)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 at java.lang.Thread.run(Thread.java:745)

 

Thank you very much.

 

Heejin Sohn.

 

 

------- Original Message -------

Sender : Hari Shreedharan<hshreedharan@cloudera.com>

Date : 2015-03-02 12:22 (GMT+09:00)

Title : Re: java.lang.OutOfMemoryError: Direct buffer memory

 

How are you passing the JVM options in. Is the event being written more than 2g? 

On Sunday, March 1, 2015, Wan Yi(武汉_技术部_搜索与精准化_万毅) <wanyi@yhd.com> wrote:

tier1.channels.c1.maxFileSize = 500000000

 

try to change the maxFileSize to a small value, may be 1M or 10M

 

 

Wayne Wan

发件人: 손희진 [mailto:heejin.sohn@samsung.com]
发送时间: 201532 8:28
收件人: user@flume.apache.org
主题: java.lang.OutOfMemoryError: Direct buffer memory

 

Hi there,

I'm testing flume with thrift source, file channel, and HDFS sink.

Also there is a flume client which sends events using thrift, and the size of each event is up to 30 Mbytes.

It works fine for a short period, but after a few minutes, following error occurs at org.apache.flume.channel.ChannelProcessor.

8:15:36.450 PM

ERROR

org.apache.flume.channel.ChannelProcessor

Error while writing to required channel: FileChannel c1 { dataDirs: [/data/2/flumechannel/data] }
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)
        at sun.nio.ch.IOUtil.write(IOUtil.java:58)
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
        at org.apache.flume.channel.file.LogFile$Writer.write(LogFile.java:313)
        at org.apache.flume.channel.file.LogFile$Writer.put(LogFile.java:267)
        at org.apache.flume.channel.file.Log.put(Log.java:633)
        at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doPut(FileChannel.java:469)
        at org.apache.flume.channel.BasicTransactionSemantics.put(BasicTransactionSemantics.java:93)
        at org.apache.flume.channel.BasicChannelSemantics.put(BasicChannelSemantics.java:80)
        at org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:265)
        at org.apache.flume.source.ThriftSource$ThriftSourceHandler.append(ThriftSource.java:253)
        at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:251)
        at org.apache.flume.thrift.ThriftSourceProtocol$Processor$append.getResult(ThriftSourceProtocol.java:236)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:478)
        at org.apache.thrift.server.Invocation.run(Invocation.java:18)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

I increased direct memory size upto 2G byte, but it didn't work.

Here's my flume configuration.

 

#source

tier1.sources.s1.type     = thrift
tier1.sources.s1.bind     = 0.0.0.0
tier1.sources.s1.port     = 30010
tier1.sources.s1.channels = c0 c1 memdefault
tier1.sources.s1.selector.type = multiplexing
tier1.sources.s1.selector.header = category
tier1.sources.s1.selector.mapping.Log4j = c0
tier1.sources.s1.selector.mapping.Data = c1
tier1.sources.s1.selector.default = memDefault

 

#channel

tier1.channels.c1.type   = memory
tier1.channels.c1.checkpointDir=/data/2/flumechannel/checkpoint
tier1.channels.c1.dataDirs=/data/2/flumechannel/data
tier1.channels.c1.transactionCapacity = 1
tier1.channels.c1.maxFileSize = 500000000

 

#sink

tier1.sinks.k1.type         = hdfs
tier1.sinks.k1.channel      = c1
tier1.sinks.k1.hdfs.path = /user/soul
tier1.sinks.k1.hdfs.round = false
tier1.sinks.k1.hdfs.fileType = DataStream
tier1.sinks.k1.hdfs.rollCount = 1
tier1.sinks.k1.hdfs.batchSize = 1
tier1.sinks.k1.hdfs.retryInterval = 10
tier1.sinks.k1.hdfs.proxyUser = soul
tier1.sinks.k1.hdfs.maxOpenFiles = 10
tier1.sinks.k1.hdfs.idleTimeOut = 900

 

and java config. option: -Xmx2g -XX:MaxDirectMemorySize=2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC

 

When I use memory channel instead of file channel, it works great.

I can't understand this phenomena.

The only clue that I have is that the exception always occurs after "org.apache.flume.channel.file.Log: Roll end"

 

 

Hope your comments.

Thank you.

 

 

 

 




--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal