flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denny Ye <denny...@gmail.com>
Subject Re: GC overhead limit exceeded
Date Thu, 11 Oct 2012 02:40:29 GMT
A simple method like this : using special DirectEvent, it stores event body
from Source into direct bytebuffer with fixed body size. Thus, when this
event has been consumed by Sink, the reference of that direct block out of
heap might have chance to be reclaimed(only have chance).

This method may be limited by maximum direct memory, by default, this size
is equals with max heap size in HotSpot. Another focus is when that direct
bytebuffer storing event body can be reclaimed. While full GC is running,
it will check the direct memory usage and clear un-referenced direct block.

We can tuning the direct memory allocation with pre-allocated direct memory
blocks. This is more complex than method mentioned above.
Did I explain my thought clearly?

-Regards
Denny Ye

2012/10/11 Senthilvel Rangaswamy <senthilvel@gmail.com>

> Denny,
>
> How to do use direct memory ?
>
> Thanks,
> Senthil
>
>
> On Wed, Oct 10, 2012 at 7:25 PM, Denny Ye <dennyy99@gmail.com> wrote:
>
>> It might be caused by :
>> 1. Less heap memory. Increasing '-Xms -Xmx' options
>> 2. Disable that prompt. It's not recommended. Using
>> '-XX:-UseGCOverheadLimit' option
>> 3. There always have too many GC limitation with memory channel. I have
>> met some case with you before. I have tried to use direct memory for
>> storing event body out of heap in the past, that's effective method and no
>> any GC happened. I think you can test it by yourself.
>>
>> -Regards
>> Denny Ye
>>
>> 2012/10/11 Brock Noland <brock@cloudera.com>
>>
>>> Yep, sounds like: agent heap size < (capacity * avg event size)
>>>
>>> Brock
>>>
>>> On Wed, Oct 10, 2012 at 8:15 PM, Harish Mandala <mvharish14988@gmail.com>
>>> wrote:
>>> > Hi,
>>> >
>>> > In flume-env.sh, please add
>>> >
>>> > JAVA_OPTS="-Xms128m -Xmx256m"
>>> >
>>> > (Or whatever amount of memory works for you. I have Xmx set to 4g)
>>> >
>>> > Regards,
>>> > Harish
>>> >
>>> >
>>> > On Wed, Oct 10, 2012 at 9:08 PM, Camp, Roy <rcamp@ebay.com> wrote:
>>> >>
>>> >> I ran into the following error – I had to restart flume to recover.
>>>  Do I
>>> >> just need to adjust a GC setting of some sort or is there a larger
>>> issue
>>> >> here?
>>> >>
>>> >>
>>> >>
>>> >> Source: thriftLegacy
>>> >>
>>> >> Channel: memory (capacity: 1,000,000)
>>> >>
>>> >> Sink: avro
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> Exception in thread "pool-4-thread-2" java.lang.OutOfMemoryError: GC
>>> >> overhead limit exceeded
>>> >>
>>> >>         at
>>> >>
>>> java.util.concurrent.LinkedBlockingDeque.<init>(LinkedBlockingDeque.java:108)
>>> >>
>>> >>         at
>>> >>
>>> org.apache.flume.channel.MemoryChannel$MemoryTransaction.<init>(MemoryChannel.java:49)
>>> >>
>>> >>         at
>>> >>
>>> org.apache.flume.channel.MemoryChannel.createTransaction(MemoryChannel.java:264)
>>> >>
>>> >>         at
>>> >>
>>> org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118)
>>> >>
>>> >>         at
>>> >>
>>> org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:260)
>>> >>
>>> >>         at
>>> >>
>>> org.apache.flume.source.thriftLegacy.ThriftLegacySource$ThriftFlumeEventServerImpl.append(ThriftLegacySource.java:96)
>>> >>
>>> >>         at
>>> >>
>>> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:276)
>>> >>
>>> >>         at
>>> >>
>>> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:256)
>>> >>
>>> >>         at
>>> >>
>>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
>>> >>
>>> >>         at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>> >>
>>> >>         at
>>> >>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>> >>
>>> >>         at java.lang.Thread.run(Thread.java:679)
>>> >>
>>> >> Exception in thread
>>> "SinkRunner-PollingRunner-LoadBalancingSinkProcessor"
>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "Avro NettyTransceiver I/O Worker 1"
>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> >>
>>> >> Oct 10, 2012 5:51:39 PM
>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>> >>
>>> >> WARNING: Unexpected exception in the selector loop.
>>> >>
>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> >>
>>> >> Oct 10, 2012 5:52:35 PM
>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>> >>
>>> >> WARNING: Unexpected exception in the selector loop.
>>> >>
>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-2"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "pool-4-thread-4" java.lang.OutOfMemoryError: GC
>>> >> overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-0"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-10"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Oct 10, 2012 5:52:39 PM
>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>> >>
>>> >> WARNING: Unexpected exception in the selector loop.
>>> >>
>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> >>
>>> >> Oct 10, 2012 5:52:39 PM
>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>> >>
>>> >> WARNING: Unexpected exception in the selector loop.
>>> >>
>>> >> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-9"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-1"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-5"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-3"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-13"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-6"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-4"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-5"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Exception in thread "lifecycleSupervisor-1-8"
>>> java.lang.OutOfMemoryError:
>>> >> GC overhead limit exceeded
>>> >>
>>> >> Oct 10, 2012 5:52:48 PM
>>> >> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>> >>
>>> >> WARNING: Unexpected exception in the selector loop.
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> Thanks,
>>> >>
>>> >>
>>> >>
>>> >> Roy
>>> >>
>>> >>
>>> >>
>>> >>
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>>
>>
>>
>
>
> --
> ..Senthil
>
> "If there's anything more important than my ego around, I want it
>  caught and shot now."
>                                                     - Douglas Adams.
>
>

Mime
View raw message