flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: GC overhead limit exceeded
Date Thu, 11 Oct 2012 01:17:10 GMT
Yep, sounds like: agent heap size < (capacity * avg event size)

Brock

On Wed, Oct 10, 2012 at 8:15 PM, Harish Mandala <mvharish14988@gmail.com> wrote:
> Hi,
>
> In flume-env.sh, please add
>
> JAVA_OPTS="-Xms128m -Xmx256m"
>
> (Or whatever amount of memory works for you. I have Xmx set to 4g)
>
> Regards,
> Harish
>
>
> On Wed, Oct 10, 2012 at 9:08 PM, Camp, Roy <rcamp@ebay.com> wrote:
>>
>> I ran into the following error – I had to restart flume to recover.  Do I
>> just need to adjust a GC setting of some sort or is there a larger issue
>> here?
>>
>>
>>
>> Source: thriftLegacy
>>
>> Channel: memory (capacity: 1,000,000)
>>
>> Sink: avro
>>
>>
>>
>>
>>
>> Exception in thread "pool-4-thread-2" java.lang.OutOfMemoryError: GC
>> overhead limit exceeded
>>
>>         at
>> java.util.concurrent.LinkedBlockingDeque.<init>(LinkedBlockingDeque.java:108)
>>
>>         at
>> org.apache.flume.channel.MemoryChannel$MemoryTransaction.<init>(MemoryChannel.java:49)
>>
>>         at
>> org.apache.flume.channel.MemoryChannel.createTransaction(MemoryChannel.java:264)
>>
>>         at
>> org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:118)
>>
>>         at
>> org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:260)
>>
>>         at
>> org.apache.flume.source.thriftLegacy.ThriftLegacySource$ThriftFlumeEventServerImpl.append(ThriftLegacySource.java:96)
>>
>>         at
>> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:276)
>>
>>         at
>> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:256)
>>
>>         at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
>>
>>         at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>
>>         at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>
>>         at java.lang.Thread.run(Thread.java:679)
>>
>> Exception in thread "SinkRunner-PollingRunner-LoadBalancingSinkProcessor"
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> Exception in thread "Avro NettyTransceiver I/O Worker 1"
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> Oct 10, 2012 5:51:39 PM
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>
>> WARNING: Unexpected exception in the selector loop.
>>
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> Oct 10, 2012 5:52:35 PM
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>
>> WARNING: Unexpected exception in the selector loop.
>>
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-2" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "pool-4-thread-4" java.lang.OutOfMemoryError: GC
>> overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-0" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-10" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Oct 10, 2012 5:52:39 PM
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>
>> WARNING: Unexpected exception in the selector loop.
>>
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> Oct 10, 2012 5:52:39 PM
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>
>> WARNING: Unexpected exception in the selector loop.
>>
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-9" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-1" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-5" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-3" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-13" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-6" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-4" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-5" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Exception in thread "lifecycleSupervisor-1-8" java.lang.OutOfMemoryError:
>> GC overhead limit exceeded
>>
>> Oct 10, 2012 5:52:48 PM
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker
>>
>> WARNING: Unexpected exception in the selector loop.
>>
>>
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Roy
>>
>>
>>
>>
>
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message