flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Lee <defeng...@yottaa.com>
Subject Re: 回复: About file channel yum error
Date Tue, 23 Oct 2012 03:17:14 GMT
Hi,

Today Amazon has EBS outage, maybe it's a not flume issue, thanks you.

-Kevin

On Oct 23, 2012, at 10:31 AM, Brock Noland wrote:

> Hi,
> 
> Sorry that thread dump doesn't provide much. If you use kill -3 <pid> it prints
heap information at the bottom of the of the dump. Also the OOM error message long with a
configuration would be helpful.
> 
> Brock
> 
> On Mon, Oct 22, 2012 at 9:26 PM, Kevin Lee <defeng.li@yottaa.com> wrote:
> Hey,
> 
> I got a OOM error too, and the detail as follow:
> 
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> flume    22902 59.6 23.5 2813560 1802672 ?     Sl   Oct18 4210:34 /usr/java/jdk1.6.0_33/bin/java
-Xms500m -Xmx1500m -cp /etc/flume-ng/conf:/usr/lib/flume-ng/lib/*:/etc/hadoop/conf:/usr/lib/hadoop/lib/activation-1.1.jar
> 
> free -m
>              total       used       free     shared    buffers     cached
> Mem:          7467       7430         36          0         13       5011
> -/+ buffers/cache:       2405       5061
> Swap:         4095         15       4080
> 
> thread dump:
> 
> /usr/java/default/bin/jstack -F 22902
> Attaching to process ID 22902, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 20.8-b03
> Deadlock Detection:
> 
> No deadlocks found.
> 
> Thread 11929: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20,
line=196 (Compiled frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=68, line=2025 (Compiled frame)
>  - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609
(Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602
(Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23734: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23733: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23732: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23731: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23730: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23729: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23728: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23727: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23726: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23725: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609
(Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602
(Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted
frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23723: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23722: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23721: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23720: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23719: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23718: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23717: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23716: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23715: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23714: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609
(Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602
(Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Interpreted
frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23705: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23702: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Interpreted
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Interpreted frame)
>  - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Interpreted frame)
>  - org.apache.hadoop.fs.DelegationTokenRenewer.run() @bci=6, line=152 (Interpreted frame)
> 
> 
> Thread 23700: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=399 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23696: (state = IN_NATIVE)
>  - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame;
information may be imprecise)
>  - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
>  - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
>  - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
>  - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
>  - org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23,
line=219 (Compiled frame)
>  - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted
frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59,
line=886 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted
frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23693: (state = IN_NATIVE)
>  - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame;
information may be imprecise)
>  - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=210 (Compiled frame)
>  - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=65 (Compiled frame)
>  - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=69 (Compiled frame)
>  - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=80 (Compiled frame)
>  - org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23,
line=219 (Compiled frame)
>  - org.jboss.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted
frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59,
line=886 (Interpreted frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted
frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23666: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20,
line=196 (Compiled frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=68, line=2025 (Compiled frame)
>  - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609
(Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602
(Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23662: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20,
line=196 (Compiled frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
@bci=68, line=2025 (Compiled frame)
>  - java.util.concurrent.DelayQueue.take() @bci=57, line=164 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609
(Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602
(Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23660: (state = BLOCKED)
>  - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=156 (Compiled
frame)
>  - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42,
line=1987 (Compiled frame)
>  - java.util.concurrent.DelayQueue.take() @bci=28, line=160 (Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=609
(Compiled frame)
>  - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=602
(Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=78, line=947 (Compiled frame)
>  - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=18, line=907 (Compiled frame)
>  - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)
> 
> 
> Thread 23651: (state = BLOCKED)
> 
> 
> Thread 23650: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
>  - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
>  - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)
> 
> 
> Thread 23649: (state = BLOCKED)
>  - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
>  - java.lang.Object.wait() @bci=2, line=485 (Compiled frame)
>  - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Compiled frame)
> 
> 
> Thread 23645: (state = BLOCKED)
>  - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise)
>  - org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware,
org.apache.flume.lifecycle.LifecycleState[], long) @bci=109, line=71 (Compiled frame)
>  - org.apache.flume.lifecycle.LifecycleController.waitForOneOf(org.apache.flume.lifecycle.LifecycleAware,
org.apache.flume.lifecycle.LifecycleState[]) @bci=3, line=49 (Interpreted frame)
>  - org.apache.flume.node.Application.run() @bci=154, line=167 (Interpreted frame)
>  - org.apache.flume.node.Application.main(java.lang.String[]) @bci=21, line=70 (Interpreted
frame)
> 
> 
> 
> On Oct 16, 2012, at 11:10 PM, Brock Noland wrote:
> 
>> When the agent sees an OOM, can you take a thread dump (kill -3 <pid>) and
post it on pastebin? Please be sure to include the heap summary section at the bottom of the
dump.
>> 
>> Brock
>> 
>> On Mon, Oct 15, 2012 at 10:08 PM, Yanzhi.liu <404307305@qq.com> wrote:
>> Hello Brock:
>>     My flume's configuration is
>> agent_foo.sources = tailsource-1
>> agent_foo.channels = fileChannel
>> agent_foo.sinks = hdfsSink-1
>> agent_foo.sources.tailsource-1.type = com.chinacache.cpis.sources.CpisAvroSource
>> agent_foo.sources.tailsource-1.channels=fileChannel
>> agent_foo.sources.tailsource-1.bind=221.130.18.90
>> agent_foo.sources.tailsource-1.port=4545
>> #agent_foo.sources.tailsource-1.threads=10
>> agent_foo.channels.fileChannel.type = file
>> agent_foo.channels.fileChannel.checkpointDir=/home/hadoop/checkpoint
>> agent_foo.channels.fileChannel.dataDirs=/data/cache2/data
>> agent_foo.channels.fileChannel.maxFileSize=524288000
>> #agent_foo.channels.fileChannel.capacity = 10000
>> agent_foo.sinks.hdfsSink-1.type = hdfs
>> agent_foo.sinks.hdfsSink-1.channel = fileChannel
>> agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{deviceId}/%{hour}
>> #agent_foo.sinks.hdfsSink-1.hdfs.path = hdfs://CMN-NJ-2-579:9000/user/hadoop/fc_logs/%{month}/%{day}/%{hour}/%{deviceId}
>> agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = cpisfc-assess.log
>> agent_foo.sinks.hdfsSink-1.hdfs.rollInterval=600
>> agent_foo.sinks.hdfsSink-1.hdfs.rollCount=0
>> agent_foo.sinks.hdfsSink-1.hdfs.rollSize = 506870912
>> agent_foo.sinks.hdfsSink-1.hdfs.fileType=CompressedStream
>> agent_foo.sinks.hdfsSink-1.hdfs.codeC=gzip
>> agent_foo.sinks.hdfsSink-1.hdfs.writeFormat=Text
>> agent_foo.sinks.hdfsSink-1.hdfs.batchSize=1
>> agent_foo.sinks.hdfsSink-1.serializer=avro_event
>>     And the java configuration was JAVA_OPTS="-Xms100m -Xmx2048m -Dcom.sun.management.jmxremote";
>> Thank you very much!
>> My Name :
>> Yanzhi Liu
>> 
>> 
>> ------------------ 原始邮件 ------------------
>> 发件人: "Brock Noland"<brock@cloudera.com>;
>> 发送时间: 2012年10月16日(星期二) 中午11:01
>> 收件人: "user"<user@flume.apache.org>;
>> 主题: Re: About file channel yum error
>> 
>> What is your channel capacity and configured agent heap size?
>> 
>> On Monday, October 15, 2012 at 9:19 PM, Yanzhi.liu wrote:
>> 
>>> Hello everyone:
>>>     I am using the flume 1.3.0.But the flume clusters was working after two days
and the flume was stoped. And the flume.log was only error about the:
>>> 14 Oct 2012 15:29:17,018 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor]
(org.apache.flume.sink.hdfs.HDFSEventSink.process:448)  - process failed
>>> java.lang.OutOfMemoryError
>>>         at java.io.RandomAccessFile.readBytes(Native Method)
>>>         at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
>>>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
>>>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
>>>         at org.apache.flume.channel.file.LogFile.readDelimitedBuffer(LogFile.java:486)
>>>         at org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:258)
>>>         at org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:298)
>>>         at org.apache.flume.channel.file.Log.get(Log.java:409)
>>>         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:447)
>>>         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>>>         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:91)
>>>         at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:387)
>>>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
>>>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
>>>         at java.lang.Thread.run(Thread.java:662)
>>> So I want to how to avoid the error.
>>> Thanks very much!
>>> My Name:
>>> Yanzhi Liu
>> 
>> 
>> 
>> 
>> -- 
>> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
> 
> 
> 
> 
> -- 
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/


Mime
View raw message