flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yvonne Lam <yvo...@kiha.com>
Subject timeout on unavailable s3 collectorSink?
Date Fri, 15 Jul 2011 21:43:14 GMT
I'm new to flume, so our flume setup is correspondingly simple:  one master,
one agent, and one collector, which writes data to s3

Node-1 DatalogFlow  tailDir("<logdir>","Observation-.*\\.log",true)
autoBEChain
Consumer-1 DatalogFlow autoCollectorSource
collectorSink("s3n://<key>:<secret>@<bucket>/logs/dataSource=%{tailSrcFile}/date=%Y%m%d/","%{tailSrcFile}","30000")
version:  Flume 0.9.3-cdh3u0

On 7/12/11, s3 had a brief problem with high error rates, and the collector
stopped writing data to s3.  netstat showed the collector's http connections
were in a perpetual "close_wait" state; thread dump below.
To get things back to normal, I restarted the flume process on the
collector's physical node and and remapped/reconfigured Consumer-1.

Is there a configuration that would have allowed a graceful timeout when s3
became unavailable?  For our purposes, it would have been fine to drop data
for the period that s3 was unavailable and resume sending once it was.

thanks,

Yvonne


2011-07-13 16:28:56
Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode, sharing):

"Thrift server:class org.apache.thrift.TProcessorFactory on class
org.apache.thrift.transport.TSaneServerSocket" prio=10 tid=0x09c08400
nid=0xdda runnable [0xb428c000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    - locked <0x848d0de0> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:132)
    at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
    at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)

"logicalNode ScenarioConsumer-1-846626" prio=10 tid=0x0a0b5800 nid=0xdd8
waiting on condition [0xb40fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x848d0f90> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
    at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
    at
com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:205)
    at
com.cloudera.flume.handlers.debug.LazyOpenSource.next(LazyOpenSource.java:57)
    at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:89)

"pool-5-thread-1" prio=10 tid=0x09ce6800 nid=0xdc2 waiting on condition
[0xb4335000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x848cf678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
    at
com.cloudera.flume.handlers.thrift.ThriftEventSource.enqueue(ThriftEventSource.java:117)
    at
com.cloudera.flume.handlers.thrift.ThriftEventSource$1.append(ThriftEventSource.java:135)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServerImpl.append(ThriftFlumeEventServerImpl.java:45)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:381)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:361)
    at
org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"Thrift server:class org.apache.thrift.TProcessorFactory on class
org.apache.thrift.transport.TSaneServerSocket" prio=10 tid=0x09c07400
nid=0xdb6 runnable [0xb405c000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
    - locked <0x848cf090> (a java.net.SocksSocketImpl)
    at java.net.ServerSocket.implAccept(ServerSocket.java:462)
    at java.net.ServerSocket.accept(ServerSocket.java:430)
    at
org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:132)
    at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
    at
org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)

"logicalNode ObservationConsumer-1-846590" prio=10 tid=0x09c24400 nid=0xdb4
waiting on condition [0xb42dd000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x848d08a0> (a
java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
    at com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:213)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:170)
    - locked <0x848d02a0> (a
com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
    at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
    at
com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:108)
    at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
    at
com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
    at
com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:217)
    at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
    at
com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:75)
    at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:93)

"MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x0a0af800
nid=0x1acd in Object.wait() [0xb3fba000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
    at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)

"DestroyJavaVM" prio=10 tid=0x09bb5800 nid=0x2292 waiting on condition
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"ChokeManager" daemon prio=10 tid=0x09f97400 nid=0x22ac waiting on condition
[0xb4386000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at
com.cloudera.flume.handlers.debug.ChokeManager.run(ChokeManager.java:143)

"Heartbeat" prio=10 tid=0x09f96400 nid=0x22ab waiting on condition
[0xb43d7000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.cloudera.util.Clock$DefaultClock.doSleep(Clock.java:62)
    at com.cloudera.util.Clock.sleep(Clock.java:88)
    at
com.cloudera.flume.agent.LivenessManager$HeartbeatThread.run(LivenessManager.java:187)

"Thread-1" prio=10 tid=0x09f95800 nid=0x22aa waiting on condition
[0xb4428000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at com.cloudera.util.Clock$DefaultClock.doSleep(Clock.java:62)
    at com.cloudera.util.Clock.sleep(Clock.java:88)
    at
com.cloudera.flume.reporter.MasterReportPusher$PusherThread.run(MasterReportPusher.java:118)

"Timer-0" daemon prio=10 tid=0x09f46c00 nid=0x22a9 in Object.wait()
[0xb4489000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x844a7c88> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x844a7c88> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"26392982@qtp-15532856-1 - Acceptor0 SelectChannelConnector@0.0.0.0:35862"
prio=10 tid=0x09f0e800 nid=0x22a8 runnable [0xb44da000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x84567428> (a sun.nio.ch.Util$2)
    - locked <0x84567438> (a java.util.Collections$UnmodifiableSet)
    - locked <0x845673e8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:484)
    at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
    at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
    at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"11423854@qtp-15532856-0" prio=10 tid=0x09f03800 nid=0x22a7 in Object.wait()
[0xb452b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x8446c658> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)
    at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:625)
    - locked <0x8446c658> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

"Low Memory Detector" daemon prio=10 tid=0x09bef000 nid=0x229a runnable
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x09bed000 nid=0x2299 waiting on
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x09beb800 nid=0x2298 waiting on
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x09be4c00 nid=0x2297 in Object.wait()
[0xb486a000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x09be3400 nid=0x2296 in
Object.wait() [0xb48bc000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x844002e8> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x844002e8> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x09bd9000 nid=0x2295 runnable

"VM Periodic Task Thread" prio=10 tid=0x09bfac00 nid=0x229b waiting on
condition

JNI global references: 1108

Heap
 def new generation   total 5760K, used 5180K [0x7eeb0000, 0x7f4e0000,
0x84400000)
  eden space 5184K,  98% used [0x7eeb0000, 0x7f3b0800, 0x7f3c0000)
  from space 576K,  10% used [0x7f450000, 0x7f45e9b0, 0x7f4e0000)
  to   space 576K,   0% used [0x7f3c0000, 0x7f3c0000, 0x7f450000)
 tenured generation   total 12480K, used 7936K [0x84400000, 0x85030000,
0x8eeb0000)
   the space 12480K,  63% used [0x84400000, 0x84bc02c8, 0x84bc0400,
0x85030000)
 compacting perm gen  total 13056K, used 12836K [0x8eeb0000, 0x8fb70000,
0x92eb0000)
   the space 13056K,  98% used [0x8eeb0000, 0x8fb39030, 0x8fb39200,
0x8fb70000)
    ro space 10240K,  61% used [0x92eb0000, 0x934d8a38, 0x934d8c00,
0x938b0000)
    rw space 12288K,  60% used [0x938b0000, 0x93fe8ec0, 0x93fe9000,
0x944b0000)

2011-07-13 17:11:06
Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode, sharing):

"SpawningLogicalNode ObservationConsumer-1" prio=10 tid=0xb4115400 nid=0x8b0
waiting for monitor entry [0xb40ad000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at
com.cloudera.flume.agent.LogicalNode.openLoadNode(LogicalNode.java:166)
    - waiting to lock <0x848ced80> (a com.cloudera.flume.agent.LogicalNode)
    at com.cloudera.flume.agent.LogicalNode.loadConfig(LogicalNode.java:328)
    at
com.cloudera.flume.agent.LogicalNode.checkConfig(LogicalNode.java:371)
    at
com.cloudera.flume.agent.LivenessManager$1.run(LivenessManager.java:141)

"SpawningLogicalNode ObservationConsumer-1" prio=10 tid=0xb4119400 nid=0x8ae
waiting for monitor entry [0xb428c000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at
com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:169)
    - waiting to lock <0x848d02a0> (a
com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.endtoend.AckChecksumChecker.close(AckChecksumChecker.java:192)
    at
com.cloudera.flume.collector.CollectorSink.close(CollectorSink.java:223)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.debug.LazyOpenDecorator.close(LazyOpenDecorator.java:81)
    at
com.cloudera.flume.agent.LogicalNode.openSourceSink(LogicalNode.java:128)
    at
com.cloudera.flume.agent.LogicalNode.openLoadNode(LogicalNode.java:169)
    - locked <0x848ced80> (a com.cloudera.flume.agent.LogicalNode)
    at com.cloudera.flume.agent.LogicalNode.loadConfig(LogicalNode.java:328)
    at
com.cloudera.flume.agent.LogicalNode.checkConfig(LogicalNode.java:371)
    at
com.cloudera.flume.agent.LivenessManager$1.run(LivenessManager.java:141)

"pool-5-thread-2" prio=10 tid=0x0a0b5800 nid=0x8ad waiting on condition
[0xb40fe000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x848cf678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
    at
com.cloudera.flume.handlers.thrift.ThriftEventSource.enqueue(ThriftEventSource.java:117)
    at
com.cloudera.flume.handlers.thrift.ThriftEventSource$1.append(ThriftEventSource.java:135)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServerImpl.append(ThriftFlumeEventServerImpl.java:45)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:381)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:361)
    at
org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"pool-5-thread-1" prio=10 tid=0x09ce6800 nid=0xdc2 waiting on condition
[0xb4335000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x848cf678> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    at
java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
    at
com.cloudera.flume.handlers.thrift.ThriftEventSource.enqueue(ThriftEventSource.java:117)
    at
com.cloudera.flume.handlers.thrift.ThriftEventSource$1.append(ThriftEventSource.java:135)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServerImpl.append(ThriftFlumeEventServerImpl.java:45)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:381)
    at
com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:361)
    at
org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

"logicalNode ObservationConsumer-1-846590" prio=10 tid=0x09c24400 nid=0xdb4
waiting on condition [0xb42dd000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x848d08a0> (a
java.util.concurrent.CountDownLatch$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
    at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
    at com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:213)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:170)
    - locked <0x848d02a0> (a
com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
    at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
    at
com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:108)
    at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
    at
com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
    at
com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:217)
    at
com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
    at
com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:75)
    at
com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:93)

"MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x0a0af800
nid=0x1acd in Object.wait() [0xb3fba000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
    at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)

"DestroyJavaVM" prio=10 tid=0x09bb5800 nid=0x2292 waiting on condition
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"ChokeManager" daemon prio=10 tid=0x09f97400 nid=0x22ac waiting on condition
[0xb4386000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at
com.cloudera.flume.handlers.debug.ChokeManager.run(ChokeManager.java:143)

"Heartbeat" prio=10 tid=0x09f96400 nid=0x22ab waiting for monitor entry
[0xb43d7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at
com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:169)
    - waiting to lock <0x848d02a0> (a
com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.endtoend.AckChecksumChecker.close(AckChecksumChecker.java:192)
    at
com.cloudera.flume.collector.CollectorSink.close(CollectorSink.java:223)
    at
com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
    at
com.cloudera.flume.handlers.debug.LazyOpenDecorator.close(LazyOpenDecorator.java:81)
    at com.cloudera.flume.agent.LogicalNode.close(LogicalNode.java:469)
    at
com.cloudera.flume.agent.LogicalNodeManager.decommission(LogicalNodeManager.java:131)
    - locked <0x84448f60> (a com.cloudera.flume.agent.LogicalNodeManager)
    at
com.cloudera.flume.agent.LogicalNodeManager.decommissionAllBut(LogicalNodeManager.java:185)
    - locked <0x84448f60> (a com.cloudera.flume.agent.LogicalNodeManager)
    at
com.cloudera.flume.agent.LivenessManager.checkLogicalNodes(LivenessManager.java:119)
    at
com.cloudera.flume.agent.LivenessManager.heartbeatChecks(LivenessManager.java:154)
    at
com.cloudera.flume.agent.LivenessManager$HeartbeatThread.run(LivenessManager.java:184)

"Thread-1" prio=10 tid=0x09f95800 nid=0x22aa waiting for monitor entry
[0xb4428000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at com.cloudera.flume.agent.LogicalNode.getReport(LogicalNode.java:395)
    - waiting to lock <0x848ced80> (a com.cloudera.flume.agent.LogicalNode)
    at
com.cloudera.flume.agent.LogicalNodeManager.getReport(LogicalNodeManager.java:139)
    at com.cloudera.flume.agent.FlumeNode.getReport(FlumeNode.java:824)
    at com.cloudera.flume.agent.FlumeNode$1.getMetrics(FlumeNode.java:257)
    at
com.cloudera.flume.reporter.MasterReportPusher$PusherThread.queryReportMan(MasterReportPusher.java:94)
    at
com.cloudera.flume.reporter.MasterReportPusher$PusherThread.sendReports(MasterReportPusher.java:109)
    at
com.cloudera.flume.reporter.MasterReportPusher$PusherThread.run(MasterReportPusher.java:119)

"Timer-0" daemon prio=10 tid=0x09f46c00 nid=0x22a9 in Object.wait()
[0xb4489000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x844a7c88> (a java.util.TaskQueue)
    at java.util.TimerThread.mainLoop(Timer.java:509)
    - locked <0x844a7c88> (a java.util.TaskQueue)
    at java.util.TimerThread.run(Timer.java:462)

"26392982@qtp-15532856-1 - Acceptor0 SelectChannelConnector@0.0.0.0:35862"
prio=10 tid=0x09f0e800 nid=0x22a8 runnable [0xb44da000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x84567428> (a sun.nio.ch.Util$2)
    - locked <0x84567438> (a java.util.Collections$UnmodifiableSet)
    - locked <0x845673e8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:484)
    at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
    at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
    at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"11423854@qtp-15532856-0" prio=10 tid=0x09f03800 nid=0x22a7 in Object.wait()
[0xb452b000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x8446c658> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)
    at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:625)
    - locked <0x8446c658> (a org.mortbay.thread.QueuedThreadPool$PoolThread)

"Low Memory Detector" daemon prio=10 tid=0x09bef000 nid=0x229a runnable
[0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x09bed000 nid=0x2299 waiting on
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x09beb800 nid=0x2298 waiting on
condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x09be4c00 nid=0x2297 in Object.wait()
[0xb486a000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    - locked <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x09be3400 nid=0x2296 in
Object.wait() [0xb48bc000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x844002e8> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x844002e8> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x09bd9000 nid=0x2295 runnable

"VM Periodic Task Thread" prio=10 tid=0x09bfac00 nid=0x229b waiting on
condition

JNI global references: 1115

Heap
 def new generation   total 5760K, used 3459K [0x7eeb0000, 0x7f4e0000,
0x84400000)
  eden space 5184K,  65% used [0x7eeb0000, 0x7f204b68, 0x7f3c0000)
  from space 576K,   8% used [0x7f450000, 0x7f45c0e8, 0x7f4e0000)
  to   space 576K,   0% used [0x7f3c0000, 0x7f3c0000, 0x7f450000)
 tenured generation   total 12480K, used 7936K [0x84400000, 0x85030000,
0x8eeb0000)
   the space 12480K,  63% used [0x84400000, 0x84bc02c8, 0x84bc0400,
0x85030000)
 compacting perm gen  total 13056K, used 12836K [0x8eeb0000, 0x8fb70000,
0x92eb0000)
   the space 13056K,  98% used [0x8eeb0000, 0x8fb39178, 0x8fb39200,
0x8fb70000)
    ro space 10240K,  61% used [0x92eb0000, 0x934d8a38, 0x934d8c00,
0x938b0000)
    rw space 12288K,  60% used [0x938b0000, 0x93fe8ec0, 0x93fe9000,
0x944b0000)

Mime
View raw message