flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Slifka <robert.sli...@gmail.com>
Subject Re: timeout on unavailable s3 collectorSink?
Date Sat, 16 Jul 2011 07:09:02 GMT
We had a similar situation.

Our roll is set to the default 30s, so we have a monitor that alerts us when
we don't see the expected number of files landing on S3 within a few
minutes.

We bounced the collector and everything was back to normal.

Rob



On Fri, Jul 15, 2011 at 2:43 PM, Yvonne Lam <yvonne@kiha.com> wrote:

> I'm new to flume, so our flume setup is correspondingly simple:  one
> master, one agent, and one collector, which writes data to s3
>
> Node-1 DatalogFlow  tailDir("<logdir>","Observation-.*\\.log",true)
> autoBEChain
> Consumer-1 DatalogFlow autoCollectorSource
> collectorSink("s3n://<key>:<secret>@<bucket>/logs/dataSource=%{tailSrcFile}/date=%Y%m%d/","%{tailSrcFile}","30000")
> version:  Flume 0.9.3-cdh3u0
>
> On 7/12/11, s3 had a brief problem with high error rates, and the collector
> stopped writing data to s3.  netstat showed the collector's http connections
> were in a perpetual "close_wait" state; thread dump below.
> To get things back to normal, I restarted the flume process on the
> collector's physical node and and remapped/reconfigured Consumer-1.
>
> Is there a configuration that would have allowed a graceful timeout when s3
> became unavailable?  For our purposes, it would have been fine to drop data
> for the period that s3 was unavailable and resume sending once it was.
>
> thanks,
>
> Yvonne
>
>
> 2011-07-13 16:28:56
> Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode, sharing):
>
> "Thrift server:class org.apache.thrift.TProcessorFactory on class
> org.apache.thrift.transport.TSaneServerSocket" prio=10 tid=0x09c08400
> nid=0xdda runnable [0xb428c000]
>    java.lang.Thread.State: RUNNABLE
>     at java.net.PlainSocketImpl.socketAccept(Native Method)
>     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>     - locked <0x848d0de0> (a java.net.SocksSocketImpl)
>     at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>     at java.net.ServerSocket.accept(ServerSocket.java:430)
>     at
> org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:132)
>     at
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>     at
> org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
>
> "logicalNode ScenarioConsumer-1-846626" prio=10 tid=0x0a0b5800 nid=0xdd8
> waiting on condition [0xb40fe000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x848d0f90> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
>     at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424)
>     at
> com.cloudera.flume.handlers.thrift.ThriftEventSource.next(ThriftEventSource.java:205)
>     at
> com.cloudera.flume.handlers.debug.LazyOpenSource.next(LazyOpenSource.java:57)
>     at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:89)
>
> "pool-5-thread-1" prio=10 tid=0x09ce6800 nid=0xdc2 waiting on condition
> [0xb4335000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x848cf678> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
>     at
> com.cloudera.flume.handlers.thrift.ThriftEventSource.enqueue(ThriftEventSource.java:117)
>     at
> com.cloudera.flume.handlers.thrift.ThriftEventSource$1.append(ThriftEventSource.java:135)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServerImpl.append(ThriftFlumeEventServerImpl.java:45)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:381)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:361)
>     at
> org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
> "Thrift server:class org.apache.thrift.TProcessorFactory on class
> org.apache.thrift.transport.TSaneServerSocket" prio=10 tid=0x09c07400
> nid=0xdb6 runnable [0xb405c000]
>    java.lang.Thread.State: RUNNABLE
>     at java.net.PlainSocketImpl.socketAccept(Native Method)
>     at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>     - locked <0x848cf090> (a java.net.SocksSocketImpl)
>     at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>     at java.net.ServerSocket.accept(ServerSocket.java:430)
>     at
> org.apache.thrift.transport.TSaneServerSocket.acceptImpl(TSaneServerSocket.java:132)
>     at
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>     at
> org.apache.thrift.server.TSaneThreadPoolServer$1.run(TSaneThreadPoolServer.java:175)
>
> "logicalNode ObservationConsumer-1-846590" prio=10 tid=0x09c24400 nid=0xdb4
> waiting on condition [0xb42dd000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x848d08a0> (a
> java.util.concurrent.CountDownLatch$Sync)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>     at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>     at
> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:213)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:170)
>     - locked <0x848d02a0> (a
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>     at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>     at
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:108)
>     at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>     at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>     at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:217)
>     at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>     at
> com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:75)
>     at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:93)
>
> "MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x0a0af800
> nid=0x1acd in Object.wait() [0xb3fba000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>     - locked <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
>     at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)
>
> "DestroyJavaVM" prio=10 tid=0x09bb5800 nid=0x2292 waiting on condition
> [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "ChokeManager" daemon prio=10 tid=0x09f97400 nid=0x22ac waiting on
> condition [0xb4386000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> com.cloudera.flume.handlers.debug.ChokeManager.run(ChokeManager.java:143)
>
> "Heartbeat" prio=10 tid=0x09f96400 nid=0x22ab waiting on condition
> [0xb43d7000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at com.cloudera.util.Clock$DefaultClock.doSleep(Clock.java:62)
>     at com.cloudera.util.Clock.sleep(Clock.java:88)
>     at
> com.cloudera.flume.agent.LivenessManager$HeartbeatThread.run(LivenessManager.java:187)
>
> "Thread-1" prio=10 tid=0x09f95800 nid=0x22aa waiting on condition
> [0xb4428000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at com.cloudera.util.Clock$DefaultClock.doSleep(Clock.java:62)
>     at com.cloudera.util.Clock.sleep(Clock.java:88)
>     at
> com.cloudera.flume.reporter.MasterReportPusher$PusherThread.run(MasterReportPusher.java:118)
>
> "Timer-0" daemon prio=10 tid=0x09f46c00 nid=0x22a9 in Object.wait()
> [0xb4489000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x844a7c88> (a java.util.TaskQueue)
>     at java.util.TimerThread.mainLoop(Timer.java:509)
>     - locked <0x844a7c88> (a java.util.TaskQueue)
>     at java.util.TimerThread.run(Timer.java:462)
>
> "26392982@qtp-15532856-1 - Acceptor0 SelectChannelConnector@0.0.0.0:35862"
> prio=10 tid=0x09f0e800 nid=0x22a8 runnable [0xb44da000]
>    java.lang.Thread.State: RUNNABLE
>     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     - locked <0x84567428> (a sun.nio.ch.Util$2)
>     - locked <0x84567438> (a java.util.Collections$UnmodifiableSet)
>     - locked <0x845673e8> (a sun.nio.ch.EPollSelectorImpl)
>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     at
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:484)
>     at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
>     at
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
>     at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
> "11423854@qtp-15532856-0" prio=10 tid=0x09f03800 nid=0x22a7 in
> Object.wait() [0xb452b000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x8446c658> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>     at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:625)
>     - locked <0x8446c658> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
> "Low Memory Detector" daemon prio=10 tid=0x09bef000 nid=0x229a runnable
> [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x09bed000 nid=0x2299 waiting on
> condition [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x09beb800 nid=0x2298 waiting on
> condition [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x09be4c00 nid=0x2297 in Object.wait()
> [0xb486a000]
>    java.lang.Thread.State: WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>     - locked <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x09be3400 nid=0x2296 in
> Object.wait() [0xb48bc000]
>    java.lang.Thread.State: WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x844002e8> (a java.lang.ref.Reference$Lock)
>     at java.lang.Object.wait(Object.java:485)
>     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>     - locked <0x844002e8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x09bd9000 nid=0x2295 runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x09bfac00 nid=0x229b waiting on
> condition
>
> JNI global references: 1108
>
> Heap
>  def new generation   total 5760K, used 5180K [0x7eeb0000, 0x7f4e0000,
> 0x84400000)
>   eden space 5184K,  98% used [0x7eeb0000, 0x7f3b0800, 0x7f3c0000)
>   from space 576K,  10% used [0x7f450000, 0x7f45e9b0, 0x7f4e0000)
>   to   space 576K,   0% used [0x7f3c0000, 0x7f3c0000, 0x7f450000)
>  tenured generation   total 12480K, used 7936K [0x84400000, 0x85030000,
> 0x8eeb0000)
>    the space 12480K,  63% used [0x84400000, 0x84bc02c8, 0x84bc0400,
> 0x85030000)
>  compacting perm gen  total 13056K, used 12836K [0x8eeb0000, 0x8fb70000,
> 0x92eb0000)
>    the space 13056K,  98% used [0x8eeb0000, 0x8fb39030, 0x8fb39200,
> 0x8fb70000)
>     ro space 10240K,  61% used [0x92eb0000, 0x934d8a38, 0x934d8c00,
> 0x938b0000)
>     rw space 12288K,  60% used [0x938b0000, 0x93fe8ec0, 0x93fe9000,
> 0x944b0000)
>
> 2011-07-13 17:11:06
> Full thread dump Java HotSpot(TM) Client VM (19.1-b02 mixed mode, sharing):
>
> "SpawningLogicalNode ObservationConsumer-1" prio=10 tid=0xb4115400
> nid=0x8b0 waiting for monitor entry [0xb40ad000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>     at
> com.cloudera.flume.agent.LogicalNode.openLoadNode(LogicalNode.java:166)
>     - waiting to lock <0x848ced80> (a com.cloudera.flume.agent.LogicalNode)
>     at
> com.cloudera.flume.agent.LogicalNode.loadConfig(LogicalNode.java:328)
>     at
> com.cloudera.flume.agent.LogicalNode.checkConfig(LogicalNode.java:371)
>     at
> com.cloudera.flume.agent.LivenessManager$1.run(LivenessManager.java:141)
>
> "SpawningLogicalNode ObservationConsumer-1" prio=10 tid=0xb4119400
> nid=0x8ae waiting for monitor entry [0xb428c000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>     at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:169)
>     - waiting to lock <0x848d02a0> (a
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.close(AckChecksumChecker.java:192)
>     at
> com.cloudera.flume.collector.CollectorSink.close(CollectorSink.java:223)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.debug.LazyOpenDecorator.close(LazyOpenDecorator.java:81)
>     at
> com.cloudera.flume.agent.LogicalNode.openSourceSink(LogicalNode.java:128)
>     at
> com.cloudera.flume.agent.LogicalNode.openLoadNode(LogicalNode.java:169)
>     - locked <0x848ced80> (a com.cloudera.flume.agent.LogicalNode)
>     at
> com.cloudera.flume.agent.LogicalNode.loadConfig(LogicalNode.java:328)
>     at
> com.cloudera.flume.agent.LogicalNode.checkConfig(LogicalNode.java:371)
>     at
> com.cloudera.flume.agent.LivenessManager$1.run(LivenessManager.java:141)
>
> "pool-5-thread-2" prio=10 tid=0x0a0b5800 nid=0x8ad waiting on condition
> [0xb40fe000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x848cf678> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
>     at
> com.cloudera.flume.handlers.thrift.ThriftEventSource.enqueue(ThriftEventSource.java:117)
>     at
> com.cloudera.flume.handlers.thrift.ThriftEventSource$1.append(ThriftEventSource.java:135)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServerImpl.append(ThriftFlumeEventServerImpl.java:45)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:381)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:361)
>     at
> org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
> "pool-5-thread-1" prio=10 tid=0x09ce6800 nid=0xdc2 waiting on condition
> [0xb4335000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x848cf678> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
>     at
> com.cloudera.flume.handlers.thrift.ThriftEventSource.enqueue(ThriftEventSource.java:117)
>     at
> com.cloudera.flume.handlers.thrift.ThriftEventSource$1.append(ThriftEventSource.java:135)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServerImpl.append(ThriftFlumeEventServerImpl.java:45)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor$append.process(ThriftFlumeEventServer.java:381)
>     at
> com.cloudera.flume.handlers.thrift.ThriftFlumeEventServer$Processor.process(ThriftFlumeEventServer.java:361)
>     at
> org.apache.thrift.server.TSaneThreadPoolServer$WorkerProcess.run(TSaneThreadPoolServer.java:280)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
> "logicalNode ObservationConsumer-1-846590" prio=10 tid=0x09c24400 nid=0xdb4
> waiting on condition [0xb42dd000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x848d08a0> (a
> java.util.concurrent.CountDownLatch$Sync)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>     at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>     at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
>     at
> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:213)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:170)
>     - locked <0x848d02a0> (a
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>     at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>     at
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:108)
>     at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>     at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>     at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:217)
>     at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>     at
> com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:75)
>     at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:93)
>
> "MultiThreadedHttpConnectionManager cleanup" daemon prio=10 tid=0x0a0af800
> nid=0x1acd in Object.wait() [0xb3fba000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>     - locked <0x84589df0> (a java.lang.ref.ReferenceQueue$Lock)
>     at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)
>
> "DestroyJavaVM" prio=10 tid=0x09bb5800 nid=0x2292 waiting on condition
> [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "ChokeManager" daemon prio=10 tid=0x09f97400 nid=0x22ac waiting on
> condition [0xb4386000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>     at java.lang.Thread.sleep(Native Method)
>     at
> com.cloudera.flume.handlers.debug.ChokeManager.run(ChokeManager.java:143)
>
> "Heartbeat" prio=10 tid=0x09f96400 nid=0x22ab waiting for monitor entry
> [0xb43d7000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>     at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:169)
>     - waiting to lock <0x848d02a0> (a
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.close(AckChecksumChecker.java:192)
>     at
> com.cloudera.flume.collector.CollectorSink.close(CollectorSink.java:223)
>     at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>     at
> com.cloudera.flume.handlers.debug.LazyOpenDecorator.close(LazyOpenDecorator.java:81)
>     at com.cloudera.flume.agent.LogicalNode.close(LogicalNode.java:469)
>     at
> com.cloudera.flume.agent.LogicalNodeManager.decommission(LogicalNodeManager.java:131)
>     - locked <0x84448f60> (a com.cloudera.flume.agent.LogicalNodeManager)
>     at
> com.cloudera.flume.agent.LogicalNodeManager.decommissionAllBut(LogicalNodeManager.java:185)
>     - locked <0x84448f60> (a com.cloudera.flume.agent.LogicalNodeManager)
>     at
> com.cloudera.flume.agent.LivenessManager.checkLogicalNodes(LivenessManager.java:119)
>     at
> com.cloudera.flume.agent.LivenessManager.heartbeatChecks(LivenessManager.java:154)
>     at
> com.cloudera.flume.agent.LivenessManager$HeartbeatThread.run(LivenessManager.java:184)
>
> "Thread-1" prio=10 tid=0x09f95800 nid=0x22aa waiting for monitor entry
> [0xb4428000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>     at com.cloudera.flume.agent.LogicalNode.getReport(LogicalNode.java:395)
>     - waiting to lock <0x848ced80> (a com.cloudera.flume.agent.LogicalNode)
>     at
> com.cloudera.flume.agent.LogicalNodeManager.getReport(LogicalNodeManager.java:139)
>     at com.cloudera.flume.agent.FlumeNode.getReport(FlumeNode.java:824)
>     at com.cloudera.flume.agent.FlumeNode$1.getMetrics(FlumeNode.java:257)
>     at
> com.cloudera.flume.reporter.MasterReportPusher$PusherThread.queryReportMan(MasterReportPusher.java:94)
>     at
> com.cloudera.flume.reporter.MasterReportPusher$PusherThread.sendReports(MasterReportPusher.java:109)
>     at
> com.cloudera.flume.reporter.MasterReportPusher$PusherThread.run(MasterReportPusher.java:119)
>
> "Timer-0" daemon prio=10 tid=0x09f46c00 nid=0x22a9 in Object.wait()
> [0xb4489000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x844a7c88> (a java.util.TaskQueue)
>     at java.util.TimerThread.mainLoop(Timer.java:509)
>     - locked <0x844a7c88> (a java.util.TaskQueue)
>     at java.util.TimerThread.run(Timer.java:462)
>
> "26392982@qtp-15532856-1 - Acceptor0 SelectChannelConnector@0.0.0.0:35862"
> prio=10 tid=0x09f0e800 nid=0x22a8 runnable [0xb44da000]
>    java.lang.Thread.State: RUNNABLE
>     at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     - locked <0x84567428> (a sun.nio.ch.Util$2)
>     - locked <0x84567438> (a java.util.Collections$UnmodifiableSet)
>     - locked <0x845673e8> (a sun.nio.ch.EPollSelectorImpl)
>     at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     at
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:484)
>     at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:191)
>     at
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
>     at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
> "11423854@qtp-15532856-0" prio=10 tid=0x09f03800 nid=0x22a7 in
> Object.wait() [0xb452b000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x8446c658> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>     at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:625)
>     - locked <0x8446c658> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
> "Low Memory Detector" daemon prio=10 tid=0x09bef000 nid=0x229a runnable
> [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x09bed000 nid=0x2299 waiting on
> condition [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x09beb800 nid=0x2298 waiting on
> condition [0x00000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x09be4c00 nid=0x2297 in Object.wait()
> [0xb486a000]
>    java.lang.Thread.State: WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>     - locked <0x84400258> (a java.lang.ref.ReferenceQueue$Lock)
>     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x09be3400 nid=0x2296 in
> Object.wait() [0xb48bc000]
>    java.lang.Thread.State: WAITING (on object monitor)
>     at java.lang.Object.wait(Native Method)
>     - waiting on <0x844002e8> (a java.lang.ref.Reference$Lock)
>     at java.lang.Object.wait(Object.java:485)
>     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>     - locked <0x844002e8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x09bd9000 nid=0x2295 runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x09bfac00 nid=0x229b waiting on
> condition
>
> JNI global references: 1115
>
> Heap
>  def new generation   total 5760K, used 3459K [0x7eeb0000, 0x7f4e0000,
> 0x84400000)
>   eden space 5184K,  65% used [0x7eeb0000, 0x7f204b68, 0x7f3c0000)
>   from space 576K,   8% used [0x7f450000, 0x7f45c0e8, 0x7f4e0000)
>   to   space 576K,   0% used [0x7f3c0000, 0x7f3c0000, 0x7f450000)
>  tenured generation   total 12480K, used 7936K [0x84400000, 0x85030000,
> 0x8eeb0000)
>    the space 12480K,  63% used [0x84400000, 0x84bc02c8, 0x84bc0400,
> 0x85030000)
>  compacting perm gen  total 13056K, used 12836K [0x8eeb0000, 0x8fb70000,
> 0x92eb0000)
>    the space 13056K,  98% used [0x8eeb0000, 0x8fb39178, 0x8fb39200,
> 0x8fb70000)
>     ro space 10240K,  61% used [0x92eb0000, 0x934d8a38, 0x934d8c00,
> 0x938b0000)
>     rw space 12288K,  60% used [0x938b0000, 0x93fe8ec0, 0x93fe9000,
> 0x944b0000)
>
>

Mime
View raw message