flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Layland <stephen.layl...@gmail.com>
Subject Weird interrupted exception in DirectDriver during append
Date Mon, 17 Oct 2011 17:48:17 GMT
Hi, after letting flume idle for the weekend listening in on a small stream
of live data, we noticed several of our flume collector nodes failing with
InterruptedException's being thrown.  Logs have errors that look something
like below.  Any idea of what's going on here and how to fix it?

2011-10-17 08:02:02,107 INFO
> com.cloudera.flume.handlers.debug.StubbornAppendSink: append Interrupted
> event 'flume-node.lindenlab.com' [INFO Mon Oct 17 08:01:59 UTC 2011] {
> syslogfacility : 16 } { syslogseverity : 6 } Oct 17 01:01:59 SOME MESSAGE'
> with error: Blocked append interrupted by rotation event
> 2011-10-17 08:02:02,107 INFO com.cloudera.flume.handlers.rolling.RollSink:
> closing RollSink
> 'escapedCustomDfs("hdfs://master-node:54310/logs/raw/%Y/%m/%d/%H00","test%{rolltag}"
> )'
> 2011-10-17 08:02:02,109 ERROR
> com.cloudera.flume.core.connector.DirectDriver: Closing down due to
> exception during append calls
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223)
>         at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976)
>         at
> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
>         at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
>         at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
> 2011-10-17 08:02:02,109 INFO
> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode
> node6-22 exited with error: nulljava.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223)
>
>         at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976)
>         at
> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
>         at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
>         at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)
> 2011-10-17 08:02:02,109 INFO
> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode
> node6-22 exited with error: null
> java.lang.InterruptedException
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223)
>         at
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976)
>         at
> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175)
>         at
> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67)
>         at
> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62)
>         at
> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60)
>         at
> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81)
>         at
> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222)
>         at
> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110)


Many thanks,

-Steve

Mime
View raw message