I have been experiencing a similar error and notice it only happens when I have a large number of files open to hdfs. I am running some tests removing the bucketing to see if I can send the same files. I will let you know if I come across anything. I have around 50 nodes writing to a single collector. This error consistently happens within 10 mins of starting my collector. My exception. 2011-10-17 17:30:07,173 [Roll-TriggerThread-0] INFO com.cloudera.flume.handlers.hdfs.CustomDfsSink - done writing raw file to hdfs 2011-10-17 17:30:07,189 [logicalNode collector0_log_dir-19] ERROR com.cloudera.flume.core.connector.DirectDriver - Closing down due to exception during append calls 2011-10-17 17:30:07,190 [logicalNode collector0_log_dir-19] INFO com.cloudera.flume.core.connector.DirectDriver - Connector logicalNode collector0_log_dir-19 exited with error: Blocked append interrupted by rotation event java.lang.InterruptedException: Blocked append interrupted by rotation event at com.cloudera.flume.handlers.rolling.RollSink.append(RollSink.java:209) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.core.MaskDecorator.append(MaskDecorator.java:43) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:169) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:71) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) at com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.core.extractors.DateExtractor.append(DateExtractor.java:129) at com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) at com.cloudera.flume.core.extractors.RegexExtractor.append(RegexExtractor.java:88) at com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:133) 2011-10-17 17:30:07,191 [logicalNode collector0_log_dir-19] INFO com.cloudera.flume.collector.CollectorSource - closed 2011-10-17 17:30:08,191 [logicalNode collector0_log_dir-19] INFO com.cloudera.flume.handlers.thrift.ThriftEventSource - Closed server on port 36892... 2011-10-17 17:30:08,191 [logicalNode collector0_log_dir-19] INFO com.cloudera.flume.handlers.thrift.ThriftEventSource - Queue still has 1000 elements ... 2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] WARN com.cloudera.flume.handlers.thrift.ThriftEventSource - Close timed out due to no progress. Closing despite having 1000 values still enqueued 2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] INFO com.cloudera.flume.handlers.rolling.RollSink - closing RollSink 'escapedCustomDfs("hdfs:// van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00","raw-%{rolltag}" )' 2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] INFO com.cloudera.flume.handlers.rolling.RollSink - double close 'escapedCustomDfs("hdfs:// van-mang-perf-hadoop-namenode1.net:8020/rawLogs/%{dateyear}-%{datemonth}-%{dateday}/%{datehr}00","raw-%{rolltag}" )' 2011-10-17 17:30:18,200 [logicalNode collector0_log_dir-19] ERROR com.cloudera.flume.core.connector.DirectDriver - Exiting driver logicalNode collector0_log_dir-19 in error state CollectorSource | RegexExtractor because Blocked append interrupted by rotation event On Mon, Oct 17, 2011 at 12:47 PM, Stephen Layland wrote: > Hi, we're not actually using flume nodes, but just the collector at the > moment. We're listening on a syslog port and dumping straight to HDFS for > now. After some digging, I'm pretty sure it's related to this: > > https://issues.apache.org/jira/browse/FLUME-757 > > -Steve > > > On Mon, Oct 17, 2011 at 11:53 AM, AD wrote: > >> weird i have been seeing the same thing. Do you have the node and >> collector on different hosts? Are you using hbase by chance? >> >> >> On Mon, Oct 17, 2011 at 1:48 PM, Stephen Layland < >> stephen.layland@gmail.com> wrote: >> >>> Hi, after letting flume idle for the weekend listening in on a small >>> stream of live data, we noticed several of our flume collector nodes failing >>> with InterruptedException's being thrown. Logs have errors that look >>> something like below. Any idea of what's going on here and how to fix it? >>> >>> 2011-10-17 08:02:02,107 INFO >>>> com.cloudera.flume.handlers.debug.StubbornAppendSink: append Interrupted >>>> event 'flume-node.lindenlab.com' [INFO Mon Oct 17 08:01:59 UTC 2011] { >>>> syslogfacility : 16 } { syslogseverity : 6 } Oct 17 01:01:59 SOME MESSAGE' >>>> with error: Blocked append interrupted by rotation event >>>> 2011-10-17 08:02:02,107 INFO >>>> com.cloudera.flume.handlers.rolling.RollSink: closing RollSink >>>> 'escapedCustomDfs("hdfs://master-node:54310/logs/raw/%Y/%m/%d/%H00","test%{rolltag}" >>>> )' >>>> 2011-10-17 08:02:02,109 ERROR >>>> com.cloudera.flume.core.connector.DirectDriver: Closing down due to >>>> exception during append calls >>>> java.lang.InterruptedException >>>> at >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >>>> at >>>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >>>> at >>>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >>>> at >>>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >>>> at >>>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >>>> 2011-10-17 08:02:02,109 INFO >>>> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode >>>> node6-22 exited with error: nulljava.lang.InterruptedException >>>> at >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >>>> >>>> at >>>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >>>> at >>>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >>>> at >>>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >>>> at >>>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >>>> 2011-10-17 08:02:02,109 INFO >>>> com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode >>>> node6-22 exited with error: null >>>> java.lang.InterruptedException >>>> at >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1223) >>>> at >>>> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:976) >>>> at >>>> com.cloudera.flume.handlers.rolling.RollSink.close(RollSink.java:296) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.handlers.debug.InsistentOpenDecorator.close(InsistentOpenDecorator.java:175) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.close(EventSinkDecorator.java:67) >>>> at >>>> com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:78) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.debug.InsistentAppendDecorator.append(InsistentAppendDecorator.java:110) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.endtoend.AckChecksumChecker.append(AckChecksumChecker.java:113) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.batch.UnbatchingDecorator.append(UnbatchingDecorator.java:62) >>>> at >>>> com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:60) >>>> at >>>> com.cloudera.flume.handlers.batch.GunzipDecorator.append(GunzipDecorator.java:81) >>>> at >>>> com.cloudera.flume.collector.CollectorSink.append(CollectorSink.java:222) >>>> at >>>> com.cloudera.flume.core.connector.DirectDriver$PumperThread.run(DirectDriver.java:110) >>> >>> >>> Many thanks, >>> >>> -Steve >>> >> >> > -- Thanks Cameron Gandevia