Hello there,

My flume seems to be getting stuck in the weird state and I'm not sure what to do about it.

Metrics endpoint looks like Flume is relatively happy, but I'm seeing the error below

curl localhost:5653/metrics
{"SINK.s3-sink":{"BatchCompleteCount":"0","ConnectionFailedCount":"0","EventDrainAttemptCount":"0","ConnectionCreatedCount":"0","Type":"SINK","BatchEmptyCount":"0","ConnectionClosedCount":"0","EventDrainSuccessCount":"0","StopTime":"0","StartTime":"1439133984193","BatchUnderflowCount":"0"},"SOURCE.spooling-directory":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"0","AppendBatchReceivedCount":"1","EventAcceptedCount":"0","AppendReceivedCount":"0"}

typekit@prod-polka-51145e82:/var/log/flume-ng$ ls -la /mnt/flume/data/log-1955.meta
-rw-r--r-- 1 root root 47 Jul 26 15:57 /mnt/flume/data/log-1955.meta

The error I see isĀ 

2015-08-09 15:31:24,236 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)] Unable to deliver event. Exception follows.
java.lang.IllegalStateException: Channel closed [channel=fileChannel]. Due to java.io.FileNotFoundException: /mnt/flume/data/log-1955.meta (Too many open files)
at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:340)
at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:368)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: /mnt/flume/data/log-1955.meta (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at org.apache.flume.channel.file.LogFileV3$SequentialReader.<init>(LogFileV3.java:318)
at org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:165)
at org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:264)
at org.apache.flume.channel.file.Log.doReplay(Log.java:519)
at org.apache.flume.channel.file.Log.replay(Log.java:445)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

Is /metrics basically useless to look at the health of Flume?

Please help.

-George