flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: Unable to deliver event. Exception follows.
Date Mon, 10 Aug 2015 17:28:13 GMT
Looks like the either the file limit is set to too low or some file
descriptor leakage. Assuming former, try standard *nix tools like
ulimit and lsof to find open fd and if needed increase the ulimit to a
suitable value.

HTH !

On Sun, Aug 9, 2015 at 8:36 AM, George Blazer <gblazer@gmail.com> wrote:
> Hello there,
>
> My flume seems to be getting stuck in the weird state and I'm not sure what
> to do about it.
>
> Metrics endpoint looks like Flume is relatively happy, but I'm seeing the
> error below
>
> curl localhost:5653/metrics
> {"SINK.s3-sink":{"BatchCompleteCount":"0","ConnectionFailedCount":"0","EventDrainAttemptCount":"0","ConnectionCreatedCount":"0","Type":"SINK","BatchEmptyCount":"0","ConnectionClosedCount":"0","EventDrainSuccessCount":"0","StopTime":"0","StartTime":"1439133984193","BatchUnderflowCount":"0"},"SOURCE.spooling-directory":{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"0","AppendBatchReceivedCount":"1","EventAcceptedCount":"0","AppendReceivedCount":"0"}
>
> typekit@prod-polka-51145e82:/var/log/flume-ng$ ls -la
> /mnt/flume/data/log-1955.meta
> -rw-r--r-- 1 root root 47 Jul 26 15:57 /mnt/flume/data/log-1955.meta
>
> The error I see is
>
> 2015-08-09 15:31:24,236 (SinkRunner-PollingRunner-DefaultSinkProcessor)
> [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:160)]
> Unable to deliver event. Exception follows.
> java.lang.IllegalStateException: Channel closed [channel=fileChannel]. Due
> to java.io.FileNotFoundException: /mnt/flume/data/log-1955.meta (Too many
> open files)
> at
> org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:340)
> at
> org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
> at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:368)
> at
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: /mnt/flume/data/log-1955.meta (Too
> many open files)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:146)
> at
> org.apache.flume.channel.file.LogFileV3$SequentialReader.<init>(LogFileV3.java:318)
> at
> org.apache.flume.channel.file.LogFileFactory.getSequentialReader(LogFileFactory.java:165)
> at
> org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:264)
> at org.apache.flume.channel.file.Log.doReplay(Log.java:519)
> at org.apache.flume.channel.file.Log.replay(Log.java:445)
> at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:290)
> at
> org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> Is /metrics basically useless to look at the health of Flume?
>
> Please help.
>
> -George
>
>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message