flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yatchmenoff, Sam" <syatchmen...@janrain.com>
Subject Exceptions after reloading configuration
Date Wed, 16 Jan 2013 20:41:34 GMT
I have Flume 1.2.0 running in a production system with 3 collectors fed by
~30 agents running on our application servers. If I make a change to the
node configuration on the collectors, when the configuration is reloaded
automatically, the collectors will occasionally fail and repeatedly report
the following exception:

2013-01-16 20:31:22,353 ERROR flume.SinkRunner: Unable to deliver event.
Exception follows.
org.apache.flume.EventDeliveryException: Failed to process transaction
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:218)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:297)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
at
org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:195)
... 3 more

After about a dozen of those, I will start seeing this exception:

2013-01-16 20:32:27,374 ERROR flume.SinkRunner: Unable to deliver event.
Exception follows.
org.apache.flume.EventDeliveryException: Unable to rotate file
/mnt/rawlog/1358365369665-49 while delivering event
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:155)
at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.io.IOException: Stream Closed
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:297)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.flume.sink.RollingFileSink.process(RollingFileSink.java:149)
... 3 more


Here is configuration for that agent:

agent1.sources = source1 source2
agent1.sinks = sink1
agent1.channels = channel1

# Describe/configure source1
agent1.sources.source1.type = avro
agent1.sources.source1.bind = 0.0.0.0
agent1.sources.source1.port = 35853

agent1.sources.source2.type = netcat
agent1.sources.source2.bind = localhost
agent1.sources.source2.port = 35854
agent1.sources.source2.max-line-length = 524288

# Describe sink1
agent1.sinks.sink1.type = FILE_ROLL
agent1.sinks.sink1.sink.directory = /mnt/rawlog
agent1.sinks.sink1.sink.rollInterval = 60

# Use a channel which buffers events in memory
agent1.channels.channel1.type = file
agent1.channels.channel1.checkpointDir =
/mnt/flume-ng/file-channel1/checkpoint
agent1.channels.channel1.dataDirs = /mnt/flume-ng/file-channel1/data
agent1.channels.channel1.capacity = 100000

# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sources.source2.channels = channel1
agent1.sinks.sink1.channel = channel1

Any ideas about what's causing this exception would be greatly appreciated.

Mime
View raw message