flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Westmacott <peter.westmac...@plumbee.co.uk>
Subject Problem decreasing File Channel capacity
Date Wed, 18 Nov 2015 11:06:10 GMT
Hello,

We’ve got a couple of flume agents running happily (1.5.0). We recently
decided to reduce the File Channel capacity (we had way more capacity than
we were regularly using and we have plenty of upstream buffering).

On one agent this was fine, we adjusted the configuration and restarted the
agent. Apart from occasional log messages telling us that the channel was
full, we had no issues. (If the sink slows down then we want to communicate
the back pressure quickly - so this is by design. Is there a way to tell
Flume not to worry about this?)

The other agent failed to restart. The error we saw was:

2015-10-05 10:37:27,169 [lifecycleSupervisor-1-0] ERROR
(org.apache.flume.channel.file.Log.replay:481)  - Failed to initialize Log
on [channel=ebs-1]

java.lang.IllegalStateException: Unable to add FlumeEventPointer
[fileID=39020, offset=44376807]. Queue depth = 3000, Capacity = 3000

        at
org.apache.flume.channel.file.ReplayHandler.processCommit(ReplayHandler.java:395)

        at
org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:335)

        at org.apache.flume.channel.file.Log.doReplay(Log.java:508)

        at org.apache.flume.channel.file.Log.replay(Log.java:462)

        at
org.apache.flume.channel.file.FileChannel.start(FileChannel.java:279)

...

We hypothesised that this was because replaying the Write-Ahead-Log caused
the Queue to exceed its new capacity. (This agent had higher throughput
than the first.) Our question is this: Is there a recommended approach to
reducing a FileChannel capacity with a low (near-zero) chance of error?

In the meantime we will attempt to work around this by draining the
channel, and waiting for a checkpoint to occur on the empty channel state
before reducing the channel capacity.

Any thoughts on any of this would be gratefully received. Apologies if we
missed a previous answer to any of this in the mailing list archives.

Peter Westmacott

Mime
View raw message