flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Søren ...@syntonetic.com>
Subject fix for hanging collector node
Date Tue, 24 Jan 2012 10:12:11 GMT
Dear flume community

We have encountered a serious problem in our collector node (flume 
0.9.4). After browsing the mail group it seems to be a well known issue:
..........
2012-01-23 15:26:31,611 INFO 
com.cloudera.flume.handlers.debug.StubbornAppendSink: append Interrupted 
event '[eventcontent]' with error: Blocked append interrupted by 
rotation event
2012-01-23 15:26:31,612 INFO 
com.cloudera.flume.handlers.rolling.RollSink: closing RollSink 
'escapedCustomDfs("s3n://flume-log/eventlog/","el-%{rolltag}" )'
2012-01-23 15:26:31,612 ERROR 
com.cloudera.flume.core.connector.DirectDriver: Closing down due to 
exception during append calls
2012-01-23 15:26:31,612 INFO 
com.cloudera.flume.core.connector.DirectDriver: Connector logicalNode 
CollectorEventlog-22 exited with error: null
.........

We are using S3 sink which supports the theory that the the node is 
sensitive to timeout or similar from the sink.
The bug should be solved with:
+ patch FLUME-762
+ patch FLUME-798
+ setting flume.collector.roll.timeout to 0

Do I get it right so far?

We have installed flume via apt, and haven't dealed with other than the 
binary build of flume until now.
Are there updated binaries out there?
If not, what is the easiest and safest way to apply those patches?

Thanks for a great product anyway. We are looking forward to flume-ng 
with support for thrift source/S3 sink.

Thanks in advance
Søren


Mime
View raw message