flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sverre Bakke <sverre.ba...@gmail.com>
Subject Flume error handling
Date Mon, 10 Nov 2014 12:00:43 GMT

When creating a new EventDrivenSource running as an executor, what is
the correct approach to handling shutdown gracefully?

I am writing a custom source that will poll a compressed file line by
line using BufferedReader and pushing these lines to a
ChannelProcessor using processEvent(). This as a result of Spooling
Directory not supporting compressed files. This also means that most
of the time, my Flume source will be blocking on
BufferedReader.readLine() or blocking on

If I shutdown the executor from the stop() method of my source, the
typical response from Flume will be that the ChannelProcessor will
generate a ChannelException. In what situations can I expect that the
ChannelException actually is the result of a shutdown (e.g. ctrl+c)
rather than some other issue that should be handled as a truly
exceptional situation/error? Or am I approaching graceful shutdown
completely wrong?

Is there any specific order in which the Flume sources, interceptors
and sinks are signaled to shut down?

I feel that when it comes to error handling (and shutdowns), the
developer guide and javadoc is a bit lacking unfortunately.


View raw message