flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sverre Bakke <sverre.ba...@gmail.com>
Subject Re: Flume error handling
Date Wed, 12 Nov 2014 20:38:03 GMT

One thing I find weird about Flume deserializers are that the
resettableinputstream is incompatible with the Java standard library input
streams. Otherwise, this would probably be solved by 10 lines of code.

On Nov 10, 2014 8:27 PM, "Hari Shreedharan" <hshreedharan@cloudera.com>

> You could implement a deserializer that reads compressed files. You’d have
> to decompress the file and return the data event by event though. Take a
> look at something like the BlobSerializer as an example (it is a pretty
> simple one that demostrates the behavior).
> If you shutdown the executor which you started in the start() method, and
> you shut it down in the stop method, you probably will get an exception
> from the Executor framework complaining that the executor was shutdown and
> not a channel exception. If you get a channel exception it means that the
> channel is full or can’t accept events right now for some other reason —
> not that the source can’t handle it. If that is the case, the bug is in the
> source implementation. Do you see that behavior in a source bundled with
> flume?
> Thanks,
> Hari
> On Mon, Nov 10, 2014 at 4:01 AM, Sverre Bakke <sverre.bakke@gmail.com>
> wrote:
>> Hi,
>> When creating a new EventDrivenSource running as an executor, what is
>> the correct approach to handling shutdown gracefully?
>> I am writing a custom source that will poll a compressed file line by
>> line using BufferedReader and pushing these lines to a
>> ChannelProcessor using processEvent(). This as a result of Spooling
>> Directory not supporting compressed files. This also means that most
>> of the time, my Flume source will be blocking on
>> BufferedReader.readLine() or blocking on
>> ChannelProcessor.processEvent().
>> If I shutdown the executor from the stop() method of my source, the
>> typical response from Flume will be that the ChannelProcessor will
>> generate a ChannelException. In what situations can I expect that the
>> ChannelException actually is the result of a shutdown (e.g. ctrl+c)
>> rather than some other issue that should be handled as a truly
>> exceptional situation/error? Or am I approaching graceful shutdown
>> completely wrong?
>> Is there any specific order in which the Flume sources, interceptors
>> and sinks are signaled to shut down?
>> I feel that when it comes to error handling (and shutdowns), the
>> developer guide and javadoc is a bit lacking unfortunately.
>> Regards,
>> Sverre

View raw message