flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marina <ppi...@yahoo.com>
Subject handling of modified files in spooldir - similar to Flume-2052
Date Fri, 22 Jun 2018 16:33:10 GMT
Hi,I know that Flume-2052 fixed the issue with encountering invalid characters in the input
and ignoring them if so configured.I was wondering if the same could be done for the infamous
"java.lang.IllegalStateException: File has changed size since being read:" issue.Currently,
Flume just barfs when this happens, kills the processing thread but leaves the process itself
running. This makes it hard to detect this failure using usual process monitoring (the process
is up) - while the actual processing is dead. Also, if this happens, Flume does not just ignore
the offending file in the spooldir, but becomes a "zombie" forever .... until you somehow
detect this and restart it.

Are there any plans to make handling of this situation configurable, just as was done for
Flume-2052 issue?I think it would be a much more robust and user-friendly behavior if Flume
could report the offense, but ignore it and continue processing other good files if so configured.
Thank you!Marina

View raw message