flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Feng Longda <hustjac...@gmail.com>
Subject Several taildir issues.
Date Fri, 29 Jun 2018 08:50:14 GMT
Hi,



Recently, I want one tail logs tools, I found Flume is the right
tools. But when I study taildir source code. I found several issues.



1.  TailFile.readEvent just read one line every time, but in fact it
read BUFFER_SIZE, Sometimes it contain multiple lines, but the next
line will be read in the next time, it waste time.

2.  If there are huge files in the monitored directory, it will cost
much time on scanning directory every time when new file is coming or
one file has been removed, at the same time updating position to
positionFile will cost much time too.

3.  Flume doesn’t support monitory subdirectory, for example
/home/hadoop/logs/*/*.log, it will occur exception in the
TaildirMatcher.getMatchingFilesNoCache.





Regards

Longda

Mime
View raw message