flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan <jonny2...@gmail.com>
Subject Questions regarding flume tail
Date Wed, 07 Sep 2011 17:32:19 GMT

I have been reading through the source of Flume's tail implementation and I
have some questions that I was not able to figure out on my own.

1. What are tail's actions when it reaches the end of a file. Is it
flume's responsibility to delete the file upon completion or is that the
responsibility of the program generating the logs?
2. In the scenario the program generating the log files is doing some sort
of file rotation, at an interval long enough that tail is able to see the
file change. Is there some way of knowing how long flume takes to process
the file?

If it helps the use case is that we have a program generating a bunch of
different custom log files and we want some way to move the to HDFS and make
sure that we don't fill up local storage without losing any infomation.


View raw message