flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lior Harel <harel.l...@gmail.com>
Subject Re: Questions regarding flume tail
Date Wed, 07 Sep 2011 19:00:04 GMT
your use case is the classic one. flume can handle that.
1. when flume reaches the end of file it just blocks, waiting for new log
events to be written. this is similar to the behavior of tail -f (i.e.
follow)
2. flume doesn't process the file, it watches the file for new log lines,
those lines will be sent as they happen. you just need to tail the main
file.

On Wed, Sep 7, 2011 at 8:32 PM, Jonathan <jonny2112@gmail.com> wrote:

> Hi,
>
> I have been reading through the source of Flume's tail implementation and I
> have some questions that I was not able to figure out on my own.
>
> 1. What are tail's actions when it reaches the end of a file. Is it
> flume's responsibility to delete the file upon completion or is that the
> responsibility of the program generating the logs?
> 2. In the scenario the program generating the log files is doing some sort
> of file rotation, at an interval long enough that tail is able to see the
> file change. Is there some way of knowing how long flume takes to process
> the file?
>
> If it helps the use case is that we have a program generating a bunch of
> different custom log files and we want some way to move the to HDFS and make
> sure that we don't fill up local storage without losing any infomation.
>
> Jonathan
>

Mime
View raw message