flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sadananda Hegde <saduhe...@gmail.com>
Subject picking up new files in Flume NG
Date Tue, 16 Oct 2012 04:37:07 GMT

I have a scenario where in the client application is continuously pushing
xml messages. Actually the application is writing these messages to files
(new files; same directory). So we will be keep getting new files
throughout the day. I am trying to configure Flume agents on these
applcation servers (4 of them) to pick up the new data and transfer them to
HDFS on a hadoop cluster. How should I configure my source to pick up new
files (and exclude the files that have been processed already)? I don't
think Exec source with tail  -F will work in this scenario because data is
not getting added to existing files; rather new files get created.

Thank you very much for your time and support.


View raw message