Thanks Jeff. This is useful. Can the spoolDir be on a different machine? We may have to setup a different process to copy files into 'spoolDir', right? Note: We have 'read only' access to these files. Any recommendations about this?
On Wed, Apr 16, 2014 at 5:16 PM, Jeff Lord <email@example.com> wrote:
On Wed, Apr 16, 2014 at 5:14 PM, Something Something <firstname.lastname@example.org> wrote:username@machinename:/var/log/logdir/<multiple files>Hello,Needless to say I am newbie to Flume, but I've got a basic flow working in which I am importing a log file from my linux box to hdfs. I am usinga1.sources.r1.command = tail -F /var/log/xyz.logwhich is working like a stream of messages. This is good!Now what I want to do is copy log files from a directory on a remote machine on a regular basis. For example:
One way to do it is to simply 'scp' files from the remote directory into my box on a regular basis, but what's the best way to do this in Flume? Please let me know.
Thanks for the help.