The spooling directory source runs as part of the agent.
The source also needs write access to the files as it renames them upon completion of ingest. Perhaps you could use rsync to copy the files somewhere that you have write access to?

On Wed, Apr 16, 2014 at 5:26 PM, Something Something <> wrote:
Thanks Jeff.  This is useful.  Can the spoolDir be on a different machine?  We may have to setup a different process to copy files into 'spoolDir', right?  Note:  We have 'read only' access to these files.  Any recommendations about this?

On Wed, Apr 16, 2014 at 5:16 PM, Jeff Lord <> wrote:

On Wed, Apr 16, 2014 at 5:14 PM, Something Something <> wrote:

Needless to say I am newbie to Flume, but I've got a basic flow working in which I am importing a log file from my linux box to hdfs.  I am using

a1.sources.r1.command = tail -F /var/log/xyz.log

which is working like a stream of messages.  This is good!

Now what I want to do is copy log files from a directory on a remote machine on a regular basis.  For example:

username@machinename:/var/log/logdir/<multiple files>

One way to do it is to simply 'scp' files from the remote directory into my box on a regular basis, but what's the best way to do this in Flume?  Please let me know.

Thanks for the help.