flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Something Something <mailinglist...@gmail.com>
Subject Re: Import files from a directory on remote machine
Date Thu, 17 Apr 2014 00:26:26 GMT
Thanks Jeff.  This is useful.  Can the spoolDir be on a different machine?
We may have to setup a different process to copy files into 'spoolDir',
right?  Note:  We have 'read only' access to these files.  Any
recommendations about this?


On Wed, Apr 16, 2014 at 5:16 PM, Jeff Lord <jlord@cloudera.com> wrote:

> http://flume.apache.org/FlumeUserGuide.html#spooling-directory-source
>
>
> On Wed, Apr 16, 2014 at 5:14 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
>> Hello,
>>
>> Needless to say I am newbie to Flume, but I've got a basic flow working
>> in which I am importing a log file from my linux box to hdfs.  I am using
>>
>> a1.sources.r1.command = tail -F /var/log/xyz.log
>>
>> which is working like a stream of messages.  This is good!
>>
>> Now what I want to do is copy log files from a directory on a remote
>> machine on a regular basis.  For example:
>>
>> username@machinename:/var/log/logdir/<multiple files>
>>
>> One way to do it is to simply 'scp' files from the remote directory into
>> my box on a regular basis, but what's the best way to do this in Flume?
>> Please let me know.
>>
>> Thanks for the help.
>>
>>
>>
>

Mime
View raw message