Steve - I appreciate you time on this...
Yes, I want to use flume to copy .xml or .whatever files from a server outside the cluster to hdfs. That server does l have flume installed on it
Id like the same behavior as "spooling directory" but from a remote machine --> to hdfs
So, from all my reading flume looks like it completely designed for streaming "live" logs and program outputs...
Doesn't seem to be known for being a filewatcher and grabbing files as they show up, then shiping and writing to hdfs
Of can it?
Ok I can think fragmentation with individual "small" files but doesn't "spool directory behaviour" face the same issue?
I've done quite a bit of reading but one can easily get into the weeds :) - All I need to do is this simple task.