flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jyotsna G <jyotsn...@gmail.com>
Subject Flume spooldir source- subdirectory issue
Date Thu, 19 Apr 2018 03:57:02 GMT
Hi All,

I am currently using Flume 1.7 . Configured a spooling directory source. I
have enabled recursiveDirectorySearch=true to look in to the sub
directories for files.

source.spoolDir=/tmp/test

and under /tmp/test, subdirectories get created with data files
/tmp/test/data1/file.csv , /tmp/test/data2/file2.csv .

I want the exact sub directory structure to be created in the HDFS sink
path.

/sink/data1/file.csv

 /sink/data2/file2.csv

When i use the %{file} for HDFS sink filepath, i get the complete absolute
path, and %{basename} gives me only the file name. I want to extract the
sub directory structure from the spooldir source path. Any way to achieve
this?
Thanks ,
Jyotsna

Mime
View raw message