flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ferenc Szabo <szabofe...@apache.org>
Subject Re: Flume spooldir source- subdirectory issue
Date Fri, 20 Apr 2018 10:07:35 GMT
Hi Jyotsna,

I guess this issue is solved by the custom interceptor mentioned in your
other mail.

let me know if You need any other help.

Regards,
Ferenc

On Thu, Apr 19, 2018 at 5:57 AM, Jyotsna G <jyotsna.g@gmail.com> wrote:

> Hi All,
>
> I am currently using Flume 1.7 . Configured a spooling directory source. I
> have enabled recursiveDirectorySearch=true to look in to the sub
> directories for files.
>
> source.spoolDir=/tmp/test
>
> and under /tmp/test, subdirectories get created with data files
> /tmp/test/data1/file.csv , /tmp/test/data2/file2.csv .
>
> I want the exact sub directory structure to be created in the HDFS sink
> path.
>
> /sink/data1/file.csv
>
>  /sink/data2/file2.csv
>
> When i use the %{file} for HDFS sink filepath, i get the complete absolute
> path, and %{basename} gives me only the file name. I want to extract the
> sub directory structure from the spooldir source path. Any way to achieve
> this?
> Thanks ,
> Jyotsna
>

Mime
View raw message