flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakati, Pavan" <pavan.jak...@cgi.com>
Subject RE: How to read content from files which are in nested directory
Date Wed, 22 Jul 2015 11:06:12 GMT
Hi Ashish ,

I used recursiveDirectorySearch in my conf and set it to true . Yet I am unable to read content
under subdirectory . 

Regards,
PavaN


-----Original Message-----
From: Jakati, Pavan 
Sent: Monday, July 20, 2015 6:08 PM
To: user@flume.apache.org
Subject: RE: How to read content from files which are in nested directory

Hi Ashish 

Thanks for the help. Can you pls help with procedure to update or pull the patch into my system
. Thanks. 

Regards,
PavaN


-----Original Message-----
From: Ashish [mailto:paliwalashish@gmail.com]
Sent: Saturday, July 18, 2015 12:44 AM
To: user@flume.apache.org
Subject: Re: How to read content from files which are in nested directory

One more way to achieve the same is using this patch
https://issues.apache.org/jira/browse/FLUME-1899

On Fri, Jul 17, 2015 at 7:14 AM, Johny Rufus <jrufus@cloudera.com> wrote:
> The spooling directory source as of now supports only reading from a 
> flat directory and wont read files from subdirectories.
> You could write an external script that transfers all the files in all 
> the date directories to a common directory which spooling source 
> points to. (If this fits your use case)
>
> Thanks,
> Rufus
>
> On Fri, Jul 17, 2015 at 6:40 AM, Jakati, Pavan <pavan.jakati@cgi.com> wrote:
>>
>> Hi Folks,
>>
>>
>>
>> I am consuming Apache flume to read logs which are stored in 
>> following format , logs under date directory . How do I read the logs 
>> using spooldir source.
>>
>>
>>
>>
>>
>> Directory Structure :
>>
>>
>>
>>
>>
>> flume]# ll /ADAPTORS/MAIL_CONNECT/logs/
>>
>> total 0
>>
>> drwxr-xr-x 2 root root 29 Jun  3 18:04 06-03-15
>>
>> drwxr-xr-x 2 root root 56 Jun  4 18:16 06-04-15
>>
>> drwxr-xr-x 2 root root 29 Jun  5 14:03 06-05-15
>>
>> drwxr-xr-x 2 root root 29 Jun  8 12:43 06-08-15
>>
>> drwxr-xr-x 2 root root 29 Jun  9 14:47 06-09-15
>>
>> drwxr-xr-x 2 root root 29 Jun 10 18:49 06-10-15
>>
>> drwxr-xr-x 2 root root 29 Jun 11 17:22 06-11-15
>>
>> drwxr-xr-x 2 root root 29 Jun 12 11:37 06-12-15
>>
>> drwxr-xr-x 2 root root 29 Jun 15 11:39 06-15-15
>>
>> drwxr-xr-x 2 root root 29 Jun 24 10:45 06-23-15
>>
>> drwxr-xr-x 2 root root 29 Jun 24 10:35 06-24-15
>>
>> drwxr-xr-x 2 root root 29 Jun 25 17:28 06-25-15
>>
>>
>>
>>
>>
>> My current Configuration :
>>
>> agent1.sources.source1.type = spooldir
>>
>> agent1.sources.source1.channels = channel1
>>
>> agent1.sources.source1.spoolDir =
>> /ADAPTORS/MAIL_CONNECT/logs/07-17-15
>>
>> agent1.sources.source1.fileHeader = true
>>
>>
>>
>>
>>
>> Pls suggest if i need to use some other source type .
>>
>>
>>
>> Can I use any script inside conf file to change the directory 
>> dynamically . Thanks
>>
>>
>>
>>
>>
>> Regards,
>>
>> PavaN
>>
>>
>
>



--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Mime
View raw message