flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: How to read content from files which are in nested directory
Date Wed, 22 Jul 2015 20:21:01 GMT
Did you applied the patch? If yes, then probably you need to debug
what's going on. Start with some logs.

On Wed, Jul 22, 2015 at 4:06 AM, Jakati, Pavan <pavan.jakati@cgi.com> wrote:
> Hi Ashish ,
>
> I used recursiveDirectorySearch in my conf and set it to true . Yet I am unable to read
content under subdirectory .
>
> Regards,
> PavaN
>
>
> -----Original Message-----
> From: Jakati, Pavan
> Sent: Monday, July 20, 2015 6:08 PM
> To: user@flume.apache.org
> Subject: RE: How to read content from files which are in nested directory
>
> Hi Ashish
>
> Thanks for the help. Can you pls help with procedure to update or pull the patch into
my system . Thanks.
>
> Regards,
> PavaN
>
>
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com]
> Sent: Saturday, July 18, 2015 12:44 AM
> To: user@flume.apache.org
> Subject: Re: How to read content from files which are in nested directory
>
> One more way to achieve the same is using this patch
> https://issues.apache.org/jira/browse/FLUME-1899
>
> On Fri, Jul 17, 2015 at 7:14 AM, Johny Rufus <jrufus@cloudera.com> wrote:
>> The spooling directory source as of now supports only reading from a
>> flat directory and wont read files from subdirectories.
>> You could write an external script that transfers all the files in all
>> the date directories to a common directory which spooling source
>> points to. (If this fits your use case)
>>
>> Thanks,
>> Rufus
>>
>> On Fri, Jul 17, 2015 at 6:40 AM, Jakati, Pavan <pavan.jakati@cgi.com> wrote:
>>>
>>> Hi Folks,
>>>
>>>
>>>
>>> I am consuming Apache flume to read logs which are stored in
>>> following format , logs under date directory . How do I read the logs
>>> using spooldir source.
>>>
>>>
>>>
>>>
>>>
>>> Directory Structure :
>>>
>>>
>>>
>>>
>>>
>>> flume]# ll /ADAPTORS/MAIL_CONNECT/logs/
>>>
>>> total 0
>>>
>>> drwxr-xr-x 2 root root 29 Jun  3 18:04 06-03-15
>>>
>>> drwxr-xr-x 2 root root 56 Jun  4 18:16 06-04-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun  5 14:03 06-05-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun  8 12:43 06-08-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun  9 14:47 06-09-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun 10 18:49 06-10-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun 11 17:22 06-11-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun 12 11:37 06-12-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun 15 11:39 06-15-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun 24 10:45 06-23-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun 24 10:35 06-24-15
>>>
>>> drwxr-xr-x 2 root root 29 Jun 25 17:28 06-25-15
>>>
>>>
>>>
>>>
>>>
>>> My current Configuration :
>>>
>>> agent1.sources.source1.type = spooldir
>>>
>>> agent1.sources.source1.channels = channel1
>>>
>>> agent1.sources.source1.spoolDir =
>>> /ADAPTORS/MAIL_CONNECT/logs/07-17-15
>>>
>>> agent1.sources.source1.fileHeader = true
>>>
>>>
>>>
>>>
>>>
>>> Pls suggest if i need to use some other source type .
>>>
>>>
>>>
>>> Can I use any script inside conf file to change the directory
>>> dynamically . Thanks
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>>
>>> PavaN
>>>
>>>
>>
>>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message