flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jakati, Pavan" <pavan.jak...@cgi.com>
Subject RE: How to read content from files which are in nested directory
Date Mon, 20 Jul 2015 12:38:43 GMT
Hi Johny


Thanks for the update. Can you pls let me know if I can add the script into my flume conf
and do the required actions. I do not wish to run script outside flume.

Regards,
PavaN

From: Johny Rufus [mailto:jrufus@cloudera.com]
Sent: Friday, July 17, 2015 7:45 PM
To: user@flume.apache.org
Subject: Re: How to read content from files which are in nested directory

The spooling directory source as of now supports only reading from a flat directory and wont
read files from subdirectories.
You could write an external script that transfers all the files in all the date directories
to a common directory which spooling source points to. (If this fits your use case)

Thanks,
Rufus

On Fri, Jul 17, 2015 at 6:40 AM, Jakati, Pavan <pavan.jakati@cgi.com<mailto:pavan.jakati@cgi.com>>
wrote:
Hi Folks,

I am consuming Apache flume to read logs which are stored in following format , logs under
date directory . How do I read the logs using spooldir source.


Directory Structure :


flume]# ll /ADAPTORS/MAIL_CONNECT/logs/
total 0
drwxr-xr-x 2 root root 29 Jun  3 18:04 06-03-15
drwxr-xr-x 2 root root 56 Jun  4 18:16 06-04-15
drwxr-xr-x 2 root root 29 Jun  5 14:03 06-05-15
drwxr-xr-x 2 root root 29 Jun  8 12:43 06-08-15
drwxr-xr-x 2 root root 29 Jun  9 14:47 06-09-15
drwxr-xr-x 2 root root 29 Jun 10 18:49 06-10-15
drwxr-xr-x 2 root root 29 Jun 11 17:22 06-11-15
drwxr-xr-x 2 root root 29 Jun 12 11:37 06-12-15
drwxr-xr-x 2 root root 29 Jun 15 11:39 06-15-15
drwxr-xr-x 2 root root 29 Jun 24 10:45 06-23-15
drwxr-xr-x 2 root root 29 Jun 24 10:35 06-24-15
drwxr-xr-x 2 root root 29 Jun 25 17:28 06-25-15


My current Configuration :
agent1.sources.source1.type = spooldir
agent1.sources.source1.channels = channel1
agent1.sources.source1.spoolDir = /ADAPTORS/MAIL_CONNECT/logs/07-17-15
agent1.sources.source1.fileHeader = true


Pls suggest if i need to use some other source type .

Can I use any script inside conf file to change the directory dynamically . Thanks


Regards,
PavaN


Mime
View raw message