flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mahendran m <mahendra...@hotmail.com>
Subject RE: Need help on spooling directory source
Date Tue, 18 Nov 2014 06:22:16 GMT
Hi Manohar ,
I am using the flume version 1.5.0.1
Thanks,Mahendran

> From: Manohar.CS@itcinfotech.com
> To: user@flume.apache.org
> Subject: RE: Need help on spooling directory source
> Date: Tue, 18 Nov 2014 06:05:19 +0000
> 
> Yes Ashish is right. Which flume version your are using?
> 
> Version 1.4.0 does not support recursive directory checks, so, if you store your logs
in a directory with a timestamp (i.e. /logs/project_name/2013/12/26), you will have to move
all files to a single directory (i.e. /logs/project_name).
> 
> 
> 
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com] 
> Sent: Tuesday, November 18, 2014 11:32 AM
> To: user@flume.apache.org
> Subject: Re: Need help on spooling directory source
> 
> AFAIK, it doesn't work recursively. It would pick files only from specified directory.
> 
> Here is the patch you can try https://issues.apache.org/jira/browse/FLUME-1899
> 
> HTH !
> 
> 
> On Tue, Nov 18, 2014 at 11:21 AM, mahendran m <mahendranec@hotmail.com> wrote:
> > Hi All,
> >
> > I was working on moving the IIS logs into HDFS using flume spooling 
> > directory source i was running the 3 website in IIS server , so my 
> > logging directory will looks like below
> >
> > LogFiles
> >
> > |
> > |-- W3SVC1
> >
> > |   |
> >
> > |        `--website1.log
> >
> > |
> >
> > |-- W3SVC1
> >
> > |   |
> >
> > |        `--website2.log
> >
> > |
> >
> > `-- W3SCV3
> >     |
> >     `-- website3.log
> >
> >
> >
> > Now my use case is that i want to move the log of website 3 that  is 
> > website3.log file to HDFS. So i have configured the  CONF file as 
> > below
> >
> >
> >
> > agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles\\W3SVC3
> >
> >
> > and it is works perfectly . now My use case is changed that i want 
> > move all the log files under the folder LogFiles  , now  i changed the 
> > CONF as below
> >
> >
> > agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles
> >
> >
> > but it does not working . not moving any files from LogFiles 
> > directory. How can i achieve my use case ?
> >
> >
> > Thanks,
> >
> > Mahendran
> >
> >
> 
> 
> 
> --
> thanks
> ashish
> 
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
> 
> Please consider the environment before printing this e-mail
> 
> Disclaimer: This  communication  is  for the exclusive use of the intended recipient(s)
and  shall  not attach any liability on the originator or ITC Infotech India Ltd./its  Holding
company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of
this e-mail are intended for your use only and it shall  not be forwarded to any third party,
without first obtaining written authorization from the originator or ITC Infotech India Ltd./
its Holding company/its  Subsidiaries/ its Group Companies. It may contain information which
is confidential and legally privileged and the same shall not be used or dealt with  by any
 third  party  in  any manner whatsoever without the specific consent  of  ITC  Infotech India
Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
 		 	   		  
Mime
View raw message