flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish <paliwalash...@gmail.com>
Subject Re: No threads used in Spooling Directory
Date Mon, 16 Feb 2015 10:56:25 GMT
This could be of help https://issues.apache.org/jira/browse/FLUME-2413 (you
need this at Sink level but this should work as well)

HTH!


On Mon, Feb 16, 2015 at 4:01 PM, mahendran m <mahendranec@hotmail.com>
wrote:

> I am moving the IIS logs into remote hdfs using spooling directory and I
> need to update how much data (size not no of events) has been moved
> successfully in real time in UI
>
> Thanks,
> ------------------------------
> Date: Mon, 16 Feb 2015 12:59:06 +0530
> Subject: Re: No threads used in Spooling Directory
> From: paliwalashish@gmail.com
> To: user@flume.apache.org
>
>
> What Flume Dev's meant was, Flume is not intended to do File transfers as
> a whole. A File shall be read and an Event created out of the content based
> on how an Event needs to be created (each line as an Event by default).
>
> To answer you question (to the best of my knowledge)
>
> Q1. I would like to know how much  file has been processed in parallel by
> SpoolDir ?
> Nope
>
> Q2. Is it possible to how much* size* of file has been moved by SpoolDir
> to sink  at any point using built-in API?
> Nope
> If you can share what you want to achieve, we may be able to provide some
> pointers.
> thanks
> ashish
>
>
>
> On Fri, Feb 13, 2015 at 5:43 PM, mahendran m <mahendranec@hotmail.com>
> wrote:
>
> I need to move IIS logs to hdfs , then which is suitable for this purpose ?
>
> Thanks ,
>
> ------------------------------
> From: Manohar.CS@itcinfotech.com
> To: user@flume.apache.org
> Subject: RE: No threads used in Spooling Directory
> Date: Fri, 13 Feb 2015 11:00:48 +0000
>
>
>  I had posted for similar requirement few days back , but from core team
> members got to know that spoolDir is not suitable for fileBased handling,
> its more of event based content transfer that is being done.
>
>  *From:* mahendran m [mailto:mahendranec@hotmail.com]
> *Sent:* Friday, February 13, 2015 11:02 AM
> *To:* FlumeMaling List
> *Subject:* No threads used in Spooling Directory
>
>  HI ,
>
>  Q1. I would like to know how much  file has been processed in parallel
> by SpoolDir ?
>
>  Q2. Is it possible to how much* size* of file has been moved by SpoolDir
> to sink  at any point using built-in API?
>
>  Thanks ,
>
>
>
> Please consider the environment before printing this e-mail
>
>
> Disclaimer: This  communication  is  for the exclusive use of the intended recipient(s)
and  shall  not attach any liability on the originator or ITC Infotech India Ltd./its  Holding
company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of
this e-mail are intended for your use only and it shall  not be forwarded to any third party,
without first obtaining written authorization from the originator or ITC Infotech India Ltd./
its Holding company/its  Subsidiaries/ its Group Companies. It may contain information which
is confidential and legally privileged and the same shall not be used or dealt with  by any
 third  party  in  any manner whatsoever without the specific consent  of  ITC  Infotech India
Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
>
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>



-- 
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal

Mime
View raw message