flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subramanyam Satyanarayana <su...@attributor.com>
Subject Re: How to tail a flume stream?
Date Mon, 14 Nov 2011 18:23:29 GMT
We have used the fan-out sink & seems to work pretty well, writing out both
to HDFS & local file system

Here is the configuration schema

exec config 'collector'  'CollectorSource(35853)'
'[collectorSink("hdfs://<namenode>:54310/<location>/%Y/%m/%d/","%{host}-%{tailSrcFile}.log"),collectorSink("file:///local_file_path","%{host}-%{tailSrcFile}.log")]'

~Subbu

On Mon, Nov 14, 2011 at 3:57 AM, Meyer, Dennis <dennis.meyer@adtech.com>wrote:

> Hi,
>
> As far as I understood a flume agent can sent to multiple sinks. Can't you
> just configure two sinks, one as HDFS and on as normal file system? I'm
> interested in that functionality as well. Does anybody know?
>
> Thanks,
> Dennis
>
>
>
>
>
> Am 11.11.11 13:50 schrieb "Torsten Curdt" unter <tcurdt@vafer.org>:
>
> >No, we have the ASF paperwork on file but we never got around to release
> >it yet.
> >
> >We also added a disk based ring buffer in case of hdfs cluster
> >downtime, fixed some syslog bugs and worked on the ganglia integration
> >(not working as it should and we ended up going a simpler route).
> >
> >I bet it will need to some love to turn into a patch for trunk. I can
> >bring it up and see if we can open source over on github without
> >spending the additional time and then see if there is interest in
> >porting some things over.
> >
> >cheers,
> >Torsten
>
>


-- 
Thanks!!
~Subbu

Mime
View raw message