flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juho Mäkinen <j...@applifier.com>
Subject How to tail a flume stream?
Date Fri, 11 Nov 2011 12:01:45 GMT
We're in process of migrating from Scribe (due to limited features for
our needs) to Flume, and I'm wondering how to implement a stream
tailing to suit our needs.

With scribe, we duplicated the log stream to both HDFS and a standard
file on a standard non-hdfs directory, so that we could "tail -f" the
files in the filesystem to watch the messages on real time. This is
very effective for debugging and developing.

It seems that, at least at some point, the collectorSink could not
sync/append to file:// destinations and as far as I know, this has not
yet been fixed. Am I right? If this would work, we could just keep
tail -f'ing the files on the local filesystem.

The other think I tried to look, is that could pretty easily provide a
tail feature, where anybody could start tailing a stream. As far as I
know, this could be implemented with relatively easy due to the flume
master node. A cli command could connect to the master and ask that
the agents would start to send a copy of a specific flume stream to
the cli and thus enabling to dump the feed to the console. To my
current knowledge, nothing like this has been implemented. Am I right?

I'd appreciate all answers,

 - Juho Mäkinen

View raw message