flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hsieh <...@cloudera.com>
Subject Re: Roadmap / Partitioning by key
Date Mon, 27 Jun 2011 06:07:01 GMT

A basic ability to demultiplex (demux) events exists today but is only
available for writing files to different dirs in HDFS.  The ability to do
content-based routing for computational purposes is not currently on the
road map.  While architecturally be possible to demux in Flume, Flume is
currently focused on sending data from many places to a few.

Can you describe your use case or what you would want to do with if you had
this capability?  This would help us frame this discussion.

If there are a small finite number of  categories, demuxing could
potentially built as plugins for today's Flume.  For something more general
or adaptive, a larger development effort would be required.

Another approach that could be done today would be to send data from Flume
to a system that does demux and custom routing (starting to go down the
complex-event-processing path)..

1) Flume could potentially connect to S4 and deliver it data.  Flume could
have a path that delivers to hdfs, and have another copy sent to S4.
2) Flume could send data to FlumeBase (a system built on top of Flume) which
may (or may not) provide this capability.
3) Flume could send data to an open-source system called Esper. (I don't
know much about it currently)


On Sat, Jun 25, 2011 at 6:31 PM, Evan Chan <ev@ooyala.com> wrote:

> Hi Flume community,
> I hope that the incubator list is being read....  hello to everyone, I'm
> new to Flume.
> Is there a roadmap for future development of Flume?
> I'm interested in particular to see if the ability to have a sink that can
> route events to different nodes based on a key (something that Yahoo S4 can
> do) will be in the roadmap, and how hard it would be to develop a feature
> like that.
> thanks!
> Evan
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> | @ooyala<http://www.twitter.com/ooyala>

// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// jon@cloudera.com

View raw message