flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Sammer <esam...@cloudera.com>
Subject Re: Is Flume NG good for production?
Date Wed, 04 Jan 2012 04:14:08 GMT
On Tue, Jan 3, 2012 at 6:37 PM, Shumin Wu <shumin.wu@gmail.com> wrote:

>  Hi,
> I have been using Flume 0.9.4/5 for a while. Now I am thinking of
> migrating to Flume NG. But there are a fews things I would like to make
> sure before I do that.
> 1. As my application needs flume to connect with a legacy MySql database,
> I wrote my own JDBC channel plugin using the goodies from Spring framework.
> Flume NG's JdbcChannelProviderImpl uses DBCP.  What is the best practice
> say if I want to use my Spring jdbc channel? Does the current framework
> allow me to write a plugin? Or should I just make a patch?

In NG you wouldn't use a channel plugin. You'd write a sink that writes to
MySQL where you can do anything you want. Flume's channel is just used for
delivery from a source to a sink.

> 2.  How does NG's plugin framework differ from OG? I still want to use the
> flume-plugin-hbasesink. Maybe I have missed something, but the page of
> "Flume and HBase Integration" does not mention this. Neither does the Flume
> NG document.

Not sure what you mean by plugin framework, but Flume NG is API
incompatible with OG sources and sinks. Specifically, no one has ported the
HBase sink to the NG APIs yet.

> 3. How good is Flume NG for a production environment in general? Anyone in
> the community already tried it out in production already?

It depends on what features you're looking for. Currently, NG is considered
(at least by the active committers) alpha or beta quality. It's a developer
preview release meant to let early adopters try it out and for plugin
developers to get an idea of what the APIs will look like. Some features
are more mature than others but I can't promise it's bug free nor has it
seen exhaustive testing under very high load. I'm not sure if that answers
your question.

We (again, the committers, but also the community) are extremely interested
in feedback. Maybe you could help test on low value data (if there is such
a thing) where the potential for data loss is less scary.

> Thanks,
> Shumin

Eric Sammer
twitter: esammer
data: www.cloudera.com

View raw message