flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shumin Wu <shumin...@gmail.com>
Subject Re: Is Flume NG good for production?
Date Wed, 04 Jan 2012 06:21:43 GMT
Hi Eric,

Thanks for answering my questions. Your help is appreciated.

1. The JDBC channel I wrote for Flume is a plugin package including a
Decorator and a Sink, which are used to prepare and dump data to MySQL
database. Flume OG allows clients to write their own plugins through
extending EventSink.Base class and load them in runtime when FlumeMaster
gets started. The flume-plugin-helloworld is a demo of this feature. I see
that the AbstractSink replaces this EventSink class in NG, but I don't see
the corresponding plugin design in NG, which is the SinkBuilder in OG. If
NG can get that part in, that would be a lot easier for users to work with
their own sources/sinks.

2. Will the community consider integrating the HBase Sink to NG in the
roadmap, just as the HDFS sink? If so, can I open an JIRA for this feature?

3. Based on what you are saying, I think NG is a bit premature for
production. But I would like to do some load testing and share with the
community. Are there any existing results posted somewhere I could use as a

Thanks again for your information.


On Tue, Jan 3, 2012 at 8:14 PM, Eric Sammer <esammer@cloudera.com> wrote:

> On Tue, Jan 3, 2012 at 6:37 PM, Shumin Wu <shumin.wu@gmail.com> wrote:
>>  Hi,
>> I have been using Flume 0.9.4/5 for a while. Now I am thinking of
>> migrating to Flume NG. But there are a fews things I would like to make
>> sure before I do that.
>> 1. As my application needs flume to connect with a legacy MySql database,
>> I wrote my own JDBC channel plugin using the goodies from Spring framework.
>> Flume NG's JdbcChannelProviderImpl uses DBCP.  What is the best practice
>> say if I want to use my Spring jdbc channel? Does the current framework
>> allow me to write a plugin? Or should I just make a patch?
> In NG you wouldn't use a channel plugin. You'd write a sink that writes to
> MySQL where you can do anything you want. Flume's channel is just used for
> delivery from a source to a sink.
>> 2.  How does NG's plugin framework differ from OG? I still want to use
>> the flume-plugin-hbasesink. Maybe I have missed something, but the page of
>> "Flume and HBase Integration" does not mention this. Neither does the Flume
>> NG document.
> Not sure what you mean by plugin framework, but Flume NG is API
> incompatible with OG sources and sinks. Specifically, no one has ported the
> HBase sink to the NG APIs yet.
>> 3. How good is Flume NG for a production environment in general? Anyone
>> in the community already tried it out in production already?
> It depends on what features you're looking for. Currently, NG is
> considered (at least by the active committers) alpha or beta quality. It's
> a developer preview release meant to let early adopters try it out and for
> plugin developers to get an idea of what the APIs will look like. Some
> features are more mature than others but I can't promise it's bug free nor
> has it seen exhaustive testing under very high load. I'm not sure if that
> answers your question.
> We (again, the committers, but also the community) are extremely
> interested in feedback. Maybe you could help test on low value data (if
> there is such a thing) where the potential for data loss is less scary.
>> Thanks,
>> Shumin
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com

View raw message