flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 纪君祥 <zhongyan.f...@alipay.com>
Subject RE: How to avoid restart master when adding one customer own plugin?
Date Mon, 19 Sep 2011 02:09:20 GMT
Thanks, Mingjie

I think adding one script decorator can works well.

But what I worry is restarting master will refresh all configuration, it will bring trouble
for tail/text, or other source/sink.

I have done one test with 'tail' source, if restart master, it will leading to resending old
data in the file and missing some data.

Longda

-----Original Message-----
From: Mingjie Lai [mailto:mjlai09@gmail.com]
Sent: Saturday, September 17, 2011 3:20 AM
To: flume-user@incubator.apache.org
Subject: Re: How to avoid restart master when adding one customer own plugin?

Longda.

AFAIK, Flume doesn't have a way to load a plugin on the fly. The plugin
class names are in the config file, and shared among all flume nodes +
master. Also the class files need to be loaded when flume starts up.
When you have a new plugin, you have to 1) reconfigure flume at all
nodes, 2) copy the new class file to flume classpath, 3) restart all nodes.

Assume there is a way for flume to dynamically config/load plugin class
names, we still need a way to ship the class file to each node which can
not be handled by flume easily.

Personally I don't think it's a strong operational requirement.

Or you can have a script decorator plugin to avoid flume restart:

http://blog.infochimps.com/2011/03/02/simple-flume-decorators-with-jruby/

-mingjie

On 09/15/2011 10:51 PM, 纪君祥 wrote:
> Hi all,
>
> Could anyone tell me how to avoid restart master when adding one plugin?
> Any help are appreciated.
>
> I found that when I implement my own plugin, if I want to use it, I need
> restart the master node, otherwise, it will report configuration error.
>
> Can we avoid it?
>
> Thanks a lot
>
> Longda
>
>
> ------------------------------------------------------------------------
>
> This email (including any attachments) is confidential and may be
> legally privileged. If you received this email in error, please delete
> it immediately and do not copy it or use it for any purpose or disclose
> its contents to any other person. Thank you.
>
> 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人,
> 请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮
> 件之内容。谢谢。

________________________________

This email (including any attachments) is confidential and may be legally privileged. If you
received this email in error, please delete it immediately and do not copy it or use it for
any purpose or disclose its contents to any other person. Thank you.

本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人,请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
Mime
View raw message