flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Stewart <James.Stew...@Optiver.com.au>
Subject RE: Integrating Flume a Esper
Date Wed, 28 Mar 2012 23:46:18 GMT
Not sure if it's useful, but here's a detailed blog post describing how to get the plugins
working with an rpm install of Flume 0.9.4:

http://jimter.net/installing-flume-0-9-4-example-plugins/

James

-----Original Message-----
From: alo alt [mailto:wget.null@googlemail.com]
Sent: Thursday, 29 March 2012 12:51 AM
To: flume-user@incubator.apache.org
Subject: Re: Integrating Flume a Esper

Oh, now we clear.
flumeNG (aka old 728 and now 1.1.0-branch) does not support own sinks atm. Here you should
take a look into the source code (flume-ng-sinks) to write own plugins, feel free to submit
a jira with a review request. Please note, NG isn't ready for production yet.

sorry for confusion, we all talk about flume 0.9.4, which will be shipped with cloudera's
distribution and support decorators.

- Alex



--
Alexander Lorenz
http://mapredit.blogspot.com

On Mar 28, 2012, at 3:43 PM, shekhar sharma wrote:

> Hello Alexander,
> I have created a class known as EsperSink.java and EsperListener.java, compiled them
and created a jar(esper-sink.jar) file.
>
> I have put the esper-sink.jar along with all esper related jar into
> the lib folder (/flume-1.1.0-incubating-SNAPSHOT/lib)
>
> And my agent file is something like this:
> # Sources
> agent1.sources.log.type = exec
> agent1.sources.log.command=/usr/bin/tail -F
> /home/dev/EsperPOC/SamplePOC/test.txt
> agent1.sources.log.channels = log
>
> # Channgels
> agent1.channels.log.type = memory
>
> # Sinks
> agent1.sinks.log.type = Esper
> agent1.sinks.log.hostname = namenode
> agent1.sinks.log.port = 41414
> agent1.sinks.log.batch-size = 10
> agent1.sinks.log.runner.type = polling
> agent1.sinks.log.runner.polling.interval = 5 agent1.sinks.log.channel
> = log
>
> # Load everything
> agent1.sources = log
> agent1.sinks = log
> agent1.channels = log
>
>
> When i am running the agent, it is throwing the following error:
> 2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)]
Failed to load configuration data. Exception follows.
> org.apache.flume.FlumeException: Unable to load sink type: Esper,
> class: Esper
>
> There is no file with a name flume-conf.xml only property template and this property
template is meant for defining sources and sinks.
> Anyways i will create flume-conf.xml and will add the plugins
>
> Regards,
> Som
>
> On Wed, Mar 28, 2012 at 6:59 PM, alo alt <wget.null@googlemail.com> wrote:
> You have to put the them into ./lib (from a flume view) and enable the jars in flume-conf.xml:
>
> <property>
>  <name>flume.plugin.classes</name>
>  <value>here.the.plugins</value>
>  <description>Comma separated list of plugin classes</description>
> </property>
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote:
>
> > Hel,
> > i have written a custom sink for Esper, can you please how would i make FLume to
recognize this is my new sink..Do i have to specify in configuration file. Can you please
help me?
> >
> > Regards,
> > Som
> >
> > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <shekhar2581@gmail.com> wrote:
> > Thanks for a quick response..
> > But the sinks written for cloudera flume (i was going through hello world plugin)
use cloudera classes and all.. For FLume-728 the sinks  written (HDFSSink and IRCSink) is
different from cloudera's.
> >  CLoudera flume requires cloudera's hadoop to be installed, if i am not wrong?
> >
> > My idea is to collect the events from Flume agent and send to Esper, where i will
query the streams using EPL.
> > Can you please tell me what all things are required to write a custom sink. i mean
what needs to be done so that Flume node recognizes the sink.
> >
> > Thanks
> >
> > Regards,
> > Som
> >
> >
> > On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com> wrote:
> > Hi,
> >
> > Cloudera uses in 3u3 Flume-798
> >
> > Apache flume isn't different, you can build it among your installation and tweak
the configs to match them or use the binary distribution:
> > https://github.com/cloudera/flume/downloads
> >
> > As you read, apache flume 1.1.0 was just released:
> > http://www.apache.org/dyn/closer.cgi/incubator/flume/
> >
> > You have to write plugins for sinks, here you found some created from the community:
> > https://cwiki.apache.org/FLUME/flume-plugins.html
> >
> > And the docs:
> > http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extend
> > ing_via_sink_source_decorator_plugins
> >
> > Q3 - I don't know, if you use serialized data whats with avro?
> >
> > - Alex
> >
> > --
> > Alexander Lorenz
> > http://mapredit.blogspot.com
> >
> > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
> >
> > > Hello every one,
> > > I am trying to integrate FLume and Esper by writing a custom sink for Esper.
But as per the documentation , i am unable to do so. i have tried the hello world plugin,
but that one also not working.
> > > I am using FLume-728.
> > > I have few question:
> > >
> > > Q1. Is Apache Flume is different from Cloudera's Flume?
> > >        If i am using Cloudera's FLume, then does it have any dependency that
i have to use CLoudera's distribution of Hadoop.
> > >
> > > Q2. What are the necessary steps for writing custom sink ?
> > >
> > >
> > > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old java objects),
XML or Map as events. Now if i try integrating both, how feasible it is?
> > >
> > > Regards,
> > > Som Shekhar Sharma
> > >
> > >
> >
> >
> >
>
>


Information contained in this communication (including any attachments) is confidential and
may be privileged or subject to copyright.  If you have received this communication in error
you are not authorised to use the information in any way and Optiver requests that you notify
the sender by return email, destroy all copies and delete the information from your system.
 Optiver does not represent, warrant or guarantee that this communication is free from computer
viruses or other defects or that the integrity of this communication has been maintained.
 Any views expressed in this communication are those of the individual sender.  Optiver does
not accept liability for any loss or damage caused directly or indirectly by this communication
or its use.

Please consider the environment before printing this email.

Mime
View raw message