flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@googlemail.com>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 13:51:08 GMT
Oh, now we clear.
flumeNG (aka old 728 and now 1.1.0-branch) does not support own sinks atm. Here you should
take a look into the source code (flume-ng-sinks) to write own plugins, feel free to submit
a jira with a review request. Please note, NG isn't ready for production yet.

sorry for confusion, we all talk about flume 0.9.4, which will be shipped with cloudera's
distribution and support decorators. 

- Alex



--
Alexander Lorenz
http://mapredit.blogspot.com

On Mar 28, 2012, at 3:43 PM, shekhar sharma wrote:

> Hello Alexander,
> I have created a class known as EsperSink.java and EsperListener.java, compiled them
and created a jar(esper-sink.jar) file.
> 
> I have put the esper-sink.jar along with all esper related jar into the lib folder (/flume-1.1.0-incubating-SNAPSHOT/lib)
> 
> And my agent file is something like this:
> # Sources
> agent1.sources.log.type = exec
> agent1.sources.log.command=/usr/bin/tail -F /home/dev/EsperPOC/SamplePOC/test.txt
> agent1.sources.log.channels = log
> 
> # Channgels
> agent1.channels.log.type = memory
> 
> # Sinks
> agent1.sinks.log.type = Esper
> agent1.sinks.log.hostname = namenode
> agent1.sinks.log.port = 41414
> agent1.sinks.log.batch-size = 10
> agent1.sinks.log.runner.type = polling
> agent1.sinks.log.runner.polling.interval = 5
> agent1.sinks.log.channel = log
> 
> # Load everything
> agent1.sources = log
> agent1.sinks = log
> agent1.channels = log
> 
> 
> When i am running the agent, it is throwing the following error:
> 2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)]
Failed to load configuration data. Exception follows.
> org.apache.flume.FlumeException: Unable to load sink type: Esper, class: Esper
> 
> There is no file with a name flume-conf.xml only property template and this property
template is meant for defining sources and sinks.
> Anyways i will create flume-conf.xml and will add the plugins
> 
> Regards,
> Som
> 
> On Wed, Mar 28, 2012 at 6:59 PM, alo alt <wget.null@googlemail.com> wrote:
> You have to put the them into ./lib (from a flume view) and enable the jars in flume-conf.xml:
> 
> <property>
>  <name>flume.plugin.classes</name>
>  <value>here.the.plugins</value>
>  <description>Comma separated list of plugin classes</description>
> </property>
> 
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
> 
> On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote:
> 
> > Hel,
> > i have written a custom sink for Esper, can you please how would i make FLume to
recognize this is my new sink..Do i have to specify in configuration file. Can you please
help me?
> >
> > Regards,
> > Som
> >
> > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <shekhar2581@gmail.com> wrote:
> > Thanks for a quick response..
> > But the sinks written for cloudera flume (i was going through hello world plugin)
use cloudera classes and all.. For FLume-728 the sinks  written (HDFSSink and IRCSink) is
different from cloudera's.
> >  CLoudera flume requires cloudera's hadoop to be installed, if i am not wrong?
> >
> > My idea is to collect the events from Flume agent and send to Esper, where i will
query the streams using EPL.
> > Can you please tell me what all things are required to write a custom sink. i mean
what needs to be done so that Flume node recognizes the sink.
> >
> > Thanks
> >
> > Regards,
> > Som
> >
> >
> > On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com> wrote:
> > Hi,
> >
> > Cloudera uses in 3u3 Flume-798
> >
> > Apache flume isn't different, you can build it among your installation and tweak
the configs to match them or use the binary distribution:
> > https://github.com/cloudera/flume/downloads
> >
> > As you read, apache flume 1.1.0 was just released:
> > http://www.apache.org/dyn/closer.cgi/incubator/flume/
> >
> > You have to write plugins for sinks, here you found some created from the community:
> > https://cwiki.apache.org/FLUME/flume-plugins.html
> >
> > And the docs:
> > http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins
> >
> > Q3 - I don't know, if you use serialized data whats with avro?
> >
> > - Alex
> >
> > --
> > Alexander Lorenz
> > http://mapredit.blogspot.com
> >
> > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
> >
> > > Hello every one,
> > > I am trying to integrate FLume and Esper by writing a custom sink for Esper.
But as per the documentation , i am unable to do so. i have tried the hello world plugin,
but that one also not working.
> > > I am using FLume-728.
> > > I have few question:
> > >
> > > Q1. Is Apache Flume is different from Cloudera's Flume?
> > >        If i am using Cloudera's FLume, then does it have any dependency that
i have to use CLoudera's distribution of Hadoop.
> > >
> > > Q2. What are the necessary steps for writing custom sink ?
> > >
> > >
> > > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old java objects),
XML or Map as events. Now if i try integrating both, how feasible it is?
> > >
> > > Regards,
> > > Som Shekhar Sharma
> > >
> > >
> >
> >
> >
> 
> 


Mime
View raw message