flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shekhar sharma <shekhar2...@gmail.com>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 13:43:53 GMT
Hello Alexander,
I have created a class known as EsperSink.java and EsperListener.java,
compiled them and created a jar(esper-sink.jar) file.

I have put the esper-sink.jar along with all esper related jar into the lib
folder (/flume-1.1.0-incubating-SNAPSHOT/lib)

And my agent file is something like this:
# Sources
agent1.sources.log.type = exec
agent1.sources.log.command=/usr/bin/tail -F
/home/dev/EsperPOC/SamplePOC/test.txt
agent1.sources.log.channels = log

# Channgels
agent1.channels.log.type = memory

# Sinks
agent1.sinks.log.type = Esper
agent1.sinks.log.hostname = namenode
agent1.sinks.log.port = 41414
agent1.sinks.log.batch-size = 10
agent1.sinks.log.runner.type = polling
agent1.sinks.log.runner.polling.interval = 5
agent1.sinks.log.channel = log

# Load everything
agent1.sources = log
agent1.sinks = log
agent1.channels = log


When i am running the agent, it is throwing the following error:

2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR -
org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:205)]
Failed to load configuration data. Exception follows.

org.apache.flume.FlumeException: Unable to load sink type: Esper, class:
Esper

There is no file with a name flume-conf.xml only property template and this
property template is meant for defining sources and sinks.
Anyways i will create flume-conf.xml and will add the plugins

Regards,
Som

On Wed, Mar 28, 2012 at 6:59 PM, alo alt <wget.null@googlemail.com> wrote:

> You have to put the them into ./lib (from a flume view) and enable the
> jars in flume-conf.xml:
>
> <property>
>  <name>flume.plugin.classes</name>
>  <value>here.the.plugins</value>
>  <description>Comma separated list of plugin classes</description>
> </property>
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote:
>
> > Hel,
> > i have written a custom sink for Esper, can you please how would i make
> FLume to recognize this is my new sink..Do i have to specify in
> configuration file. Can you please help me?
> >
> > Regards,
> > Som
> >
> > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <shekhar2581@gmail.com>
> wrote:
> > Thanks for a quick response..
> > But the sinks written for cloudera flume (i was going through hello
> world plugin) use cloudera classes and all.. For FLume-728 the sinks
>  written (HDFSSink and IRCSink) is different from cloudera's.
> >  CLoudera flume requires cloudera's hadoop to be installed, if i am not
> wrong?
> >
> > My idea is to collect the events from Flume agent and send to Esper,
> where i will query the streams using EPL.
> > Can you please tell me what all things are required to write a custom
> sink. i mean what needs to be done so that Flume node recognizes the sink.
> >
> > Thanks
> >
> > Regards,
> > Som
> >
> >
> > On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com>
> wrote:
> > Hi,
> >
> > Cloudera uses in 3u3 Flume-798
> >
> > Apache flume isn't different, you can build it among your installation
> and tweak the configs to match them or use the binary distribution:
> > https://github.com/cloudera/flume/downloads
> >
> > As you read, apache flume 1.1.0 was just released:
> > http://www.apache.org/dyn/closer.cgi/incubator/flume/
> >
> > You have to write plugins for sinks, here you found some created from
> the community:
> > https://cwiki.apache.org/FLUME/flume-plugins.html
> >
> > And the docs:
> >
> http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins
> >
> > Q3 - I don't know, if you use serialized data whats with avro?
> >
> > - Alex
> >
> > --
> > Alexander Lorenz
> > http://mapredit.blogspot.com
> >
> > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
> >
> > > Hello every one,
> > > I am trying to integrate FLume and Esper by writing a custom sink for
> Esper. But as per the documentation , i am unable to do so. i have tried
> the hello world plugin, but that one also not working.
> > > I am using FLume-728.
> > > I have few question:
> > >
> > > Q1. Is Apache Flume is different from Cloudera's Flume?
> > >        If i am using Cloudera's FLume, then does it have any
> dependency that i have to use CLoudera's distribution of Hadoop.
> > >
> > > Q2. What are the necessary steps for writing custom sink ?
> > >
> > >
> > > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old
> java objects), XML or Map as events. Now if i try integrating both, how
> feasible it is?
> > >
> > > Regards,
> > > Som Shekhar Sharma
> > >
> > >
> >
> >
> >
>
>

Mime
View raw message