flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alo alt <wget.n...@googlemail.com>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 13:29:20 GMT
You have to put the them into ./lib (from a flume view) and enable the jars in flume-conf.xml:

<property>    
 <name>flume.plugin.classes</name>
 <value>here.the.plugins</value>
 <description>Comma separated list of plugin classes</description>
</property>

--
Alexander Lorenz
http://mapredit.blogspot.com

On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote:

> Hel,
> i have written a custom sink for Esper, can you please how would i make FLume to recognize
this is my new sink..Do i have to specify in configuration file. Can you please help me?
> 
> Regards,
> Som
> 
> On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <shekhar2581@gmail.com> wrote:
> Thanks for a quick response..
> But the sinks written for cloudera flume (i was going through hello world plugin) use
cloudera classes and all.. For FLume-728 the sinks  written (HDFSSink and IRCSink) is different
from cloudera's. 
>  CLoudera flume requires cloudera's hadoop to be installed, if i am not wrong?
> 
> My idea is to collect the events from Flume agent and send to Esper, where i will query
the streams using EPL.
> Can you please tell me what all things are required to write a custom sink. i mean what
needs to be done so that Flume node recognizes the sink.
> 
> Thanks
> 
> Regards,
> Som
> 
> 
> On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com> wrote:
> Hi,
> 
> Cloudera uses in 3u3 Flume-798
> 
> Apache flume isn't different, you can build it among your installation and tweak the
configs to match them or use the binary distribution:
> https://github.com/cloudera/flume/downloads
> 
> As you read, apache flume 1.1.0 was just released:
> http://www.apache.org/dyn/closer.cgi/incubator/flume/
> 
> You have to write plugins for sinks, here you found some created from the community:
> https://cwiki.apache.org/FLUME/flume-plugins.html
> 
> And the docs:
> http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins
> 
> Q3 - I don't know, if you use serialized data whats with avro?
> 
> - Alex
> 
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
> 
> On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
> 
> > Hello every one,
> > I am trying to integrate FLume and Esper by writing a custom sink for Esper. But
as per the documentation , i am unable to do so. i have tried the hello world plugin, but
that one also not working.
> > I am using FLume-728.
> > I have few question:
> >
> > Q1. Is Apache Flume is different from Cloudera's Flume?
> >        If i am using Cloudera's FLume, then does it have any dependency that i have
to use CLoudera's distribution of Hadoop.
> >
> > Q2. What are the necessary steps for writing custom sink ?
> >
> >
> > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old java objects),
XML or Map as events. Now if i try integrating both, how feasible it is?
> >
> > Regards,
> > Som Shekhar Sharma
> >
> >
> 
> 
> 


Mime
View raw message