The sink which i have written is at very initial stage, i am just trying to send the events to the Esper engine and printing the events on the console...i just want to verify the flow, Weblogs to FLume to esper engine.
Currently i am using exec as source, memory as channel and esper as sink, but Flume is not recognizing the sink...
Well in the Apache Flume, we need to specify the sink name in the file "SinkType.java".
I am little bit reluctant to use Cloudera's because my other set up of CEP engine is based on all Apache's distribution..
I will share the code soon, you can help me in refining the code..
We have some more plug-ins available here :
There are based on Cloudera flume cdh3u3.
Some are very inspire on existing sinks plugins like HbaseSink (backport
to depend on maven cdh3u3 with maven), RabbitMQSink (which include
fail-over on RMQ cluster)
Others are decorators (haproxy, geoip, filter) and a configurable format
Maybe this could help you for you own plugin.
Le 28/03/2012 10:26, shekhar sharma a écrit :
> Thanks for a quick response..
> But the sinks written for cloudera flume (i was going through hello
> world plugin) use cloudera classes and all.. For FLume-728 the sinks
> written (HDFSSink and IRCSink) is different from cloudera's.
> CLoudera flume requires cloudera's hadoop to be installed, if i am
> not wrong?
> My idea is to collect the events from Flume agent and send to Esper,
> where i will query the streams using EPL.
> Can you please tell me what all things are required to write a custom
> sink. i mean what needs to be done so that Flume node recognizes the sink.
> On Wed, Mar 28, 2012 at 1:06 PM, alo alt <email@example.com
> <mailto:firstname.lastname@example.org>> wrote:
> Cloudera uses in 3u3 Flume-798
> Apache flume isn't different, you can build it among your
> installation and tweak the configs to match them or use the binary
> As you read, apache flume 1.1.0 was just released:
> You have to write plugins for sinks, here you found some created
> from the community:
> And the docs:
> Q3 - I don't know, if you use serialized data whats with avro?
> - Alex
> Alexander Lorenz
> On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
> > Hello every one,
> > I am trying to integrate FLume and Esper by writing a custom
> sink for Esper. But as per the documentation , i am unable to do
> so. i have tried the hello world plugin, but that one also not
> > I am using FLume-728.
> > I have few question:
> > Q1. Is Apache Flume is different from Cloudera's Flume?
> > If i am using Cloudera's FLume, then does it have any
> dependency that i have to use CLoudera's distribution of Hadoop.
> > Q2. What are the necessary steps for writing custom sink ?
> > Q3. I have implemented a POC in Esper, and it takes POJO (Plain
> old java objects), XML or Map as events. Now if i try integrating
> both, how feasible it is?
> > Regards,
> > Som Shekhar Sharma