flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien HARDY <dha...@figarocms.fr>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 12:10:10 GMT
Hello,

We have some more plug-ins available here :
https://github.com/figarocms/flume-plugins
There are based on Cloudera flume cdh3u3.

Some are very inspire on existing sinks plugins like HbaseSink (backport
to depend on maven cdh3u3 with maven), RabbitMQSink (which include
fail-over on RMQ cluster)
Others are decorators (haproxy, geoip, filter) and a configurable format
(json_object)

Maybe this could help you for you own plugin.

Regards,

-- 
Damien


Le 28/03/2012 10:26, shekhar sharma a écrit :
> Thanks for a quick response..
> But the sinks written for cloudera flume (i was going through hello
> world plugin) use cloudera classes and all.. For FLume-728 the sinks 
> written (HDFSSink and IRCSink) is different from cloudera's.
>  CLoudera flume requires cloudera's hadoop to be installed, if i am
> not wrong?
>
> My idea is to collect the events from Flume agent and send to Esper,
> where i will query the streams using EPL.
> Can you please tell me what all things are required to write a custom
> sink. i mean what needs to be done so that Flume node recognizes the sink.
>
> Thanks
>
> Regards,
> Som
>
> On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com
> <mailto:wget.null@googlemail.com>> wrote:
>
>     Hi,
>
>     Cloudera uses in 3u3 Flume-798
>
>     Apache flume isn't different, you can build it among your
>     installation and tweak the configs to match them or use the binary
>     distribution:
>     https://github.com/cloudera/flume/downloads
>
>     As you read, apache flume 1.1.0 was just released:
>     http://www.apache.org/dyn/closer.cgi/incubator/flume/
>
>     You have to write plugins for sinks, here you found some created
>     from the community:
>     https://cwiki.apache.org/FLUME/flume-plugins.html
>
>     And the docs:
>     http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins
>
>     Q3 - I don't know, if you use serialized data whats with avro?
>
>     - Alex
>
>     --
>     Alexander Lorenz
>     http://mapredit.blogspot.com
>
>     On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
>
>     > Hello every one,
>     > I am trying to integrate FLume and Esper by writing a custom
>     sink for Esper. But as per the documentation , i am unable to do
>     so. i have tried the hello world plugin, but that one also not
>     working.
>     > I am using FLume-728.
>     > I have few question:
>     >
>     > Q1. Is Apache Flume is different from Cloudera's Flume?
>     >        If i am using Cloudera's FLume, then does it have any
>     dependency that i have to use CLoudera's distribution of Hadoop.
>     >
>     > Q2. What are the necessary steps for writing custom sink ?
>     >
>     >
>     > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain
>     old java objects), XML or Map as events. Now if i try integrating
>     both, how feasible it is?
>     >
>     > Regards,
>     > Som Shekhar Sharma
>     >
>     >
>
>



Mime
View raw message