flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shekhar sharma <shekhar2...@gmail.com>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 08:26:19 GMT
Thanks for a quick response..
But the sinks written for cloudera flume (i was going through hello world
plugin) use cloudera classes and all.. For FLume-728 the sinks  written
(HDFSSink and IRCSink) is different from cloudera's.
 CLoudera flume requires cloudera's hadoop to be installed, if i am not
wrong?

My idea is to collect the events from Flume agent and send to Esper, where
i will query the streams using EPL.
Can you please tell me what all things are required to write a custom sink.
i mean what needs to be done so that Flume node recognizes the sink.

Thanks

Regards,
Som

On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com> wrote:

> Hi,
>
> Cloudera uses in 3u3 Flume-798
>
> Apache flume isn't different, you can build it among your installation and
> tweak the configs to match them or use the binary distribution:
> https://github.com/cloudera/flume/downloads
>
> As you read, apache flume 1.1.0 was just released:
> http://www.apache.org/dyn/closer.cgi/incubator/flume/
>
> You have to write plugins for sinks, here you found some created from the
> community:
> https://cwiki.apache.org/FLUME/flume-plugins.html
>
> And the docs:
>
> http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins
>
> Q3 - I don't know, if you use serialized data whats with avro?
>
> - Alex
>
> --
> Alexander Lorenz
> http://mapredit.blogspot.com
>
> On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
>
> > Hello every one,
> > I am trying to integrate FLume and Esper by writing a custom sink for
> Esper. But as per the documentation , i am unable to do so. i have tried
> the hello world plugin, but that one also not working.
> > I am using FLume-728.
> > I have few question:
> >
> > Q1. Is Apache Flume is different from Cloudera's Flume?
> >        If i am using Cloudera's FLume, then does it have any dependency
> that i have to use CLoudera's distribution of Hadoop.
> >
> > Q2. What are the necessary steps for writing custom sink ?
> >
> >
> > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old
> java objects), XML or Map as events. Now if i try integrating both, how
> feasible it is?
> >
> > Regards,
> > Som Shekhar Sharma
> >
> >
>
>

Mime
View raw message