flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Meyer, Dennis" <dennis.me...@adtech.com>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 12:02:27 GMT
Hi Som,

I'd be interested also in connection flume against Esper. Would you mind sharing your code
if it's meant to be open source?

Thanks,
Dennis

Von: shekhar sharma <shekhar2581@gmail.com<mailto:shekhar2581@gmail.com>>
Antworten an: <flume-user@incubator.apache.org<mailto:flume-user@incubator.apache.org>>
Datum: Wed, 28 Mar 2012 15:45:35 +0530
An: <flume-user@incubator.apache.org<mailto:flume-user@incubator.apache.org>>
Betreff: Re: Integrating Flume a Esper

Hel,
i have written a custom sink for Esper, can you please how would i make FLume to recognize
this is my new sink..Do i have to specify in configuration file. Can you please help me?

Regards,
Som

On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <shekhar2581@gmail.com<mailto:shekhar2581@gmail.com>>
wrote:
Thanks for a quick response..
But the sinks written for cloudera flume (i was going through hello world plugin) use cloudera
classes and all.. For FLume-728 the sinks  written (HDFSSink and IRCSink) is different from
cloudera's.
 CLoudera flume requires cloudera's hadoop to be installed, if i am not wrong?

My idea is to collect the events from Flume agent and send to Esper, where i will query the
streams using EPL.
Can you please tell me what all things are required to write a custom sink. i mean what needs
to be done so that Flume node recognizes the sink.

Thanks

Regards,
Som


On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com<mailto:wget.null@googlemail.com>>
wrote:
Hi,

Cloudera uses in 3u3 Flume-798

Apache flume isn't different, you can build it among your installation and tweak the configs
to match them or use the binary distribution:
https://github.com/cloudera/flume/downloads

As you read, apache flume 1.1.0 was just released:
http://www.apache.org/dyn/closer.cgi/incubator/flume/

You have to write plugins for sinks, here you found some created from the community:
https://cwiki.apache.org/FLUME/flume-plugins.html

And the docs:
http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins

Q3 - I don't know, if you use serialized data whats with avro?

- Alex

--
Alexander Lorenz
http://mapredit.blogspot.com

On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:

> Hello every one,
> I am trying to integrate FLume and Esper by writing a custom sink for Esper. But as per
the documentation , i am unable to do so. i have tried the hello world plugin, but that one
also not working.
> I am using FLume-728.
> I have few question:
>
> Q1. Is Apache Flume is different from Cloudera's Flume?
>        If i am using Cloudera's FLume, then does it have any dependency that i have to
use CLoudera's distribution of Hadoop.
>
> Q2. What are the necessary steps for writing custom sink ?
>
>
> Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old java objects), XML
or Map as events. Now if i try integrating both, how feasible it is?
>
> Regards,
> Som Shekhar Sharma
>
>




Mime
View raw message