flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shekhar sharma <shekhar2...@gmail.com>
Subject Re: Integrating Flume a Esper
Date Wed, 28 Mar 2012 10:15:35 GMT
Hel,
i have written a custom sink for Esper, can you please how would i make
FLume to recognize this is my new sink..Do i have to specify in
configuration file. Can you please help me?

Regards,
Som

On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma <shekhar2581@gmail.com>wrote:

> Thanks for a quick response..
> But the sinks written for cloudera flume (i was going through hello world
> plugin) use cloudera classes and all.. For FLume-728 the sinks  written
> (HDFSSink and IRCSink) is different from cloudera's.
>  CLoudera flume requires cloudera's hadoop to be installed, if i am not
> wrong?
>
> My idea is to collect the events from Flume agent and send to Esper, where
> i will query the streams using EPL.
> Can you please tell me what all things are required to write a custom
> sink. i mean what needs to be done so that Flume node recognizes the sink.
>
> Thanks
>
> Regards,
> Som
>
>
> On Wed, Mar 28, 2012 at 1:06 PM, alo alt <wget.null@googlemail.com> wrote:
>
>> Hi,
>>
>> Cloudera uses in 3u3 Flume-798
>>
>> Apache flume isn't different, you can build it among your installation
>> and tweak the configs to match them or use the binary distribution:
>> https://github.com/cloudera/flume/downloads
>>
>> As you read, apache flume 1.1.0 was just released:
>> http://www.apache.org/dyn/closer.cgi/incubator/flume/
>>
>> You have to write plugins for sinks, here you found some created from the
>> community:
>> https://cwiki.apache.org/FLUME/flume-plugins.html
>>
>> And the docs:
>>
>> http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_via_sink_source_decorator_plugins
>>
>> Q3 - I don't know, if you use serialized data whats with avro?
>>
>> - Alex
>>
>> --
>> Alexander Lorenz
>> http://mapredit.blogspot.com
>>
>> On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote:
>>
>> > Hello every one,
>> > I am trying to integrate FLume and Esper by writing a custom sink for
>> Esper. But as per the documentation , i am unable to do so. i have tried
>> the hello world plugin, but that one also not working.
>> > I am using FLume-728.
>> > I have few question:
>> >
>> > Q1. Is Apache Flume is different from Cloudera's Flume?
>> >        If i am using Cloudera's FLume, then does it have any dependency
>> that i have to use CLoudera's distribution of Hadoop.
>> >
>> > Q2. What are the necessary steps for writing custom sink ?
>> >
>> >
>> > Q3.  I have implemented a POC in Esper, and it takes POJO (Plain old
>> java objects), XML or Map as events. Now if i try integrating both, how
>> feasible it is?
>> >
>> > Regards,
>> > Som Shekhar Sharma
>> >
>> >
>>
>>
>

Mime
View raw message