From flume-user-return-857-apmail-incubator-flume-user-archive=incubator.apache.org@incubator.apache.org Wed Mar 28 23:47:04 2012 Return-Path: X-Original-To: apmail-incubator-flume-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-flume-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1B033926D for ; Wed, 28 Mar 2012 23:47:04 +0000 (UTC) Received: (qmail 99978 invoked by uid 500); 28 Mar 2012 23:47:03 -0000 Delivered-To: apmail-incubator-flume-user-archive@incubator.apache.org Received: (qmail 99924 invoked by uid 500); 28 Mar 2012 23:47:03 -0000 Mailing-List: contact flume-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: flume-user@incubator.apache.org Delivered-To: mailing list flume-user@incubator.apache.org Received: (qmail 99916 invoked by uid 99); 28 Mar 2012 23:47:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2012 23:47:03 +0000 X-ASF-Spam-Status: No, hits=0.0 required=5.0 tests=SPF_PASS,TVD_RCVD_SPACE_BRACKET X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [202.147.45.51] (HELO mx1.optiver.com.au) (202.147.45.51) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2012 23:46:58 +0000 Received: from optiver.com.au (localhost [127.0.0.1]) by mx1.optiver.com.au (Sentrion-MTA-4.2.2/Sentrion-MTA-4.2.2) with ESMTP id q2SNkXET024180 for ; Thu, 29 Mar 2012 10:46:34 +1100 Received: from mail.comp.optiver.com ([opsywnsr0149.comp.optiver.com [10.70.144.38]]) by mx1.optiver.com.au with SMTP id q2SNkKKb024152 ; Thu, 29 Mar 2012 10:46:21 +1100 Received: from OPSYWNSR0152.comp.optiver.com ([10.70.144.40]) by opsywnsr0149.comp.optiver.com ([10.70.144.38]) with mapi; Thu, 29 Mar 2012 10:46:20 +1100 From: James Stewart To: "flume-user@incubator.apache.org" Date: Thu, 29 Mar 2012 10:46:18 +1100 Subject: RE: Integrating Flume a Esper Thread-Topic: Integrating Flume a Esper Thread-Index: Ac0M6efBk+FCjJuKSm6DwEUGI0CuKgAUtLlw Message-ID: <3B7D6CD11AEFB4488B8F2E3E60E9F6DAD90F7B06F5@OPSYWNSR0152.comp.optiver.com> References: <6B885041-027B-4A69-9759-577953FF82D2@gmail.com> <986E3455-9456-4028-A85E-698CF2DAE01F@gmail.com> In-Reply-To: <986E3455-9456-4028-A85E-698CF2DAE01F@gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BCCJournal: Audit copy made Not sure if it's useful, but here's a detailed blog post describing how to = get the plugins working with an rpm install of Flume 0.9.4: http://jimter.net/installing-flume-0-9-4-example-plugins/ James -----Original Message----- From: alo alt [mailto:wget.null@googlemail.com] Sent: Thursday, 29 March 2012 12:51 AM To: flume-user@incubator.apache.org Subject: Re: Integrating Flume a Esper Oh, now we clear. flumeNG (aka old 728 and now 1.1.0-branch) does not support own sinks atm. = Here you should take a look into the source code (flume-ng-sinks) to write = own plugins, feel free to submit a jira with a review request. Please note,= NG isn't ready for production yet. sorry for confusion, we all talk about flume 0.9.4, which will be shipped w= ith cloudera's distribution and support decorators. - Alex -- Alexander Lorenz http://mapredit.blogspot.com On Mar 28, 2012, at 3:43 PM, shekhar sharma wrote: > Hello Alexander, > I have created a class known as EsperSink.java and EsperListener.java, co= mpiled them and created a jar(esper-sink.jar) file. > > I have put the esper-sink.jar along with all esper related jar into > the lib folder (/flume-1.1.0-incubating-SNAPSHOT/lib) > > And my agent file is something like this: > # Sources > agent1.sources.log.type =3D exec > agent1.sources.log.command=3D/usr/bin/tail -F > /home/dev/EsperPOC/SamplePOC/test.txt > agent1.sources.log.channels =3D log > > # Channgels > agent1.channels.log.type =3D memory > > # Sinks > agent1.sinks.log.type =3D Esper > agent1.sinks.log.hostname =3D namenode > agent1.sinks.log.port =3D 41414 > agent1.sinks.log.batch-size =3D 10 > agent1.sinks.log.runner.type =3D polling > agent1.sinks.log.runner.polling.interval =3D 5 agent1.sinks.log.channel > =3D log > > # Load everything > agent1.sources =3D log > agent1.sinks =3D log > agent1.channels =3D log > > > When i am running the agent, it is throwing the following error: > 2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR - org.apache.flume.co= nf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractF= ileConfigurationProvider.java:205)] Failed to load configuration data. Exce= ption follows. > org.apache.flume.FlumeException: Unable to load sink type: Esper, > class: Esper > > There is no file with a name flume-conf.xml only property template and th= is property template is meant for defining sources and sinks. > Anyways i will create flume-conf.xml and will add the plugins > > Regards, > Som > > On Wed, Mar 28, 2012 at 6:59 PM, alo alt wrote= : > You have to put the them into ./lib (from a flume view) and enable the ja= rs in flume-conf.xml: > > > flume.plugin.classes > here.the.plugins > Comma separated list of plugin classes > > > -- > Alexander Lorenz > http://mapredit.blogspot.com > > On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote: > > > Hel, > > i have written a custom sink for Esper, can you please how would i make= FLume to recognize this is my new sink..Do i have to specify in configurat= ion file. Can you please help me? > > > > Regards, > > Som > > > > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma = wrote: > > Thanks for a quick response.. > > But the sinks written for cloudera flume (i was going through hello wor= ld plugin) use cloudera classes and all.. For FLume-728 the sinks written = (HDFSSink and IRCSink) is different from cloudera's. > > CLoudera flume requires cloudera's hadoop to be installed, if i am not= wrong? > > > > My idea is to collect the events from Flume agent and send to Esper, wh= ere i will query the streams using EPL. > > Can you please tell me what all things are required to write a custom s= ink. i mean what needs to be done so that Flume node recognizes the sink. > > > > Thanks > > > > Regards, > > Som > > > > > > On Wed, Mar 28, 2012 at 1:06 PM, alo alt wro= te: > > Hi, > > > > Cloudera uses in 3u3 Flume-798 > > > > Apache flume isn't different, you can build it among your installation = and tweak the configs to match them or use the binary distribution: > > https://github.com/cloudera/flume/downloads > > > > As you read, apache flume 1.1.0 was just released: > > http://www.apache.org/dyn/closer.cgi/incubator/flume/ > > > > You have to write plugins for sinks, here you found some created from t= he community: > > https://cwiki.apache.org/FLUME/flume-plugins.html > > > > And the docs: > > http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extend > > ing_via_sink_source_decorator_plugins > > > > Q3 - I don't know, if you use serialized data whats with avro? > > > > - Alex > > > > -- > > Alexander Lorenz > > http://mapredit.blogspot.com > > > > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote: > > > > > Hello every one, > > > I am trying to integrate FLume and Esper by writing a custom sink for= Esper. But as per the documentation , i am unable to do so. i have tried t= he hello world plugin, but that one also not working. > > > I am using FLume-728. > > > I have few question: > > > > > > Q1. Is Apache Flume is different from Cloudera's Flume? > > > If i am using Cloudera's FLume, then does it have any dependen= cy that i have to use CLoudera's distribution of Hadoop. > > > > > > Q2. What are the necessary steps for writing custom sink ? > > > > > > > > > Q3. I have implemented a POC in Esper, and it takes POJO (Plain old = java objects), XML or Map as events. Now if i try integrating both, how fea= sible it is? > > > > > > Regards, > > > Som Shekhar Sharma > > > > > > > > > > > > > > Information contained in this communication (including any attachments) is = confidential and may be privileged or subject to copyright. If you have re= ceived this communication in error you are not authorised to use the inform= ation in any way and Optiver requests that you notify the sender by return = email, destroy all copies and delete the information from your system. Opt= iver does not represent, warrant or guarantee that this communication is fr= ee from computer viruses or other defects or that the integrity of this com= munication has been maintained. Any views expressed in this communication = are those of the individual sender. Optiver does not accept liability for = any loss or damage caused directly or indirectly by this communication or i= ts use. Please consider the environment before printing this email.