From flume-user-return-855-apmail-incubator-flume-user-archive=incubator.apache.org@incubator.apache.org Wed Mar 28 13:51:38 2012 Return-Path: X-Original-To: apmail-incubator-flume-user-archive@minotaur.apache.org Delivered-To: apmail-incubator-flume-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4F2509195 for ; Wed, 28 Mar 2012 13:51:38 +0000 (UTC) Received: (qmail 26018 invoked by uid 500); 28 Mar 2012 13:51:38 -0000 Delivered-To: apmail-incubator-flume-user-archive@incubator.apache.org Received: (qmail 25986 invoked by uid 500); 28 Mar 2012 13:51:38 -0000 Mailing-List: contact flume-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: flume-user@incubator.apache.org Delivered-To: mailing list flume-user@incubator.apache.org Received: (qmail 25974 invoked by uid 99); 28 Mar 2012 13:51:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2012 13:51:38 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of wget.null@googlemail.com designates 74.125.82.175 as permitted sender) Received: from [74.125.82.175] (HELO mail-we0-f175.google.com) (74.125.82.175) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2012 13:51:33 +0000 Received: by wera1 with SMTP id a1so661658wer.6 for ; Wed, 28 Mar 2012 06:51:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; bh=p9z6h0plZjbmvQdIVzHxo7pOJaiUNBdOOINrIn0Ux6w=; b=v6eu3S1g0dkwMmq4A8tCX43H3Liga6OQCdrB+Wjk8Fu9tE2Ovw0wpcToBz/bvJ05nL gOcPDtxt0XaTv/IQyRxSOpiNopZlFAgdgRxtkWSW+pk+o23H+2RWgMK2+otLqaBO7unO cZ+jIz1NFkmzaMCqjPK6J4TxjUBThAwHpGAwKYTfHCyo3+951eBu5wQ2AyxyBPhZybwq xfCyTu93DprksumZnZfj9HOANNKJ/qBWoAzlF+5I8LW1nLguFWEdm0kcvMzh+KBteW/f CtH3c0q53tlGZVCuZYuV1GRqKOKFZlUV7liDyLcqmtyFyw+qi1sgohWMlEplafAbebUn l3eg== Received: by 10.180.101.72 with SMTP id fe8mr7392255wib.4.1332942671733; Wed, 28 Mar 2012 06:51:11 -0700 (PDT) Received: from [192.168.200.103] (HSI-KBW-149-172-23-146.hsi13.kabel-badenwuerttemberg.de. [149.172.23.146]) by mx.google.com with ESMTPS id n8sm13615189wix.10.2012.03.28.06.51.09 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 28 Mar 2012 06:51:10 -0700 (PDT) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Apple Message framework v1257) Subject: Re: Integrating Flume a Esper From: alo alt In-Reply-To: Date: Wed, 28 Mar 2012 15:51:08 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <986E3455-9456-4028-A85E-698CF2DAE01F@gmail.com> References: <6B885041-027B-4A69-9759-577953FF82D2@gmail.com> To: flume-user@incubator.apache.org X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org Oh, now we clear. flumeNG (aka old 728 and now 1.1.0-branch) does not support own sinks = atm. Here you should take a look into the source code (flume-ng-sinks) = to write own plugins, feel free to submit a jira with a review request. = Please note, NG isn't ready for production yet. sorry for confusion, we all talk about flume 0.9.4, which will be = shipped with cloudera's distribution and support decorators.=20 - Alex -- Alexander Lorenz http://mapredit.blogspot.com On Mar 28, 2012, at 3:43 PM, shekhar sharma wrote: > Hello Alexander, > I have created a class known as EsperSink.java and EsperListener.java, = compiled them and created a jar(esper-sink.jar) file. >=20 > I have put the esper-sink.jar along with all esper related jar into = the lib folder (/flume-1.1.0-incubating-SNAPSHOT/lib) >=20 > And my agent file is something like this: > # Sources > agent1.sources.log.type =3D exec > agent1.sources.log.command=3D/usr/bin/tail -F = /home/dev/EsperPOC/SamplePOC/test.txt > agent1.sources.log.channels =3D log >=20 > # Channgels > agent1.channels.log.type =3D memory >=20 > # Sinks > agent1.sinks.log.type =3D Esper > agent1.sinks.log.hostname =3D namenode > agent1.sinks.log.port =3D 41414 > agent1.sinks.log.batch-size =3D 10 > agent1.sinks.log.runner.type =3D polling > agent1.sinks.log.runner.polling.interval =3D 5 > agent1.sinks.log.channel =3D log >=20 > # Load everything > agent1.sources =3D log > agent1.sinks =3D log > agent1.channels =3D log >=20 >=20 > When i am running the agent, it is throwing the following error: > 2012-03-27 20:50:50,317 (conf-file-poller-0) [ERROR - = org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRu= nnable.run(AbstractFileConfigurationProvider.java:205)] Failed to load = configuration data. Exception follows. > org.apache.flume.FlumeException: Unable to load sink type: Esper, = class: Esper >=20 > There is no file with a name flume-conf.xml only property template and = this property template is meant for defining sources and sinks. > Anyways i will create flume-conf.xml and will add the plugins >=20 > Regards, > Som >=20 > On Wed, Mar 28, 2012 at 6:59 PM, alo alt = wrote: > You have to put the them into ./lib (from a flume view) and enable the = jars in flume-conf.xml: >=20 > > flume.plugin.classes > here.the.plugins > Comma separated list of plugin classes > >=20 > -- > Alexander Lorenz > http://mapredit.blogspot.com >=20 > On Mar 28, 2012, at 12:15 PM, shekhar sharma wrote: >=20 > > Hel, > > i have written a custom sink for Esper, can you please how would i = make FLume to recognize this is my new sink..Do i have to specify in = configuration file. Can you please help me? > > > > Regards, > > Som > > > > On Wed, Mar 28, 2012 at 1:56 PM, shekhar sharma = wrote: > > Thanks for a quick response.. > > But the sinks written for cloudera flume (i was going through hello = world plugin) use cloudera classes and all.. For FLume-728 the sinks = written (HDFSSink and IRCSink) is different from cloudera's. > > CLoudera flume requires cloudera's hadoop to be installed, if i am = not wrong? > > > > My idea is to collect the events from Flume agent and send to Esper, = where i will query the streams using EPL. > > Can you please tell me what all things are required to write a = custom sink. i mean what needs to be done so that Flume node recognizes = the sink. > > > > Thanks > > > > Regards, > > Som > > > > > > On Wed, Mar 28, 2012 at 1:06 PM, alo alt = wrote: > > Hi, > > > > Cloudera uses in 3u3 Flume-798 > > > > Apache flume isn't different, you can build it among your = installation and tweak the configs to match them or use the binary = distribution: > > https://github.com/cloudera/flume/downloads > > > > As you read, apache flume 1.1.0 was just released: > > http://www.apache.org/dyn/closer.cgi/incubator/flume/ > > > > You have to write plugins for sinks, here you found some created = from the community: > > https://cwiki.apache.org/FLUME/flume-plugins.html > > > > And the docs: > > = http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_extending_vi= a_sink_source_decorator_plugins > > > > Q3 - I don't know, if you use serialized data whats with avro? > > > > - Alex > > > > -- > > Alexander Lorenz > > http://mapredit.blogspot.com > > > > On Mar 28, 2012, at 5:49 AM, shekhar sharma wrote: > > > > > Hello every one, > > > I am trying to integrate FLume and Esper by writing a custom sink = for Esper. But as per the documentation , i am unable to do so. i have = tried the hello world plugin, but that one also not working. > > > I am using FLume-728. > > > I have few question: > > > > > > Q1. Is Apache Flume is different from Cloudera's Flume? > > > If i am using Cloudera's FLume, then does it have any = dependency that i have to use CLoudera's distribution of Hadoop. > > > > > > Q2. What are the necessary steps for writing custom sink ? > > > > > > > > > Q3. I have implemented a POC in Esper, and it takes POJO (Plain = old java objects), XML or Map as events. Now if i try integrating both, = how feasible it is? > > > > > > Regards, > > > Som Shekhar Sharma > > > > > > > > > > > > >=20 >=20