From user-return-1782-apmail-flume-user-archive=flume.apache.org@flume.apache.org Tue Jul 24 00:31:54 2012 Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6454F9582 for ; Tue, 24 Jul 2012 00:31:54 +0000 (UTC) Received: (qmail 28935 invoked by uid 500); 24 Jul 2012 00:31:54 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 28878 invoked by uid 500); 24 Jul 2012 00:31:54 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 28870 invoked by uid 99); 24 Jul 2012 00:31:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Jul 2012 00:31:54 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of brock@cloudera.com designates 209.85.212.51 as permitted sender) Received: from [209.85.212.51] (HELO mail-vb0-f51.google.com) (209.85.212.51) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Jul 2012 00:31:47 +0000 Received: by vbip1 with SMTP id p1so5245952vbi.38 for ; Mon, 23 Jul 2012 17:31:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=cmAtJyJ8mu8kw2EGUqNlNvy0BDfXgE/tBOjK+dUNNFo=; b=dWnOaPfivN4tzRQhKVdc+jpOOGBPZfRvmL5wNpOcEQFw2qsj/vzMVEoCPw92Em8ekd zzNHLG3RXJ2x5WbHP+0OiK7Z3QUdNX285IT4vbssoOK4Mj1lfkk3F8KIoHLEcFA4xhO/ HKqXg2/Xind66ZWxDI7RllgIc+FvswR0sg0pLX4OEpSDpPTQb6GSxJgXbU/zUfHU8OUx 4Xu9k7woy6Y25bSr1ygSrvbco80aEu/v6pqON2pmLsDWWN+wdp9NJ6qo+ikQ0MP41i9t nwjplUreXLyetkW31EF4DXRP1AyS3no02cJqP9wsyzQlMBWwZ1XwIDrI4Hn7sp6gOY6a +BGg== Received: by 10.220.223.201 with SMTP id il9mr14256764vcb.64.1343089886272; Mon, 23 Jul 2012 17:31:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.220.18.195 with HTTP; Mon, 23 Jul 2012 17:31:06 -0700 (PDT) In-Reply-To: References: <23353898-65DA-49A5-9827-1CAAB269F9EF@apple.com> From: Brock Noland Date: Mon, 23 Jul 2012 19:31:06 -0500 Message-ID: Subject: Re: Flume-NG Python avro Client To: user@flume.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQlWH5LIoEBUULjJHhutR+cs1DvEX0y/buwFjvvATR6BbkDQq+2e5g4Jze0ZInpoUmv+2Mla The root cause here appears to be that the Python library has only an HTTPTransciever and the FlumeSource is expecting data in "NettyTransciever" format. On Mon, Jul 23, 2012 at 7:00 AM, wrote: > Hello Mubarak, > Thanks for your quick reply, is there any workaround for sending data to Avro source using python ? > > Thanks > simk > > -----Mubarak Seyed wrote: ----- > To: user@flume.apache.org > From: Mubarak Seyed > Date: 07/23/2012 01:19PM > Subject: Re: Flume-NG Python avro Client > > Hi, > > This is a known issue in Avro. Please refer https://issues.apache.org/jira/browse/AVRO-1111 > > > > Thanks. > -Mubarak > > > > > On Jul 23, 2012, at 4:15 AM, dinesh.simkhada@gridcore.se wrote: > Hi, > I am trying to build a flume client in python that send message to Avro source but always got 'java.lang.OutOfMemoryError: Java heap space' error. I tried same thing with java it works properly > I have attached my source and error log, any help/suggestions exports out there ? > > ---------- Source > import sys > import httplib > import avro.ipc as ipc > import avro.protocol as protocol > > server_addr = ('192.168.104.2', 3452) > PROTOCOL = protocol.parse(open("flume.avpr").read()) > > def sendData(): > client = ipc.HTTPTransceiver(server_addr[0], server_addr[1]) > requestor = ipc.Requestor(PROTOCOL, client) > > event = dict() > event['headers'] = {'name': 'abc', 'address': 'zyx'} > event['body'] = bytes('hello') > > params = dict() > params['event'] = event > print("Result : " + requestor.request('append', params)) > > client.close() > > > if __name__ == '__main__': > sendData() > > > ----------- error msg > 2012-07-23 10:57:54,903 WARN ipc.NettyServer: Unexpected exception from downstream. > java.lang.OutOfMemoryError: Java heap space > at java.util.ArrayList.(ArrayList.java:112) > at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154) > at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131) > at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282) > at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) > at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) > at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351) > at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282) > at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > > > -- Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/