flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Flume-NG Python avro Client
Date Tue, 24 Jul 2012 00:31:06 GMT
The root cause here appears to be that the Python library has only an
HTTPTransciever and the FlumeSource is expecting data in
"NettyTransciever" format.

On Mon, Jul 23, 2012 at 7:00 AM,  <dinesh.simkhada@gridcore.se> wrote:
> Hello Mubarak,
> Thanks for your quick reply, is there any workaround for sending data to Avro source
using python ?
>
> Thanks
> simk
>
> -----Mubarak Seyed <seyed@apple.com> wrote: -----
> To: user@flume.apache.org
> From: Mubarak Seyed <seyed@apple.com>
> Date: 07/23/2012 01:19PM
> Subject: Re: Flume-NG Python avro Client
>
> Hi,
>
> This is a known issue in Avro. Please refer https://issues.apache.org/jira/browse/AVRO-1111
>
>
>
> Thanks.
> -Mubarak
>
>
>
>
> On Jul 23, 2012, at 4:15 AM, dinesh.simkhada@gridcore.se wrote:
> Hi,
> I am trying to build a flume client in python that send message to Avro source but always
got 'java.lang.OutOfMemoryError: Java heap space'  error. I tried same thing with java it
works properly
> I have attached my source and error log, any help/suggestions exports out there ?
>
> ---------- Source
> import sys
> import httplib
> import avro.ipc as ipc
> import avro.protocol as protocol
>
> server_addr = ('192.168.104.2', 3452)
> PROTOCOL = protocol.parse(open("flume.avpr").read())
>
> def sendData():
>     client = ipc.HTTPTransceiver(server_addr[0], server_addr[1])
>     requestor = ipc.Requestor(PROTOCOL, client)
>
>     event = dict()
>     event['headers'] = {'name': 'abc', 'address': 'zyx'}
>     event['body'] = bytes('hello')
>
>     params = dict()
>     params['event'] = event
>     print("Result : " + requestor.request('append', params))
>
>     client.close()
>
>
> if __name__ == '__main__':
>     sendData()
>
>
> ----------- error msg
> 2012-07-23 10:57:54,903 WARN ipc.NettyServer: Unexpected exception from downstream.
> java.lang.OutOfMemoryError: Java heap space
>     at java.util.ArrayList.<init>(ArrayList.java:112)
>     at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:154)
>     at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:131)
>     at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
>     at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
>     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
>     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
>     at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
>     at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
>     at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:662)
>
>
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message