flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Using Python and Flume to store avro data
Date Thu, 08 Nov 2012 18:51:10 GMT
The next release of Flume-1.3.0 adds support for an HTTP source, which will allow you to send
data to Flume via HTTP/JSON(the representation of the data is pluggable - but a JSON representation
is default). You could use this to write data to Flume from Python, which I believe has good
http and json support. 


Hari Shreedharan

On Thursday, November 8, 2012 at 10:45 AM, Bart Verwilst wrote:

> Hi,
> I've been spending quite a few hours trying to push avro data to Flume 
> so i can store it on HDFS, this all with Python.
> It seems like something that is impossible for now, since the only way 
> to push avro data to Flume is by the use of deprecated thrift binding 
> that look pretty cumbersome to get working.
> I would like to know what's the best way to import avro data into Flume 
> with Python? Maybe Flume isnt the right tool and I should use something 
> else? My goal is to have multiple python workers pushing data to HDFS 
> which ( by means of Flume in this case ) consolidates this all in 1 file 
> there.
> Any thoughts?
> Thanks!
> Bart 

View raw message