flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Grimes <frankgrime...@yahoo.com>
Subject Flume-NG, Avro and NodeJS
Date Wed, 15 Feb 2012 15:22:56 GMT
Hi All,

We are currently using Flume-0.9.5-SNAPSHOT to send Flume Thrift events to Node.js for realtime
monitoring of events.
To do so we are making use of the node-thrift module (https://github.com/wadey/node-thrift)
to run a Flume Thrift source within Node.js.

We were wondering if with a move to Flume-NG and/or Avro, we could configure a Flume sink
to send events as Avro JSON instead of Avro binary to ease/simplify the consumption of events
by Node.js.

Simplification aside, we're also noticing that our Node.js process uses a fair amount of CPU
to handle our current load of about 1500 events per second.
Specifically, 40%-50% with occasional spikes to 100%. (not sure what's causing the spikes
yet, perhaps V8 GC)
So we're also hoping that the latest Node.js/V8 might be able to parse JSON with less CPU
usage than node-thrift can parse Thrift.

Any ideas/suggestions are welcome!


Frank Grimes
View raw message