flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher Shannon <cshannon...@gmail.com>
Subject Avro 2 Avro transfer on Flume 1.3 screamingly slow
Date Mon, 11 Nov 2013 15:16:30 GMT
Hello,
I am trying to get a decent throughput rate of transfer for an Avro sink to
Avro Source transfer, but the best I've been able to get is about 300
records a minute. This
thread<http://mail-archives.apache.org/mod_mbox/flume-user/201309.mbox/%3cCAHsgzwvDBDvpzR4-19DsJjTokSv0BTQrQMRn5gRJ87LcmjK7iQ@mail.gmail.com%3e>on
the mailing list suggests that hardware and disk partitioning is the
main consideration to achieving good throughput.
My setup is simple:
*Agent 1:*
Source: SpoolingDirectory
Channel: Memory
Sink: Avro
*Agent 2:*
Source: Avro
Channel: Memory
Sink: HDFS
So, what else needs to be done to optimize the data transfer?
All the best,
Chris
p.s. Anticipating suggestion that I upgrade to Flume 1.4, I work for a big
company that has acquired Flume bundled through another product from an
even bigger company. They will not consider an upgrade at this time. (And
given the very slow throughput I'm seeing, I would think that 1.3 can do a
lot better.)

Mime
View raw message