flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simone Roselli <simonerosell...@gmail.com>
Subject Flume-ng 1.6 reliable setup
Date Fri, 09 Oct 2015 00:32:42 GMT
Hi,

I'm currently plan to migrate from Flume 0.9 to Flume-ng 1.6, but I'm
having troubles trying to find a reliable setup for this one.

My sink is a 3 nodes Kafka cluster. I must avoid *to lose events in case
the main sink is down*, broken or unreachable for a while.

In Flume 0.9, I use a memory channel with the *store on failure *feature,
which starts writing events on the local disk in case the target sink is
not available.

In Flume-ng 1.6 the same behaviour would be accomplished by setting up
a *Spillable
memory channel, *but the problem with this solution is written in the end
of the channel's description: "*This channel is currently experimental and
not recommended for use in production."*

In Flume-ng 1.6, it's possible to setup a pool of *Failover sinks*. So, I
was thinking to hypothetically configure a *File Roll *as Secondary sink in
case the Primary is down. However, once the Primary sink would be back
online, the data placed on the Secondary sink (local disk) won't be
automatically pushed on the Primary one.

Another option would be setting up a *file channel*: write each event on
the disk and then sink. Without mentioning that I don't love the idea to
write/delete each single event continuously on a SSD, this setup is taking
90% of CPU. The same exactly configuration but using a memory channel takes
3%.

Other solutions to evaluate ?

Simone

Mime
View raw message