flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn Edström <bj...@spotify.com>
Subject Events silently dropped in DFO on upstream collector ERROR
Date Tue, 01 Nov 2011 09:43:44 GMT
Hello List,

I have a setup like this:

agent: source | agentDFOChain("collector1:35853", "collector2:35853")
collector1: collectorSource(35853) | collectorSink(
"hdfs://namenode.company.net:54310/flume/", "%Y_%m_%d_%H-")
collector2: collectorSource(35853) | collectorSink(
"hdfs://namenode.company.net:54310/flume/", "%Y_%m_%d_%H-")

If both collectors are running, and then collector1 gets into an ERROR
state (such as because of the DirectDriver issues discussed on this
list), events are silently dropped. No fail-over takes place to the
other node in the chain, and no events are written to disk.

Otherwise, as long as no collector is in ERROR, everything works as
expected. If I cleanly shut down collector1, the other collector will
start receiving traffic (as expected). If I then shut down collector2,
the agent will start writing data to /var/lib/flume as it can't send
the data upstream (as expected).

Is this a known issue?

Best regards
Björn Edström

View raw message