flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Question Failure Behavior of HDFS Sink
Date Tue, 08 Sep 2015 11:10:08 GMT
as I understand it the HDFS sink uses the transaction system to verify that
all the elements in a transaction are written. This is what I would call
at-least-once semantics.

My question is now what happens if the writing fails in the middle of
writing the elements in the transaction. When the transaction is retried
some of the elements might be written again, i.e. the output contains
duplicates. Is this assumption correct or is there something in place that
prevents this from happening?

Thanks for your time,

View raw message