flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun MA <mj.saber1...@gmail.com>
Subject Flafka: how to differentiate the unfinished .tmp file and the abandoned one
Date Thu, 09 Jul 2015 00:55:52 GMT
Hello Community,

I’m using Flafka (Kafka channel and HDFS sink). I met an awkward problem that I don’t
know how to determinate if a .tmp file is being written or it is been abandoned? If sink is
writing events to a file, it will have a postfix .tmp, but if the agent goes down (control
+ d) while writing to that file, it will not rename the file but left it with .tmp postfix.
When restart the agent, it will not do anything to that .tmp file. But the events in that
.tmp file is not redundant because at the kafka channel side, the offset is already committed.

So my question is that if there is a way to differentiate the working .tmp file and the finished
.tmp file?

View raw message