flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmytro Popovych <drudim...@gmail.com>
Subject HDFS sink: inUseSuffix for s3
Date Mon, 14 Dec 2015 16:02:24 GMT
Hi all,

I noticed that it isn’t possible to specify empty inUseSuffix for the HDFS sink.
As a result, we can't avoid a redundant rename operation for files in s3.
But afaik, s3 native filesystem keeps a temporary file on the local fs of the machine.
So, we anyway copy on s3 the final data, right?

At the same time the rename operation for s3 is copy + delete,
which is kinda useless if we already have a final data in the "temporary" file.

The question is: how can I avoid the rename operation in a HDFS sink?

Thanks in advance!

- Dmytro Popovych
View raw message