flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: HDFS Sink fails to rename file in Flume 1.4
Date Tue, 03 Feb 2015 22:49:55 GMT
Do you have some logs around the time the rename does not happen? We recently pushed a workaround
for this - when HDFS is unable to close the file, we were not renaming it. But since the data
is safe (it is hflushed anyway), we decided to change that and rename the file anyway: https://issues.apache.org/jira/browse/FLUME-2586

Thanks, Hari

On Tue, Feb 3, 2015 at 2:48 PM, Stefan Krawczyk <stefan@nextdoor.com>

> Hi Fellow Flume Users,
> I'm noticing that we get some "stranded" files in S3 that never get renamed
> from .tmp to .txt.
> Questions:
> 1) has this problem been fixed in 1.5, I see some jiras that might address
> the problem, but none are quite that descriptive for me to figure that out?
> 2) if this hasn't been addressed, is there a bug for it that I can track?
> Otherwise any tips or advice appreciated.
> Cheers,
> Stefan
View raw message