flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: HDFS Sink fails to rename file in Flume 1.4
Date Tue, 03 Feb 2015 23:13:40 GMT
I don’t think there is a fixed timeline yet. But I think we can start working on it soon. 




Thanks, Hari

On Tue, Feb 3, 2015 at 3:12 PM, Stefan Krawczyk <stefan@nextdoor.com>
wrote:

> Yes I have some logs, but they don't show any errors as far as I can tell;
> in comparison to other files, the log statement that the file is being
> renamed is missing.
> Do you have something in particular I should look for?
> What's the timeline for 1.6 at this stage?
> Cheers,
> Stefan
> On Tue, Feb 3, 2015 at 2:49 PM, Hari Shreedharan <hshreedharan@cloudera.com>
> wrote:
>> Do you have some logs around the time the rename does not happen? We
>> recently pushed a workaround for this - when HDFS is unable to close the
>> file, we were not renaming it. But since the data is safe (it is hflushed
>> anyway), we decided to change that and rename the file anyway:
>> https://issues.apache.org/jira/browse/FLUME-2586
>>
>>
>> Thanks,
>> Hari
>>
>>
>> On Tue, Feb 3, 2015 at 2:48 PM, Stefan Krawczyk <stefan@nextdoor.com>
>> wrote:
>>
>>> Hi Fellow Flume Users,
>>>
>>> I'm noticing that we get some "stranded" files in S3 that never get
>>> renamed from .tmp to .txt.
>>>
>>> Questions:
>>> 1) has this problem been fixed in 1.5, I see some jiras that might
>>> address the problem, but none are quite that descriptive for me to figure
>>> that out?
>>> 2) if this hasn't been addressed, is there a bug for it that I can track?
>>>
>>> Otherwise any tips or advice appreciated.
>>>
>>> Cheers,
>>>
>>> Stefan
>>>
>>
>>
Mime
View raw message