flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Connolly <juhani_conno...@cyberagent.co.jp>
Subject Re: hdfs.idleTimeout ,what's it used for ?
Date Fri, 18 Jan 2013 06:37:17 GMT
>> @Mohit:
>> When flume dies unexpectedly the .tmp file remains. When it restarts 
>> there is some logic in HDFS sink to recover it(and continue writing 
>> from there). I'm not actually sure of the specifics. You may want to 
>> try and just kill -9 a running flume process on a test machine and 
>> then start it up, look at the logs and see what happens with the output.
> Does it also work when there is a long delay before flume gets 
> started? We are bucketing by the hr so if start occurs in the next 
> hour but flume actually died in previous hr and had  .tmp then does it 
> still cleanup on restart

I'm not sure. I think your best bet here is to simulate this on a test 
server. Start flume, after a bit kill 9 the process, wait until the 
bucket becomes invalid, and restart.

My gut feeling is that it will recover if you have events with the 
timestamp belonging to that bucket still incoming (in your persistent 
channelor read in after recovery). If that path doesn't get touched 
again though, it will probably remain as a .tmp file? *This could be 
blatantly wrong, so I suggest you test it*

View raw message