flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Williams <william...@gmail.com>
Subject HDFS S3A Sink problems
Date Wed, 16 Dec 2015 18:51:30 GMT
I've got an HDFS Sink pointed to S3 using the s3a filesystem using
flume 1.5.  It mostly works.  Occasionally, I'll see a
FileNotFoundException when it attempts to open the tmp s3a output
file.  If I look further back in the logs, I notice several
HostNotFoundExceptions which looks like it's in a retry loop of some
sort.

One curious thing is that do also see previous to this an
"IOException:  Callable timed out...".  I notice that happens on the
close of the BucketWriter. Reading into it a bit, I notice that the
tmp file appears to be deleted in a finally block in the
S3AOutputStream, which would mean this the original
FileNotFoundException is somewhat expected.   Now, obviously I can
increase the timeout but ultimately I'd loose data in this scenario
which makes me think I'm doing something wrong or there's a bug
somewhere.

Has anyone else noticed this or have some insights on this?

Thanks,
--tim

Mime
View raw message