mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Bannier <benjamin.bann...@mesosphere.io>
Subject Re: Review Request 69085: Stout: Always `fsync` created directories in POSIX `mkdir`.
Date Wed, 24 Oct 2018 20:50:36 GMT


> On Oct. 22, 2018, 8:05 p.m., Benjamin Bannier wrote:
> > 3rdparty/stout/include/stout/os/posix/mkdir.hpp
> > Lines 75 (patched)
> > <https://reviews.apache.org/r/69085/diff/1/?file=2100406#file2100406line76>
> >
> >     I am not sure we should abort on the first error here. Many documented failure
scenarios of `::fsync` are either very unlikley in this scope or retryable.
> >     
> >     To get to the error code we might need to invoke `::fsync` directly.
> 
> Chun-Hung Hsiao wrote:
>     It seems to me that we should abort on any of the errors listed in http://pubs.opengroup.org/onlinepubs/9699919799/functions/fsync.html
except for `EBADF`, which cannot happen.

Reopening this for some further discussion.

Thanks for looking, these most errors there do indeed look rather non-retryable (e.g., those
related to disk space). This leaves us in a situation where `os::mkdir` could return an error,
but we are not certain (1) which directories were actually created (preexisting), and (2)
which directories actually have reached the disk (buffer). Additionally, we leave the responsibility
for cleanup to the caller, but give no indication how far we got.

While (1) is already pretty bad, it seems that a user explicitly requesting syncing (2) makes
it very hard to promise good, consistent state.

As a compromise we could get rid of `dirs_to_sync` altogether, and invoke `fsync` right after
`mkdir`. That might provide a slightly more consistent picture.


- Benjamin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69085/#review209889
-----------------------------------------------------------


On Oct. 19, 2018, 7:54 p.m., Chun-Hung Hsiao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69085/
> -----------------------------------------------------------
> 
> (Updated Oct. 19, 2018, 7:54 p.m.)
> 
> 
> Review request for mesos, Benjamin Bannier, Jie Yu, and Jan Schlicht.
> 
> 
> Bugs: MESOS-9281
>     https://issues.apache.org/jira/browse/MESOS-9281
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> To ensure the directories created by `mkdir` are commited to their
> filesystems, an `fsync` will be called on the parent of each created
> directory.
> 
> 
> Diffs
> -----
> 
>   3rdparty/stout/include/stout/os/posix/mkdir.hpp 418db9af310ed763a5ae4735c2ebdd1ca38738ba

> 
> 
> Diff: https://reviews.apache.org/r/69085/diff/1/
> 
> 
> Testing
> -------
> 
> make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message