mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [mesos] abudnik commented on a change in pull request #355: Handle EBUSY when destroying a cgroup.
Date Tue, 07 Apr 2020 14:05:32 GMT
abudnik commented on a change in pull request #355: Handle EBUSY when destroying a cgroup.
URL: https://github.com/apache/mesos/pull/355#discussion_r404835680
 
 

 ##########
 File path: src/linux/cgroups.cpp
 ##########
 @@ -696,13 +696,24 @@ Try<Nothing> remove(const string& hierarchy, const string&
cgroup)
   string path = path::join(hierarchy, cgroup);
 
   // Do NOT recursively remove cgroups.
-  Try<Nothing> rmdir = os::rmdir(path, false);
-  if (rmdir.isError()) {
-    return Error(
+  // TODO The retry was added as a fix for kernel bug
+  // https://lkml.org/lkml/2020/1/15/1349
+  // where calling rmdir on a seemingly empty cgroup can fail
+  // with EBUSY while some tasks are exiting.
+  auto delay = Milliseconds(1);
+  for (auto retry = 10; ;) {
+    Try<Nothing> rmdir = os::rmdir(path, false);
+    if (!rmdir.isError()) {
+      return rmdir;
+    } else if (retry > 0) {
+      os::sleep(delay);
 
 Review comment:
   Added a few minor changes to your code:
   ```
   Future<Nothing> remove(const string& hierarchy, const string& cgroup)
   {
     // NOTE: please minimize using of `auto` keyword to keep the code consistent.
     const string path = path::join(hierarchy, cgroup);
   
     Duration delay = Milliseconds(1);
     int retry = 10;
   
     // TODO: add a comment describing the problem with the kernel bug and the way
     // we retry `rmdir` operation.
     return loop(
         // NOTE: copy `delay` by value in the first lambda function.
         [=]() mutable {
           auto timeout = process::after(delay);
           delay *= 2;
           return timeout;
         },
         // NOTE: copy `retry` by value in the second lambda function.
         [=](const Nothing&) mutable -> Future<ControlFlow<Nothing>> {
           if (::rmdir(path.c_str()) == 0) {
             return process::Break();
           } else if ((errno == EBUSY) && (retry > 0)) {
             --retry;
             return process::Continue();
           } else {
             // If the `cgroup` still exists in the hierarchy, treat this as
             // an error; otherwise, treat this as a success since the `cgroup`
             // has actually been cleaned up.
             if (os::exists(path::join(hierarchy, cgroup))) {
               return Failure(
                   "Failed to remove directory '" + path +
                   "': " + os::strerror(errno));
             }
             return process::Break();
           }
         }
       );
   }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message