cf-natali commented on a change in pull request #355: Handle EBUSY when destroying a cgroup.
URL: https://github.com/apache/mesos/pull/355#discussion_r405615701
 
 

 ##########
 File path: src/linux/cgroups.cpp
 ##########
 @@ -696,13 +696,24 @@ Try<Nothing> remove(const string& hierarchy, const string& cgroup)
   string path = path::join(hierarchy, cgroup);
 
   // Do NOT recursively remove cgroups.
-  Try<Nothing> rmdir = os::rmdir(path, false);
-  if (rmdir.isError()) {
-    return Error(
+  // TODO The retry was added as a fix for kernel bug
+  // https://lkml.org/lkml/2020/1/15/1349
+  // where calling rmdir on a seemingly empty cgroup can fail
+  // with EBUSY while some tasks are exiting.
+  auto delay = Milliseconds(1);
+  for (auto retry = 10; ;) {
+    Try<Nothing> rmdir = os::rmdir(path, false);
+    if (!rmdir.isError()) {
+      return rmdir;
+    } else if (retry > 0) {
+      os::sleep(delay);
 
 Review comment:
   Done.
   I had to use another `loop` to chain the calls to `remove` - since we want to delete directories in order, because we proceed in bottom-up fashion.
   
   I tested in the following way:
   - make check
   - wrote a reproducer which repeatedly starts tasks with a large memory limit and exceed it, so that they are called by the OOM killer
   - check that it was failing without the fix - I've been running the same test for several hours with the fix, no failure so far
   - I also checked the rmdir handling of EBUSY with strace, checking the exponential backoff
   - I also tried to reduce the maximum retries to 2 to force it to fail, and checked that failures are correctly reported

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services