From reviews-return-91514-apmail-mesos-reviews-archive=mesos.apache.org@mesos.apache.org Thu Apr 2 14:59:35 2020 Return-Path: X-Original-To: apmail-mesos-reviews-archive@minotaur.apache.org Delivered-To: apmail-mesos-reviews-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by minotaur.apache.org (Postfix) with SMTP id 3B57819BB3 for ; Thu, 2 Apr 2020 14:59:35 +0000 (UTC) Received: (qmail 89041 invoked by uid 500); 2 Apr 2020 14:59:34 -0000 Delivered-To: apmail-mesos-reviews-archive@mesos.apache.org Received: (qmail 89019 invoked by uid 500); 2 Apr 2020 14:59:34 -0000 Mailing-List: contact reviews-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: reviews@mesos.apache.org Delivered-To: mailing list reviews@mesos.apache.org Received: (qmail 88946 invoked by uid 99); 2 Apr 2020 14:59:34 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2020 14:59:34 +0000 From: GitBox To: reviews@mesos.apache.org Subject: [GitHub] [mesos] cf-natali commented on a change in pull request #355: Handle EBUSY when destroying a cgroup. Message-ID: <158583957456.7250.6277817831936042520.gitbox@gitbox.apache.org> References: In-Reply-To: Date: Thu, 02 Apr 2020 14:59:34 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit cf-natali commented on a change in pull request #355: Handle EBUSY when destroying a cgroup. URL: https://github.com/apache/mesos/pull/355#discussion_r402382128 ########## File path: src/linux/cgroups.cpp ########## @@ -696,13 +696,24 @@ Try remove(const string& hierarchy, const string& cgroup) string path = path::join(hierarchy, cgroup); // Do NOT recursively remove cgroups. - Try rmdir = os::rmdir(path, false); - if (rmdir.isError()) { - return Error( + // TODO The retry was added as a fix for kernel bug + // https://lkml.org/lkml/2020/1/15/1349 + // where calling rmdir on a seemingly empty cgroup can fail + // with EBUSY while some tasks are exiting. + auto delay = Milliseconds(1); + for (auto retry = 10; ;) { + Try rmdir = os::rmdir(path, false); + if (!rmdir.isError()) { + return rmdir; + } else if (retry > 0) { + os::sleep(delay); Review comment: > XfsDiskIsolatorProcess::initialize and IOSwitchboard::_connect are examples of how process::loop() can be used to implement non-blocking retries. > The new remove function can be used in both places you have mentioned above where the blocking cgroups::remove function is called. Sorry, I don't really see how to do this in a helper function: the problem is that we need to persist state from one call to the next - e.g. the delay and current retry count - and we don't have a scope where those variables would be live. For example if the function looked like: ``` Future remove(const std::string& hierarchy, const std::string& cgroup) { const auto path = path::join(hierarchy, cgroup); auto delay = Milliseconds(1); auto retry = 10; auto result = Future(); return loop( [&]() { return process::after(delay); }, [&](const Nothing&) -> ControlFlow { if (::rmdir(path.c_str()) == 0) { return process::Break(); } else if ((errno == EBUSY) && (retry > 0)) { delay *= 2; --retry; return process::Continue(); } else { // If the `cgroup` still exists in the hierarchy, treat this as // an error; otherwise, treat this as a success since the `cgroup` // has actually been cleaned up. if (os::exists(path::join(hierarchy, cgroup))) { result = Failure( "Failed to remove directory '" + path + "': " + os::strerror(errno)); } return process::Break(); } } ); return result; } ``` The `delay`, `retry` and `result` would be reference to local variables, so invalid. `XfsDiskIsolatorProcess::initialize` and `IOSwitchboard::_connect` are both member functions of the processes on bahalf of which they're run, so don't have this problem. TBH I'm not really familiar with libprocess, maybe it would be much easier for you to write the patch? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services