mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Mahler <>
Subject Review Request 68903: Avoid deadlock-prone blocking in master's parallel endpoint serving.
Date Wed, 03 Oct 2018 00:07:53 GMT

This is an automatically generated e-mail. To reply, visit:

Review request for mesos, Alexander Rukletsov and Benno Evers.

Repository: mesos


The approach taken to serve endpoints in parallel was to call
`process::async` for each request and block until they finish.
Blocking in this case assumes that there will be enough additional
non-blocked worker threads to execute all the requests. When the
number of blocking Processes is equal or greater than the number
of worker threads, deadlock become possible. The parallel endpoint
serving approach added another blocking Process.

While we could make blocking safe (see MESOS-8256) by, for example,
detaching a worker thread prior to blocking (a la golang), such a
change is more involved.

In lieu of making blocking safe, this patch updates the Master to
block only when it's guaranteed to be safe. This is accomplished

  (1) using a "work queue" (just a shared atomic "next index") and
      having workers pop items from this "queue",
  (2) we execute one of the workers on the master Process directly,
  (3) when this worker completes, we then know that all requests
      are either processed or being processed (i.e. the `async`
      `Process` is executing on a worker thread),
  (4) the master can then block on a count down latch, knowing
      that it will unblock because of (3).


  src/master/http.cpp d912789bcfdcf6eaefc29dba7b878fa4e02f9276 




Benjamin Mahler

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message