mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Mann <>
Subject Re: Review Request 71285: Fixed recovery of agent resources and operations after crash.
Date Wed, 14 Aug 2019 22:35:30 GMT

This is an automatically generated e-mail. To reply, visit:

(Updated Aug. 14, 2019, 10:35 p.m.)

Review request for mesos, Gastón Kleiman, James Peach, and Joseph Wu.

Bugs: MESOS-9875

Repository: mesos


Fixes an issue where the agent may incorrectly send an
OPERATION_FINISHED update for a failed offer operation
following agent failover and recovery.

The agent previously relied on the difference between the
set of checkpointed operations and the set of operation
IDs recovered from the operation status update manager to
apply any operations which had not been applied due to an
ill-timed agent failover.

However, this approach did not work in the case where a
persistent volume failed to be successfully created by
`syncCheckpointedResources()`. In order to handle this
case, this patch changes the agent code to continue with
the old approach of a two-phase-commit of persistent
volumes to disk, where the agent will fail to complete
recovery if `syncCheckpointedResources()` cannot be
executed successfully after failover.


  src/slave/paths.hpp e077587fd02bd8e35fee7ce12ae436e3dca25e47 
  src/slave/paths.cpp 28a7cf9f9c70fb31eeefe2e823cd7e19ffcf126a 
  src/slave/slave.cpp 74eb45744d6603b91676e812ed008a7b1ab39a49 
  src/slave/state.cpp cd3fac72dd57da21ed5ac46b17066531af26d42a 


Testing (updated)

Tested manually by doing the following:

1) Run a master
2) Run an agent with statically reserved resources for use in the following step
3) Run the persistent volume example framework to create a volume on the agent, which leads
to checkpointing of resources
4) Use `chattr -R +i /agent/volumes/dir` to make the agent's persistent volume directory immutable
5) Run the persistent volume example framework again; it will fail and the agent will crash
6) Restart the agent; confirm that it continues to crash
7) Use `chattr -R -i /agent/volumes/dir` to remove the immutable attribute
8) Restart the agent; confirm that it recovers successfully, with the persistent volume created
on disk
9) Run the persistent volume example framework again; it should succeed


Greg Mann

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message