mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Peach <jpe...@apache.org>
Subject Re: Review Request 71285: Fixed recovery of agent resources and operations after crash.
Date Thu, 15 Aug 2019 05:14:49 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71285/#review217215
-----------------------------------------------------------




src/slave/slave.cpp
Lines 4430 (patched)
<https://reviews.apache.org/r/71285/#comment304503>

    I think that this comment could be improved. We can see that it is renaming, so it would
be better to discuss why we do the rename now and what it means.



src/slave/slave.cpp
Lines 4439 (patched)
<https://reviews.apache.org/r/71285/#comment304502>

    Add the error to the exit message?



src/slave/state.cpp
Lines 819 (patched)
<https://reviews.apache.org/r/71285/#comment304504>

    `state::read` can already return `None()`, so what do you think about letting it return
`None()` when the file to read isn't present?
    
    I think that this would improve the code in a number of places, but it's a large enough
change that a separate review would be better.


- James Peach


On Aug. 14, 2019, 10:35 p.m., Greg Mann wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71285/
> -----------------------------------------------------------
> 
> (Updated Aug. 14, 2019, 10:35 p.m.)
> 
> 
> Review request for mesos, Gastón Kleiman, James Peach, and Joseph Wu.
> 
> 
> Bugs: MESOS-9875
>     https://issues.apache.org/jira/browse/MESOS-9875
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Fixes an issue where the agent may incorrectly send an
> OPERATION_FINISHED update for a failed offer operation
> following agent failover and recovery.
> 
> The agent previously relied on the difference between the
> set of checkpointed operations and the set of operation
> IDs recovered from the operation status update manager to
> apply any operations which had not been applied due to an
> ill-timed agent failover.
> 
> However, this approach did not work in the case where a
> persistent volume failed to be successfully created by
> `syncCheckpointedResources()`. In order to handle this
> case, this patch changes the agent code to continue with
> the old approach of a two-phase-commit of persistent
> volumes to disk, where the agent will fail to complete
> recovery if `syncCheckpointedResources()` cannot be
> executed successfully after failover.
> 
> 
> Diffs
> -----
> 
>   src/slave/paths.hpp e077587fd02bd8e35fee7ce12ae436e3dca25e47 
>   src/slave/paths.cpp 28a7cf9f9c70fb31eeefe2e823cd7e19ffcf126a 
>   src/slave/slave.cpp 74eb45744d6603b91676e812ed008a7b1ab39a49 
>   src/slave/state.cpp cd3fac72dd57da21ed5ac46b17066531af26d42a 
> 
> 
> Diff: https://reviews.apache.org/r/71285/diff/1/
> 
> 
> Testing
> -------
> 
> Tested manually by doing the following:
> 
> 1) Run a master
> 2) Run an agent with statically reserved resources for use in the following step
> 3) Run the persistent volume example framework to create a volume on the agent, which
leads to checkpointing of resources
> 4) Use `chattr -R +i /agent/volumes/dir` to make the agent's persistent volume directory
immutable
> 5) Run the persistent volume example framework again; it will fail and the agent will
crash
> 6) Restart the agent; confirm that it continues to crash
> 7) Use `chattr -R -i /agent/volumes/dir` to remove the immutable attribute
> 8) Restart the agent; confirm that it recovers successfully, with the persistent volume
created on disk
> 9) Run the persistent volume example framework again; it should succeed
> 
> 
> Thanks,
> 
> Greg Mann
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message