mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Greg Mann <g...@mesosphere.io>
Subject Re: Review Request 69980: Modified when master responds to operation status updates.
Date Thu, 14 Feb 2019 20:31:59 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69980/#review212838
-----------------------------------------------------------




src/master/master.cpp
Lines 8686-8708 (original), 8686-8712 (patched)
<https://reviews.apache.org/r/69980/#comment298729>

    Consider the case of a terminal-but-unacknowledged operation which has been sent to the
master by a reregistered agent and which has its ID set. Since we only place non-terminal
operations in `orphanedOperations`, we will get `frameworkWillAcknowledge == true` here. If
this framework never reregisters, the I think we could end up in a state where the agent retries
terminal updates for that operation forever.
    
    For such updates, I think the master needs to either:
    1) have a way to determine that this is a terminal-but-unacknowledged orphaned operation
(i.e. place it in `orphanedOperations`), or
    2) fall back to default behavior of acknowledging updates for operations that it doesn't
recognize.
    
    WDYT?



src/master/master.cpp
Lines 8721-8722 (patched)
<https://reviews.apache.org/r/69980/#comment298728>

    Instead of "The master will treat these...", I would suggest:
    
    "The master will drop these status updates so that subsequent retries can be handled by
the scheduler if it reregisters."


- Greg Mann


On Feb. 13, 2019, 11:23 p.m., Joseph Wu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69980/
> -----------------------------------------------------------
> 
> (Updated Feb. 13, 2019, 11:23 p.m.)
> 
> 
> Review request for mesos, Benno Evers, Gastón Kleiman, and Greg Mann.
> 
> 
> Bugs: MESOS-9542
>     https://issues.apache.org/jira/browse/MESOS-9542
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> When dealing with orphaned operation status updates, there are two
> cases the master must deal with:
> - The simple case is when the master knows the framework is completed.
>   These status updates can be acknowledged by the master.
> - However, a completed framework can be rotated out of the master's
>   memory.  In addition, after master failover, if an agent reregisters
>   before the framework, an operation can appear to be orphaned until
>   the framework reregisters.
> 
> This adds a fixed delay between agent reregistration and when the
> master acknowledges operation status updates from unknown frameworks.
> The delay should give frameworks ample time to reregister.
> 
> The delay is based on agent reregistration in order to mitigate the
> delay of acknowledging status updates of frameworks rotated out of
> the completed frameworks buffer.
> 
> 
> Diffs
> -----
> 
>   src/master/constants.hpp b0ab9187b8c672180e2ffb8b63cb7349dbe43ac4 
>   src/master/master.cpp 014e0e053cdf5c53a5ef8d63300205a121bed319 
> 
> 
> Diff: https://reviews.apache.org/r/69980/diff/1/
> 
> 
> Testing
> -------
> 
> TODO: This case needs unit tests.
> 
> 
> Thanks,
> 
> Joseph Wu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message