mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Mahler <bmah...@apache.org>
Subject Re: Review Request 68132: Batch '/state' requests on Master.
Date Tue, 07 Aug 2018 23:36:49 GMT


> On Aug. 1, 2018, 12:42 a.m., Benjamin Mahler wrote:
> > A couple of comments on the benchmark information before looking at the code, these
probably belong on the previous review, but since the numbers are only shown in this one I'll
leave these here:
> > 
> > * Can we compare percentiles (e.g. min, q1, q3, max) across the approaches instead
of averages? i.e. how much better does min,q1,q3,max get? Averages are generally a poor fit
for performance data because it doesn't tell us about the distribution (e.g. if we make p90
3x worse for a 10% benefit to average that's not ok), we can include p50 if we're interested
in the half-way point.
> > * Can you include the cpu model of the box you ran this on? I'm interested in how
many physical/virtual cores there are.
> > * Can you also include the regular state query benchmark measurements to make sure
we're not regressing too much on the single request case? (no need to get the non-optimized
build numbers).
> > * Some of the numbers don't look very good, e.g. Before `[min: 1.578161651secs,
max: 8.789315237secs]` After: `[4.047655443secs, 6.00752698secs]`. Can we see the distribution
here? Do you understand exactly why the lowest measurement is so much higher? Looking at the
non-optimized numbers, the minimum didn't get worse? Is the data highly variable between runs?
> > * Can you also include perf data for the optimized run? http://mesos.apache.org/documentation/latest/performance-profiling/
> 
> Alexander Rukletsov wrote:
>     * Sure. Updated https://reviews.apache.org/r/68131/ (see also preparatory work in
https://reviews.apache.org/r/68224/ and https://reviews.apache.org/r/68225/).
>     * Done.
>     * Will do, stay tuned.
>     * This is fine and expected. Due to the batching there is an extra defer in the master
queue, which affects the response time of the first request. Nothing to worry about, IMO.
>     * Will do, stay tuned.

> This is fine and expected. Due to the batching there is an extra defer in the master
queue, which affects the response time of the first request. Nothing to worry about, IMO.

Looking at the numbers for the single request benchmark case (thanks for posting those), the
batching overhead only seems to be 0.15% or 150ms (10.946s -> 11.096s). This means that
it's not just the extra defer that's causing an 85% or 1.5 second slowdown (1.512s -> 2.820s)
in the minimum request processing time. It must be something else, like the request is now
ending up in a batch (probably with more than 4 requests so that we're entering hyperthreading
territory). Let's make sure we understand why it's happening.


- Benjamin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68132/#review206711
-----------------------------------------------------------


On Aug. 7, 2018, 12:11 p.m., Alexander Rukletsov wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68132/
> -----------------------------------------------------------
> 
> (Updated Aug. 7, 2018, 12:11 p.m.)
> 
> 
> Review request for mesos, Benno Evers and Benjamin Mahler.
> 
> 
> Bugs: MESOS-9122
>     https://issues.apache.org/jira/browse/MESOS-9122
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> With this patch handlers for '/state' requests are not scheduled
> directly after authorization, but are accumulated and then scheduled
> for later parallel processing.
> 
> This approach allows, if there are N '/state' requests in the Master's
> mailbox and T is the request response time, to block the Master actor
> only once for time O(T) instead of blocking it for time N*T prior to
> this patch.
> 
> This batching technique reduces both the time Master is spending
> answering '/state' requests and the average request response time
> in presence of multiple requests in the Master's mailbox. However,
> for seldom '/state' requests that don't accumulate in the Master's
> mailbox, the response time might increase due to an added trip
> through the mailbox.
> 
> The change preserves the read-your-writes consistency model.
> 
> 
> Diffs
> -----
> 
>   src/master/http.cpp d43fbd689598612ec5946b46e2fa5e7f5e22cfa8 
>   src/master/master.hpp 209b998db8d2bad7a3812df44f0939458f48eb11 
> 
> 
> Diff: https://reviews.apache.org/r/68132/diff/2/
> 
> 
> Testing
> -------
> 
> `make check` on Mac OS 10.13.5 and various Linux distros.
> 
> Run `MasterStateQueryLoad_BENCHMARK_Test.v0State` benchmark and `MasterStateQuery_BENCHMARK_Test.GetState`,
see below.
> 
> **Setup**
> Processor: Intel i7-4980HQ 2.8 GHz with 6 MB on-chip L3 cache and 128 MB L4 cache (Crystalwell)
> Total Number of Cores: 4
> Total Number of Cores: 8
> L2 Cache (per Core): 256 KB  
> 
> Compiler: Apple LLVM version 9.1.0 (clang-902.0.39.2)
> Optimization: -O2
> 
> **MasterStateQuery_BENCHMARK_Test.GetState, v0 '/state' response time**
> 
> setup                                                    | no batching | batching
> ---------------------------------------------------------|-------------|----------
>  1000 agents,  10000 running, and  10000 completed tasks | 146.496ms   | 158.319ms
> 10000 agents, 100000 running, and 100000 completed tasks | 1.795s      | 1.899s
> 20000 agents, 200000 running, and 200000 completed tasks | 3.742s      | 4.427s
> 40000 agents, 400000 running, and 400000 completed tasks | 10.946s     | 11.096s
> 
> **MasterStateQueryLoad_BENCHMARK_Test.v0State, setup 1**
> Test setup 1: 100 agents with a total of 10000 running tasks and 10000 completed tasks;
50 '/state' and '/flags' requests will be sent in parallel with 200ms interval, i.e., total
**50 measurements** per endpoint.
> 
> /flags | no batching | batching       /state | no batching | batching
> -------------------------------   *   --------------------------------
>    min |  1.598ms    | 1.475ms           min | 100.627ms   | 105.383ms
>    p25 |  2.370ms    | 2.452ms           p25 | 102.206ms   | 107.184ms
>    p50 |  2.520ms    | 2.562ms           p50 | 103.213ms   | 108.468ms
>    p75 |  2.623ms    | 2.665ms           p75 | 104.100ms   | 109.808ms
>    p90 |  2.803ms    | 2.731ms           p90 | 106.079ms   | 111.043ms
>    max | 84.957ms    | 2.934ms           max | 153.438ms   | 154.636ms
> 
> **MasterStateQueryLoad_BENCHMARK_Test.v0State, setup 2**
> Test setup 2: 1000 agents with a total of 100000 running tasks and 100000 completed tasks;
10 '/state' and '/flags' requests will be sent in parallel with 200ms interval, i.e., total
**10 measurements** per endpoint.
> 
> /flags | no batching | batching       /state | no batching | batching
> --------------------------------  *   -------------------------------
>    min | 2.309ms     |   1.579ms         min | 1.512s      | 2.820s
>    p25 | 1.547s      | 373.609ms         p25 | 3.262s      | 3.588s
>    p50 | 3.189s      | 831.261ms         p50 | 5.052s      | 4.253s
>    p75 | 5.346s      |   2.215s          p75 | 6.846s      | 4.510s
>    p90 | 5.854s      |   2.351s          p90 | 7.883s      | 4.705s
>    max | 7.237s      |   2.444s          max | 8.517s      | 4.934s
> 
> 
> Thanks,
> 
> Alexander Rukletsov
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message