mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jiang Yan Xu <...@jxu.me>
Subject Re: Review Request 50205: Enhanced benchmark test for resources to include shared resources.
Date Tue, 02 Aug 2016 15:51:03 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50205/#review144499
-----------------------------------------------------------


Ship it!




The test itself LGTM so I'll commit it. However when I ran it it does show some performance
regression on the basic scalars.

## Head
Resources benchmark on top of 6ce156b2bed7edecb27ce50f68d262ea4ab4f88e

```
[ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0
Took 72228us to perform 50000 'total += r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 101574us to perform 50000 'total -= r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 251715us to perform 50000 'total = total + r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 293414us to perform 50000 'total = total - r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
```

## Pre-shared resources
Resources benchmark on top of 48a492cd9d7d0a194735b9b4107a35b489c596e1

```
[ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0
Took 48583us to perform 50000 'total += r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 86166us to perform 50000 'total -= r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 233148us to perform 50000 'total = total + r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 265719us to perform 50000 'total = total - r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
```

'total += r' run time sees 40% jump. (I've run these 10 times and took an average, the avg
is 48685.4us (pre-shared resources) vs. 68421.7us (post-shared resources).

The overhead seems to be due we call `addable()` and `subtractable()` twice for 'total +=
r' and 'total -= r'. After I remove one of the calls, the numbers become:

Resources benchmarks on top of 6ce156b2bed7edecb27ce50f68d262ea4ab4f88e but with addable()/subtractable()
tweaks
```
[ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0
Took 48955us to perform 50000 'total += r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 85126us to perform 50000 'total -= r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 226796us to perform 50000 'total = total + r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
Took 266480us to perform 50000 'total = total - r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
```

Let's address this issue. :)

- Jiang Yan Xu


On July 30, 2016, 12:03 a.m., Anindya Sinha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50205/
> -----------------------------------------------------------
> 
> (Updated July 30, 2016, 12:03 a.m.)
> 
> 
> Review request for mesos, Klaus Ma and Jiang Yan Xu.
> 
> 
> Bugs: MESOS-4892
>     https://issues.apache.org/jira/browse/MESOS-4892
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Enhanced benchmark test for resources to include shared resources.
> 
> 
> Diffs
> -----
> 
>   src/tests/resources_tests.cpp 4111e080b84079e100b731c9a56861b204f17388 
> 
> Diff: https://reviews.apache.org/r/50205/diff/
> 
> 
> Testing
> -------
> 
> Tests passed. Results for resources benchmark is as follows:
> 
> Minimal impact seen in Resources arithmetic with the Resources refactor changes to incorporate
shared resources.
> 
> With shared resources patch (note that 4th test below is for shared resources for scalars)
> 
> [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 (806 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 (17032 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 (1048 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3 (940 ms)
> [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test (19826 ms total)
> 
> HEAD
> 
> [----------] 3 tests from ResourcesOperators/Resources_BENCHMARK_Test
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 (726 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 (17413 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 (1026 ms)
> [----------] 3 tests from ResourcesOperators/Resources_BENCHMARK_Test (19165 ms total)
> 
> Output from stdout is:
> 
> [==========] Running 4 tests from 1 test case.
> [----------] Global test environment set-up.
> [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0
> Took 72635us to perform 50000 'total += r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
> Took 116027us to perform 50000 'total -= r' operations on cpus(*):1; gpus(*):1; mem(*):128;
disk(*):256
> Took 283213us to perform 50000 'total = total + r' operations on cpus(*):1; gpus(*):1;
mem(*):128; disk(*):256
> Took 333641us to perform 50000 'total = total - r' operations on cpus(*):1; gpus(*):1;
mem(*):128; disk(*):256
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/0 (806 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1
> Took 4.103766secs to perform 10 'total += r' operations on cpus(0, principal_0, {key_0:
value_0}):1; gpus(...
> Took 4.310104secs to perform 10 'total -= r' operations on cpus(0, principal_0, {key_0:
value_0}):1; gpus(...
> Took 4186ms to perform 10 'total = total + r' operations on cpus(0, principal_0, {key_0:
value_0}):1; gpus(...
> Took 4.380353secs to perform 10 'total = total - r' operations on cpus(0, principal_0,
{key_0: value_0}):1; gpus(...
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/1 (17032 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2
> Took 483119us to perform 1000 'total += r' operations on ports(*):[1-2, 4-5, 7-8, 10-11,
13-14, 16-17, 1...
> Took 2271us to perform 1000 'total -= r' operations on ports(*):[1-2, 4-5, 7-8, 10-11,
13-14, 16-17, 1...
> Took 559844us to perform 1000 'total = total + r' operations on ports(*):[1-2, 4-5, 7-8,
10-11, 13-14, 16-17, 1...
> Took 2423us to perform 1000 'total = total - r' operations on ports(*):[1-2, 4-5, 7-8,
10-11, 13-14, 16-17, 1...
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/2 (1048 ms)
> [ RUN      ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3
> Took 61629us to perform 50000 'total += r' operations on cpus(*):1; mem(*):128; disk(test)[persistentId:...
> Took 98664us to perform 50000 'total -= r' operations on cpus(*):1; mem(*):128; disk(test)[persistentId:...
> Took 348041us to perform 50000 'total = total + r' operations on cpus(*):1; mem(*):128;
disk(test)[persistentId:...
> Took 431057us to perform 50000 'total = total - r' operations on cpus(*):1; mem(*):128;
disk(test)[persistentId:...
> [       OK ] ResourcesOperators/Resources_BENCHMARK_Test.Arithmetic/3 (940 ms)
> [----------] 4 tests from ResourcesOperators/Resources_BENCHMARK_Test (19826 ms total)
> 
> [----------] Global test environment tear-down
> [==========] 4 tests from 1 test case ran. (19836 ms total)
> [  PASSED  ] 4 tests.
> 
> 
> Thanks,
> 
> Anindya Sinha
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message