mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Bannier <benjamin.bann...@mesosphere.io>
Subject Re: Review Request 69954: Added a SLRP unit test for persistent block volume creation.
Date Tue, 05 Mar 2019 09:58:04 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69954/#review213431
-----------------------------------------------------------



These tests seem to have issues when executed under load. When putting extra stress on the
system with `stress-ng` I was able to get e.g., `CreateDestroyPersistentVolume` to break after
only 4 iterations
```
I0305 10:39:55.705076 58946 provider.cpp:363] Received APPLY_OPERATION event
I0305 10:39:55.705096 58946 provider.cpp:1543] Received DESTROY operation '' (uuid: 7644f49c-0210-4796-a848-dac91c9518d0)
W0305 10:39:55.705139 58946 provider.cpp:2984] Dropping operation (uuid: 7644f49c-0210-4796-a848-dac91c9518d0):
Mismatched resource version 7248406f-4bf3-4966-abbe-6d7e96886435 (expected: dd4e538a-468c-42e7-a2ce-d908801b593b)
I0305 10:39:55.705273 58937 status_update_manager_process.hpp:929] Checkpointing UPDATE for
operation status update OPERATION_FAILED (Status UUID: 229a4fa4-d07c-4c52-b842-17626d0eef11)
for operation UUID 65b93d36-132e-4cf5-9b47-762be8a47df5 of framework '62009208-e484-499c-ac45-09c08ef1cf4d-0000'
on agent 62009208-e484-499c-ac45-09c08ef1cf4»
I0305 10:39:55.708223 58982 master.cpp:9865] Sending offers [ 62009208-e484-499c-ac45-09c08ef1cf4d-O3
] to framework 62009208-e484-499c-ac45-09c08ef1cf4d-0000 (default) at scheduler-e112e00f-8102-4bc0-9613-0cea89dd77d6@66.70.182.167:41287
I0305 10:39:55.738463 58950 status_update_manager_process.hpp:152] Received operation status
update OPERATION_DROPPED (Status UUID: 4a43f3e4-147e-4fef-b7e0-a1ba73cde796) for operation
UUID 7644f49c-0210-4796-a848-dac91c9518d0 of framework '62009208-e484-499c-ac45-09c08ef1cf4d-0000'
on agent 62009208-e484-499c-ac45-09c08ef1cf4d-S0
I0305 10:39:55.738852 58950 status_update_manager_process.hpp:929] Checkpointing UPDATE for
operation status update OPERATION_DROPPED (Status UUID: 4a43f3e4-147e-4fef-b7e0-a1ba73cde796)
for operation UUID 7644f49c-0210-4796-a848-dac91c9518d0 of framework '62009208-e484-499c-ac45-09c08ef1cf4d-0000'
on agent 62009208-e484-499c-ac45-09c08ef1cf»
I0305 10:39:55.751061 58975 http.cpp:1105] HTTP POST for /slave(4)/api/v1/resource_provider
from 66.70.182.167:47378
I0305 10:39:55.752094 58975 manager.cpp:941] Received UPDATE_STATE call with resources '[{"disk":{"source":{"id":"4b391eaf-657c-43c7-8b30-985be5c54645","profile":"test","type":"BLOCK","vendor":"org.apache.mesos.csi.test.local"}},"name":"disk","provider_id":{"value":"d460f526-4461-4829-9725-c557d3d0e096"},"reservations":[{"role":"storage","t»
I0305 10:39:55.752743 58975 http.cpp:1105] HTTP POST for /slave(4)/api/v1/resource_provider
from 66.70.182.167:47378
I0305 10:39:55.759934 58975 http.cpp:1105] HTTP POST for /slave(4)/api/v1/resource_provider
from 66.70.182.167:47378
I0305 10:39:55.760082 58975 slave.cpp:8111] Handling resource provider message 'UPDATE_STATE:
d460f526-4461-4829-9725-c557d3d0e096 disk(reservations: [(DYNAMIC,storage)])[BLOCK(org.apache.mesos.csi.test.local,4b391eaf-657c-43c7-8b30-985be5c54645,test)]:4096'
I0305 10:39:55.760399 58975 slave.cpp:8231] Forwarding new total resources cpus:2; mem:1024;
disk:1024; ports:[31000-32000]; disk(reservations: [(DYNAMIC,storage)])[BLOCK(org.apache.mesos.csi.test.local,4b391eaf-657c-43c7-8b30-985be5c54645,test)]:4096
W0305 10:39:55.762451 58949 master.cpp:8349] Performing explicit reconciliation with agent
for known operation 7644f49c-0210-4796-a848-dac91c9518d0 since it was not present in original
reconciliation message from agent
I0305 10:39:55.762531 58949 master.cpp:11652] Updating the state of operation '' (uuid: 65b93d36-132e-4cf5-9b47-762be8a47df5)
for framework 62009208-e484-499c-ac45-09c08ef1cf4d-0000 (latest state: OPERATION_PENDING,
status update state: OPERATION_FAILED)
I0305 10:39:55.764708 58960 hierarchical.cpp:738] Agent 62009208-e484-499c-ac45-09c08ef1cf4d-S0
(core1.hw.ca1.mesosphere.com) updated with total resources cpus:2; mem:1024; disk:1024; ports:[31000-32000];
disk(reservations: [(DYNAMIC,storage)])[BLOCK(org.apache.mesos.csi.test.local,4b391eaf-657c-43c7-8b30-985be5c54645,test)]:4096
I0305 10:39:55.772138 58949 master.cpp:8476] Removing offer 62009208-e484-499c-ac45-09c08ef1cf4d-O3
with resources cpus(allocated: storage):2; mem(allocated: storage):1024; disk(allocated: storage):1024;
ports(allocated: storage):[31000-32000]; disk(allocated: storage)(reservations: [(DYNAMIC,storage)])[BLOCK(org.apache.mesos.csi.test.local»
I0305 10:39:55.772673 58949 master.cpp:12085] Removing offer 62009208-e484-499c-ac45-09c08ef1cf4d-O3

GMOCK WARNING:
Uninteresting mock function call - returning directly.
    Function call: offerRescinded(0x7ffd5e523d20, @0x7f8874003458 62009208-e484-499c-ac45-09c08ef1cf4d-O3)
NOTE: You can safely ignore the above warning unless this call should not happen.  Do not
suppress it by blindly adding an EXPECT_CALL() if you don't mean to enforce the call.  See
https://github.com/google/googletest/blob/master/googlemock/docs/CookBook.md#knowing-when-to-expect
for details.
I0305 10:39:55.784617 58960 master.cpp:9865] Sending offers [ 62009208-e484-499c-ac45-09c08ef1cf4d-O4
] to framework 62009208-e484-499c-ac45-09c08ef1cf4d-0000 (default) at scheduler-e112e00f-8102-4bc0-9613-0cea89dd77d6@66.70.182.167:41287
../src/tests/storage_local_resource_provider_tests.cpp:2887: Failure
Mock function called more times than expected - returning directly.
    Function call: resourceOffers(0x7ffd5e523d20, @0x7f897275f2c8 { 160-byte object <D8-9D
B6-8C 89-7F 00-00 00-00 00-00 00-00 00-00 5F-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
05-00 00-00 05-00 00-00 40-96 03-90 88-7F 00-00 00-00 00-00 00-00 00-00 00-00 00-00 00-00
00-00 ... F0-DB 01-90 88-7F 00-00 70-DA 01-90 88-7F 00-00 60-AC 02-90 8»
         Expected: to be called once
           Actual: called twice - over-saturated and active
*** Aborted at 1551778795 (unix time) try "date -d @1551778795" if you are using GNU date
***
```

We should also take care to potentially expect `offerRescinded`.

- Benjamin Bannier


On March 4, 2019, 8:23 p.m., Chun-Hung Hsiao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69954/
> -----------------------------------------------------------
> 
> (Updated March 4, 2019, 8:23 p.m.)
> 
> 
> Review request for mesos and Benjamin Bannier.
> 
> 
> Bugs: MESOS-9565
>     https://issues.apache.org/jira/browse/MESOS-9565
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Test `CreateDestroyPersistentBlockVolume` verifies that SLRP would fail
> a `CREATE` operation on a BLOCK disk resource, and a followup `DESTROY`
> will be dropped (instead of failing the SLRP).
> 
> 
> Diffs
> -----
> 
>   src/tests/mock_csi_plugin.cpp 10245705ab39872da0fef1afd02213e2c7f345cb 
>   src/tests/storage_local_resource_provider_tests.cpp a661951a0a326cc342aa0c45dd0967692ae70941

> 
> 
> Diff: https://reviews.apache.org/r/69954/diff/2/
> 
> 
> Testing
> -------
> 
> make check
> 
> 
> Thanks,
> 
> Chun-Hung Hsiao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message