mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Mahler <bmah...@apache.org>
Subject Re: Review Request 71750: Improved performance of v1 operator API GetAgents call.
Date Fri, 22 Nov 2019 17:55:54 GMT


> On Nov. 21, 2019, 4:45 p.m., Andrei Sekretenko wrote:
> > src/master/http.cpp
> > Lines 2235-2237 (patched)
> > <https://reviews.apache.org/r/71750/diff/1/?file=2172061#file2172061line2235>
> >
> >     One more thought after looking at r71751-r71754: it should be possible to precalculate
the size of the would-be `output` and serialize directly into the "top-level" `CodedOutputStream`
(in this patch: the one in getAgents), without the intermediate serialization into string
and copying it into the "top-level" stream, right?
> >     
> >     Given that the plan is to refactor the common parts later, this should probably
be left as one more TODO item...

> One more thought after looking at r71751-r71754: it should be possible to precalculate
the size of the would-be output and serialize directly into the "top-level" CodedOutputStream
(in this patch: the one in getAgents), without the intermediate serialization into string
and copying it into the "top-level" stream, right?

Yep, good observation! I originally wanted to implement it that way (no intermediate strings)
but it's quite hard since we construct temporary objects in many cases (e.g. mesos::master::Response::GetAgents::Agent
or pending task object). Even without temporary object construction, the logic needed to compute
the sub-message size is quite complex since it has to have the same loops and so on as the
actual serialization code. If there was an easy / reasonably simple way to do it, would be
nice! I was curious how much of a performance impact it would have, I guess we'll find out
somewhat when we keep an up-to-date v1 state object in memory.

If the protobuf encoding allowed fixed size ints for sub-message size (rather than mandating
varint), then you could directly serialize without knowing size, and go back to write the
size after.


- Benjamin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71750/#review218732
-----------------------------------------------------------


On Nov. 12, 2019, 12:03 a.m., Benjamin Mahler wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71750/
> -----------------------------------------------------------
> 
> (Updated Nov. 12, 2019, 12:03 a.m.)
> 
> 
> Review request for mesos, Andrei Sekretenko, Greg Mann, and Meng Zhu.
> 
> 
> Bugs: MESOS-10026
>     https://issues.apache.org/jira/browse/MESOS-10026
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> This updates the handling to serialize directly to protobuf or json
> from the in-memory v0 state, bypassing expensive intermediate
> serialization / de-serialization / object construction / object
> destruction.
> 
> This initial patch shows the approach that will be used for the
> other expensive calls. Note that this type of manual writing is
> more brittle and complex, but it can be mostly eliminated if we
> keep an up-to-date v1 GetState in memory in the future.
> 
> When this approach is applied fully to GetState, it leads to the
> following improvement:
> 
> Before:
> v0 '/state' response took 6.55 secs
> v1 'GetState' application/x-protobuf response took 24.08 secs
> v1 'GetState' application/json response took 22.76 secs
> 
> After:
> v0 '/state' response took 8.00 secs
> v1 'GetState' application/x-protobuf response took 5.73 secs
> v1 'GetState' application/json response took 9.62 secs
> 
> 
> Diffs
> -----
> 
>   src/master/http.cpp 60765c9b9d6903f6ed94fa8c614055698caad0da 
>   src/master/master.hpp dc45028d2ecfb61bf9ea82d90d2393af648a6023 
> 
> 
> Diff: https://reviews.apache.org/r/71750/diff/1/
> 
> 
> Testing
> -------
> 
> make check
> 
> 
> Thanks,
> 
> Benjamin Mahler
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message