mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Mahler <bmah...@apache.org>
Subject Re: Review Request 49694: Filter out fully used agents before allocate resources.
Date Thu, 07 Jul 2016 19:08:18 GMT


> On July 6, 2016, 9:17 p.m., Benjamin Mahler wrote:
> > src/master/allocator/mesos/hierarchical.cpp, lines 1220-1228
> > <https://reviews.apache.org/r/49694/diff/1/?file=1437671#file1437671line1220>
> >
> >     I think the intention of this filtering was that the allocation loop needs to
ignore agents that are not whitelisted or that are not activated, because otherwise it would
send offers for them.
> >     
> >     If the agent doesn't have free resources, that's the intention of checking `!allocatable(resources)`
when we're trying to offer resources below. Technically, you should be checking the same here.
> >     
> >     I'm confused by this patch since you showed that it doesn't help performance
and this isn't necessary. Why did you send it?
> 
> Guangya Liu wrote:
>     Hi Ben, I did more test with 4000, 8000, 12000 fully used agent and 2000 non-fully
used agent, the test result showed that there are indeed some performnace improvement with
the number of fully used agent increase.
>     
>     1) 6000 agents (4000 fully used + 2000 non-fully used)
>     
>     Before fix:
>     Using 6000 agents and 200 frameworks with 4000 agents fully used
>     round 0 allocate took 3.224707secs to make 200 offers
>     round 1 allocate took 3.205996secs to make 200 offers
>     round 2 allocate took 3.203761secs to make 200 offers
>     round 3 allocate took 3.520678secs to make 200 offers
>     round 4 allocate took 3.599001secs to make 200 offers
>     round 5 allocate took 3.437631secs to make 200 offers
>     round 6 allocate took 3.197855secs to make 200 offers
>     round 7 allocate took 3.210213secs to make 200 offers
>     round 8 allocate took 3.482421secs to make 200 offers
>     round 9 allocate took 3.495999secs to make 200 offers
>     round 10 allocate took 3.270022secs to make 200 offers
>     round 11 allocate took 3.303073secs to make 200 offers
>     round 12 allocate took 3.574657secs to make 200 offers
>     round 13 allocate took 3.20638secs to make 200 offers
>     round 14 allocate took 3.644309secs to make 200 offers
>     round 15 allocate took 3.22866secs to make 200 offers
>     round 16 allocate took 3.234343secs to make 200 offers
>     round 17 allocate took 3.457045secs to make 200 offers
>     round 18 allocate took 3.30394secs to make 200 offers
>     round 19 allocate took 3.323927secs to make 200 offers
>     round 20 allocate took 3.477471secs to make 200 offers
>     round 21 allocate took 3.494607secs to make 200 offers
>     round 22 allocate took 3.330647secs to make 200 offers
>     
>     
>     After Fix:
>     [ RUN      ] HierarchicalAllocator_BENCHMARK_Test.FilterOutFullyUsedAgent
>     Using 6000 agents and 200 frameworks with 4000 agents fully used
>     round 0 allocate took 3.101395secs to make 200 offers
>     round 1 allocate took 3.076541secs to make 200 offers
>     round 2 allocate took 3.054527secs to make 200 offers
>     round 3 allocate took 3.089352secs to make 200 offers
>     round 4 allocate took 3.670608secs to make 200 offers
>     round 5 allocate took 3.343736secs to make 200 offers
>     round 6 allocate took 3.268496secs to make 200 offers
>     round 7 allocate took 3.313199secs to make 200 offers
>     round 8 allocate took 3.269126secs to make 200 offers
>     round 9 allocate took 3.219127secs to make 200 offers
>     round 10 allocate took 3.259214secs to make 200 offers
>     round 11 allocate took 3.825867secs to make 200 offers
>     round 12 allocate took 3.907938secs to make 200 offers
>     round 13 allocate took 3.380813secs to make 200 offers
>     round 14 allocate took 3.116237secs to make 200 offers
>     round 15 allocate took 3.140145secs to make 200 offers
>     round 16 allocate took 3.059795secs to make 200 offers
>     round 17 allocate took 3.164924secs to make 200 offers
>     round 18 allocate took 3.424397secs to make 200 offers
>     round 19 allocate took 3.12893secs to make 200 offers
>     round 20 allocate took 3.101884secs to make 200 offers
>     
>     2) 10000 agents (8000 fully used + 2000 non-fully used)
>     
>     Before Fix:
>     Using 10000 agents and 200 frameworks with 8000 agents fully used
>     round 0 allocate took 3.372285secs to make 200 offers
>     round 1 allocate took 3.411058secs to make 200 offers
>     round 2 allocate took 3.3929secs to make 200 offers
>     round 3 allocate took 3.412163secs to make 200 offers
>     round 4 allocate took 3.434624secs to make 200 offers
>     round 5 allocate took 3.774675secs to make 200 offers
>     round 6 allocate took 3.570071secs to make 200 offers
>     round 7 allocate took 3.400986secs to make 200 offers
>     round 8 allocate took 3.438448secs to make 200 offers
>     round 9 allocate took 3.407503secs to make 200 offers
>     round 10 allocate took 3.463708secs to make 200 offers
>     round 11 allocate took 3.990451secs to make 200 offers
>     round 12 allocate took 4.181017secs to make 200 offers
>     round 13 allocate took 3.7594secs to make 200 offers
>     round 14 allocate took 3.76082secs to make 200 offers
>     round 15 allocate took 3.933429secs to make 200 offers
>     round 16 allocate took 3.559281secs to make 200 offers
>     round 17 allocate took 3.45657secs to make 200 offers
>     round 18 allocate took 3.5156secs to make 200 offers
>     round 19 allocate took 3.448579secs to make 200 offers
>     round 20 allocate took 3.517771secs to make 200 offers
>     
>     After: Fix:
>     Using 10000 agents and 200 frameworks with 8000 agents fully used
>     round 0 allocate took 3.151761secs to make 200 offers
>     round 1 allocate took 3.188501secs to make 200 offers
>     round 2 allocate took 3.165528secs to make 200 offers
>     round 3 allocate took 3.155098secs to make 200 offers
>     round 4 allocate took 3.124131secs to make 200 offers
>     round 5 allocate took 3.169916secs to make 200 offers
>     round 6 allocate took 3.212185secs to make 200 offers
>     round 7 allocate took 3.161282secs to make 200 offers
>     round 8 allocate took 3.179652secs to make 200 offers
>     round 9 allocate took 3.463299secs to make 200 offers
>     round 10 allocate took 3.285085secs to make 200 offers
>     round 11 allocate took 3.20183secs to make 200 offers
>     round 12 allocate took 3.31525secs to make 200 offers
>     round 13 allocate took 3.566602secs to make 200 offers
>     round 14 allocate took 3.33295secs to make 200 offers
>     round 15 allocate took 3.368531secs to make 200 offers
>     round 16 allocate took 3.383993secs to make 200 offers
>     round 17 allocate took 3.408238secs to make 200 offers
>     round 18 allocate took 3.264405secs to make 200 offers
>     round 19 allocate took 3.258754secs to make 200 offers
>     round 20 allocate took 3.349725secs to make 200 offers
>     round 21 allocate took 3.246381secs to make 200 offers
>     
>     3) 14000 agents (12000 fully used + 2000 non-fully used)
>     
>     Before fix:
>     Using 14000 agents and 200 frameworks with 12000 agents fully used
>     round 0 allocate took 3.830423secs to make 200 offers
>     round 1 allocate took 4.14633secs to make 200 offers
>     round 2 allocate took 4.312427secs to make 200 offers
>     round 3 allocate took 3.802955secs to make 200 offers
>     round 4 allocate took 3.851801secs to make 200 offers
>     round 5 allocate took 3.837912secs to make 200 offers
>     round 6 allocate took 3.861083secs to make 200 offers
>     round 7 allocate took 3.823863secs to make 200 offers
>     round 8 allocate took 4.199612secs to make 200 offers
>     round 9 allocate took 4.489062secs to make 200 offers
>     round 10 allocate took 3.818206secs to make 200 offers
>     round 11 allocate took 3.908532secs to make 200 offers
>     round 12 allocate took 4.407943secs to make 200 offers
>     round 13 allocate took 3.934236secs to make 200 offers
>     round 14 allocate took 3.878556secs to make 200 offers
>     round 15 allocate took 3.968476secs to make 200 offers
>     round 16 allocate took 4.061709secs to make 200 offers
>     round 17 allocate took 4.073742secs to make 200 offers
>     round 18 allocate took 3.947496secs to make 200 offers
>     round 19 allocate took 4.142477secs to make 200 offers
>     round 20 allocate took 4.392632secs to make 200 offers
>     
>     After fix:
>     Using 14000 agents and 200 frameworks with 12000 agents fully used
>     round 0 allocate took 3.366275secs to make 200 offers
>     round 1 allocate took 3.21032secs to make 200 offers
>     round 2 allocate took 3.185639secs to make 200 offers
>     round 3 allocate took 3.171451secs to make 200 offers
>     round 4 allocate took 3.178399secs to make 200 offers
>     round 5 allocate took 3.776151secs to make 200 offers
>     round 6 allocate took 4.083116secs to make 200 offers
>     round 7 allocate took 3.413082secs to make 200 offers
>     round 8 allocate took 3.198207secs to make 200 offers
>     round 9 allocate took 3.793907secs to make 200 offers
>     round 10 allocate took 3.239683secs to make 200 offers
>     round 11 allocate took 3.224455secs to make 200 offers
>     round 12 allocate took 3.253067secs to make 200 offers
>     round 13 allocate took 3.249255secs to make 200 offers
>     round 14 allocate took 3.422576secs to make 200 offers
>     round 15 allocate took 3.355821secs to make 200 offers
>     round 16 allocate took 3.266337secs to make 200 offers
>     round 17 allocate took 3.276855secs to make 200 offers
>     round 18 allocate took 3.531488secs to make 200 offers
>     round 19 allocate took 3.319291secs to make 200 offers
>     round 20 allocate took 3.504048secs to make 200 offers
>     round 21 allocate took 3.325912secs to make 200 offers
>     round 22 allocate took 3.43463secs to make 200 offers
>     round 23 allocate took 3.239212secs to make 200 offers

The improvement here seems to be around 10%? This patch only helps once there are a large
number of fully used agents, otherwise it actually makes this a bit slower due to the additional
check! In most practical cases (small or medium amount of fully used agents due to bin packing
inefficiency) this will actually slow things down slightly.

So, I'd rather focus on improvements that offer large improvements to a broad variety of scenarios.
For example, by improving the performance of our 'Resources' abstraction. Make sense?


- Benjamin


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49694/#review141075
-----------------------------------------------------------


On July 6, 2016, 2:53 p.m., Guangya Liu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49694/
> -----------------------------------------------------------
> 
> (Updated July 6, 2016, 2:53 p.m.)
> 
> 
> Review request for mesos, Benjamin Mahler and Joris Van Remoortere.
> 
> 
> Bugs: MESOS-5017
>     https://issues.apache.org/jira/browse/MESOS-5017
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> Filter out fully used agents before allocate resources.
> 
> 
> Diffs
> -----
> 
>   src/master/allocator/mesos/hierarchical.cpp c1e00039606164599e25ff5f76245e4d35ec3112

>   src/tests/hierarchical_allocator_tests.cpp 0498cd5e54b0e4b87a767585a77699653aa52179

> 
> Diff: https://reviews.apache.org/r/49694/diff/
> 
> 
> Testing
> -------
> 
> make
> make check
> 
> I found that the performance actually does not have too much difference with this patch.
> 
> Without this patch:
> Using 4000 agents and 200 frameworks with 2000 agents fully used
> round 0 allocate took 3.077414secs to make 200 offers
> round 1 allocate took 3.201985secs to make 200 offers
> round 2 allocate took 3.375356secs to make 200 offers
> round 3 allocate took 3.357918secs to make 200 offers
> round 4 allocate took 3.253266secs to make 200 offers
> round 5 allocate took 3.09918secs to make 200 offers
> round 6 allocate took 3.298442secs to make 200 offers
> round 7 allocate took 3.323613secs to make 200 offers
> round 8 allocate took 3.072355secs to make 200 offers
> round 9 allocate took 3.271617secs to make 200 offers
> round 10 allocate took 3.126364secs to make 200 offers
> round 11 allocate took 3.118216secs to make 200 offers
> round 12 allocate took 3.088154secs to make 200 offers
> round 13 allocate took 3.215559secs to make 200 offers
> round 14 allocate took 3.251781secs to make 200 offers
> round 15 allocate took 3.168862secs to make 200 offers
> 
> With this patch:
> Using 4000 agents and 200 frameworks with 2000 agents fully used
> round 0 allocate took 3.536161secs to make 200 offers
> round 1 allocate took 3.061345secs to make 200 offers
> round 2 allocate took 3.061557secs to make 200 offers
> round 3 allocate took 3.12996secs to make 200 offers
> round 4 allocate took 3.124199secs to make 200 offers
> round 5 allocate took 3.05091secs to make 200 offers
> round 6 allocate took 3.072797secs to make 200 offers
> round 7 allocate took 3.167616secs to make 200 offers
> round 8 allocate took 3.259008secs to make 200 offers
> round 9 allocate took 3.2203secs to make 200 offers
> round 10 allocate took 3.244719secs to make 200 offers
> round 11 allocate took 3.258939secs to make 200 offers
> round 12 allocate took 3.225309secs to make 200 offers
> round 13 allocate took 3.094871secs to make 200 offers
> round 14 allocate took 3.27408secs to make 200 offers
> round 15 allocate took 3.143926secs to make 200 offers
> 
> 
> Thanks,
> 
> Guangya Liu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message