From reviews-return-92211-apmail-mesos-reviews-archive=mesos.apache.org@mesos.apache.org Wed Aug 12 15:35:58 2020 Return-Path: X-Original-To: apmail-mesos-reviews-archive@locus.apache.org Delivered-To: apmail-mesos-reviews-archive@locus.apache.org Received: from mailroute1-lw-us.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by minotaur.apache.org (Postfix) with ESMTP id 698E0198C9 for ; Wed, 12 Aug 2020 15:35:56 +0000 (UTC) Received: from mail.apache.org (localhost [127.0.0.1]) by mailroute1-lw-us.apache.org (ASF Mail Server at mailroute1-lw-us.apache.org) with SMTP id 11D73125A0D for ; Wed, 12 Aug 2020 15:35:56 +0000 (UTC) Received: (qmail 74694 invoked by uid 500); 12 Aug 2020 15:35:55 -0000 Delivered-To: apmail-mesos-reviews-archive@mesos.apache.org Received: (qmail 74674 invoked by uid 500); 12 Aug 2020 15:35:55 -0000 Mailing-List: contact reviews-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: reviews@mesos.apache.org Delivered-To: mailing list reviews@mesos.apache.org Received: (qmail 74644 invoked by uid 99); 12 Aug 2020 15:35:55 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2020 15:35:55 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id CD37B18144E for ; Wed, 12 Aug 2020 15:35:54 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.464 X-Spam-Level: * X-Spam-Status: No, score=1.464 tagged_above=-999 required=6.31 tests=[HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=0.2, KAM_DMARC_STATUS=0.01, KAM_LAZY_DOMAIN_SECURITY=1, KAM_LOTSOFHASH=0.25, KHOP_HELO_FCRDNS=0.001, SPF_HELO_NONE=0.001, SPF_NONE=0.001] autolearn=disabled Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id GAp_-DKv7FrT for ; Wed, 12 Aug 2020 15:35:53 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=95.217.165.199; helo=reviews-vm-he-fi.apache.org; envelope-from=noreply@reviews.apache.org; receiver= Received: from reviews-vm-he-fi.apache.org (static.199.165.217.95.clients.your-server.de [95.217.165.199]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTP id 07D927F57C for ; Wed, 12 Aug 2020 15:35:52 +0000 (UTC) Received: from reviews-vm-he-fi.apache.org (reviews-vm-he-fi.apache.org [127.0.0.1]) by reviews-vm-he-fi.apache.org (Postfix) with ESMTP id 208D4160195; Wed, 12 Aug 2020 15:35:52 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============0189059079754549916==" MIME-Version: 1.0 Subject: Re: Review Request 72740: Introduced an interface for plugging offer filtering into the allocator. From: Andrei Sekretenko To: Benjamin Mahler Cc: Andrei Sekretenko , mesos Date: Wed, 12 Aug 2020 15:35:51 -0000 Message-ID: <20200812153551.6277.35818@reviews-vm-he-fi.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: Andrei Sekretenko X-ReviewGroup: mesos X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/72740/ X-Sender: Andrei Sekretenko References: <20200811205550.14498.29403@reviews-vm-he-fi.apache.org> In-Reply-To: <20200811205550.14498.29403@reviews-vm-he-fi.apache.org> Reply-To: Andrei Sekretenko X-ReviewRequest-Repository: mesos --===============0189059079754549916== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit > On Aug. 11, 2020, 8:55 p.m., Benjamin Mahler wrote: > > include/mesos/allocator/allocator.hpp > > Lines 72-96 (patched) > > > > > > Hm.. `AgentResourcesFilter` as a name looks a bit unintuitive since resources are not involved and I would probably think it's about including or excluding resources within an agent or something, instead of dealing with agent level inclusion / exclusion using the agent metadata. > > > > I must be missing something obvious here, but why do we need the abstract interface? Is it for testing? > > > > I also wonder whether we can just call it `OfferConstrainsFilter` to just be more direct, and then say that we need it to be abstract for testing purposes (assuming that's why). > > > > The last thought is on why we need the class, will it be holding state? Otherwise, seems like this could just be a function object? I can see that potentially being very messy though when defining variables / passing it around. I think I'll reconsider using an abstract class here (and will replace it with pImpl-based `OfferConstraintsFilter`). This object is only ever going to be constructed in the scheduler API code in the `Master`, so the only possible use of an abstract inteface could be mocking the filter object in tests. However, given that the filter implements a pure fucntion (or, in future, it might implement a set of pure functions consistent with one another), I don't see any real benefits from tests using a filter mock. Moreover, mock-based filter tests might hinder such things as implementing caching in the allocator, should we eventually need that. Why not `std::function`: 1. (major issue) If we add resource-based constraints, we will likely need to provide several filtering methods consistent with one another. `isAgentExcluded(const string& role, const SlaveInfo&)`, although likely still necessary (benchmarking attempts clearly show that agents should be filtered as early as possible), will no longer be sufficient; we will have to add `isExcluded(const string& role, const SlaveInfo&, const Resources&)` which will have to evaluate the same **attribute** constraints to apply proper resource constraints. Passing two functons and requiring that they behave consistently is not a sane thing. 2. (minor issue) std::function needs to be copyable. There are no real benefits from having the filter copyable, but there are potential benefits from having a copy disabled right from the start: should we eventually need an internal cache in the filter, a non-copyable filter will make things simpler. Nevertheless, we need a layer of indirection: otherwise, the whole definition of the filter class and all its internal structures (+ a couple of RE2 headers) get indirectly included (almost) everywhere. Looks like pImpl is a way to go. - Andrei ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72740/#review221549 ----------------------------------------------------------- On Aug. 6, 2020, 4:30 p.m., Andrei Sekretenko wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72740/ > ----------------------------------------------------------- > > (Updated Aug. 6, 2020, 4:30 p.m.) > > > Review request for mesos and Benjamin Mahler. > > > Bugs: MESOS-10171 > https://issues.apache.org/jira/browse/MESOS-10171 > > > Repository: mesos > > > Description > ------- > > This patch introduces an abstract class for a pluggable object acting > as a filter for agent resources to be offered to a framework's role, > adds this object into FrameworkOptions and wires using this object > into the allocator. > > > Diffs > ----- > > include/mesos/allocator/allocator.hpp c700528e14bb42f6cea37f42dd7b72eb76a1a6b9 > src/master/allocator/mesos/hierarchical.hpp e444e470eb085cea167f84f8540d1769d662c222 > src/master/allocator/mesos/hierarchical.cpp 9e5079942263132d09c6bd9abbdc8858cd2ef138 > src/master/master.cpp 6a013e334b19bd108791d1c5fd0864c710aae8cb > > > Diff: https://reviews.apache.org/r/72740/diff/1/ > > > Testing > ------- > > > Thanks, > > Andrei Sekretenko > > --===============0189059079754549916==--