mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anindya Sinha <anindya_si...@apple.com>
Subject Re: Review Request 51879: Determine disk size when not specified in static resources.
Date Sun, 18 Sep 2016 16:54:40 GMT


> On Sept. 15, 2016, 5:49 p.m., Jiang Yan Xu wrote:
> > The addition of `parseText` and `parseValidResources` feels unnecessary to me. I
think we can reuse `convertJSON()` to achieve the same.
> > 
> > Overall I think the following steps would be simpler and more elegant.
> > 
> > ## Patch 1: Change convertJSON() to return a Try<vector<Resource>>
> > 
> > This way the empty disks in the result are not dropped.
> > 
> > ```
> > vector<Resource> convertJSON(
> >     const JSON::Array& resourcesJSON,
> >     const string& defaultRole)
> > ```
> > 
> > I also don't see why `convertJSON()` needs to fail on empty resources since they
are allowed in the regular string formatted resources specification. Maybe we can have a **patch
0** that changes that?
> > 
> > The current lone call site can just construct `Resources` from `vector<Resource>`.
> > 
> > Then make convertJSON a public static member method.
> > 
> > ## Patch 2: Let Resources class answer if a resource is a MOUNT disk, PATH disk
or ROOT disk
> > 
> > e.g.,
> > 
> > ```
> > static bool Resources::isMountDisk(const Resource& resource);
> > ```
> > 
> > These are pretty general purpose and purely for convenience. Add simple tests to
verify these methods.
> > 
> > ## Patch 3: Autodetection for MOUNT and ROOT disks
> > 
> > In `Containerizer::resources()`, process each disk from the result of `convertJSON()`.
> > 
> > For MOUNT disks that are empty (`isEmpty()`), detect their sizes and modify the
Resource objects.
> > What about the ROOT disk? If `disk:0` is in a plain string we don't do anything.
With the JSON input the operator can choose to not specify a root disk to have it autodetected,
so I guess we don't have to separately deal with ROOT disks with an explict "zero" size. If
all disks have been processed and no ROOT disk is encountered, we auto-detect it.
> > 
> > How does it sound?

Refactored this RR into 4 separate RRs:
(1) https://reviews.apache.org/r/51999/ (Patch 1): Refactor parsing of resources to include
all valid resources.
- Modified `convertJSON()` to not fail on empty resources (now is in sync with text representation).
- Converted `convertJSON()` to return `Try<vector<Resource>>`.
- `Resources::parse()` is refactored to use a new function `Resources::parseText()` that returns
`Try<vector<Resource>>` so that existing calls to `Resources::parse()` are not
affected.
(2) https://reviews.apache.org/r/52001/: Allow text based disk resources to indicate source
type and root.
(3) https://reviews.apache.org/r/52002/ (Patch 2): Added helper methods to determine types
of disk resources.
(4) https://reviews.apache.org/r/51879/ (Patch 3): Determine disk size when not specified
in static resources.
- As discussed offline, we do not add root disk if missing when `--resources` does not include
the root disk is to allow the possibility of removing the root disk should the operator desires.


> On Sept. 15, 2016, 5:49 p.m., Jiang Yan Xu wrote:
> > src/slave/containerizer/containerizer.cpp, line 177
> > <https://reviews.apache.org/r/51879/diff/1/?file=1498166#file1498166line177>
> >
> >     Call them `disks` to avoid confusion with the volumes as a Mesos concept.

Updated based on the refactor.


> On Sept. 15, 2016, 5:49 p.m., Jiang Yan Xu wrote:
> > src/slave/containerizer/containerizer.cpp, lines 260-265
> > <https://reviews.apache.org/r/51879/diff/1/?file=1498166#file1498166line260>
> >
> >     This is only for the ROOT disk, i.e., we only leave some headroom for the system
root partition.
> >     
> >     Therefore we probably don't need this method.

Agreed. However, we have 2 cases for rot disks:
a. When no disks are present in `--resources`
b. When `--resources` indicates a disk without source.

I refactored `Containerizer::diskSize()` to take in if it is a root disk and add the headroom
for root disks only.


- Anindya


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51879/#review149033
-----------------------------------------------------------


On Sept. 14, 2016, 2:29 a.m., Anindya Sinha wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51879/
> -----------------------------------------------------------
> 
> (Updated Sept. 14, 2016, 2:29 a.m.)
> 
> 
> Review request for mesos and Jiang Yan Xu.
> 
> 
> Bugs: MESOS-6062
>     https://issues.apache.org/jira/browse/MESOS-6062
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> When static resources indicate disks with a positive size, we use that
> for the disk resources on the agent. However, --resources can include
> disks with size of 0, which indicates that mesos agent determine the
> size of those disks from the host and uses that information instead.
> Note that this is not allowed for PATH disks since PATH disks can be
> carved into smaller chunks and we cannot assume that the whole
> physical disk is available to the PATH disk.
> 
> With this change, JSON or textual representation for resources that
> specify scalar value of 0 would not result in an error, but those
> resources will not be accounted for until a valid size is determined
> for such resources. A scalar value of -1 in JSON or textual formats
> still results in an invalid resource.
> 
> 
> Diffs
> -----
> 
>   include/mesos/resources.hpp 7ba422d57d47777e058c682df9aa780557782423 
>   include/mesos/v1/resources.hpp add48c7baf8fabf5cc443d60a6c96d2902fc67de 
>   src/common/resources.cpp f841e91b81975a887e64a76a709708ed7946025f 
>   src/slave/containerizer/containerizer.hpp f13669d0dfc4ce3287cfe630cabd0470dc765b51

>   src/slave/containerizer/containerizer.cpp d46882baa904fd439bffb23c324828b777228f1c

>   src/tests/resources_tests.cpp f627862870279e5a778aa83e1ddcaa88c27ba1cb 
>   src/v1/resources.cpp 4410a8f95ef0ebe6ec7aee38cd47f6f83f863867 
> 
> Diff: https://reviews.apache.org/r/51879/diff/
> 
> 
> Testing
> -------
> 
> Tests passed.
> 
> 
> Thanks,
> 
> Anindya Sinha
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message