mesos-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ilya Pronin <ipro...@twopensource.com>
Subject Re: Review Request 73136: Added support for multiple event loop / IO threads for libev.
Date Fri, 22 Jan 2021 19:56:38 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/73136/#review222515
-----------------------------------------------------------


Ship it!




Ship It!

- Ilya Pronin


On Jan. 21, 2021, 10:51 a.m., Benjamin Mahler wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/73136/
> -----------------------------------------------------------
> 
> (Updated Jan. 21, 2021, 10:51 a.m.)
> 
> 
> Review request for mesos and Ilya Pronin.
> 
> 
> Bugs: MESOS-10208
>     https://issues.apache.org/jira/browse/MESOS-10208
> 
> 
> Repository: mesos
> 
> 
> Description
> -------
> 
> The current approach to I/O in libprocess, with a single thread
> performing all of the the I/O polling and I/O syscalls, cannot keep
> up with the I/O load on massive scale mesos clusters (which use
> libev rather than libevent).
> 
> This adds support via a LIBPROCESS_LIBEV_NUM_IO_THREADS env variable
> for configuring the number of threads running libev event loops,
> which allows users to spread the IO load across multiple threads.
> 
> 
> Diffs
> -----
> 
>   3rdparty/libprocess/src/posix/libev/libev.hpp d451931871db650894e4a6e5b0d19ba876f65391

>   3rdparty/libprocess/src/posix/libev/libev.cpp b38e7a0f882a8c24950bdc6fd74a4d25fc68549e

>   3rdparty/libprocess/src/posix/libev/libev_poll.cpp 96913a65507ca3540066e28448684d1e3fa540ca

> 
> 
> Diff: https://reviews.apache.org/r/73136/diff/3/
> 
> 
> Testing
> -------
> 
> So far only manual testing:
> 
> ```
> # Error
> $ make check -j16 TEST_DRIVER="" GTEST_FILTER="-ProcessRemoteLinkTest.RemoteLinkLeak"
LIBPROCESS_LIBEV_NUM_IO_THREADS=0
> $ make check -j16 TEST_DRIVER="" GTEST_FILTER="-ProcessRemoteLinkTest.RemoteLinkLeak"
LIBPROCESS_LIBEV_NUM_IO_THREADS=1025
> 
> # Success
> $ make check -j16 TEST_DRIVER="" GTEST_FILTER="-ProcessRemoteLinkTest.RemoteLinkLeak"
LIBPROCESS_LIBEV_NUM_IO_THREADS=1
> $ make check -j16 TEST_DRIVER="" GTEST_FILTER="-ProcessRemoteLinkTest.RemoteLinkLeak"
LIBPROCESS_LIBEV_NUM_IO_THREADS=32
> ```
> 
> Will follow up with some test(s) that leverage the reinitialize support in libprocess,
so that the testing doesn't need to be done manually.
> 
> 
> Thanks,
> 
> Benjamin Mahler
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message