incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Fisher <w...@apache.org>
Subject Re: [DISCUSS] YuniKorn Proposal
Date Wed, 11 Dec 2019 20:07:31 GMT
Hi -

Very interesting proposal. +1.

Please note that two of your nominated mentors will need to ask to join the IPMC by request
at private@i.a.o

Regards,
Dave

> On Dec 11, 2019, at 11:50 AM, Weiwei Yang <wwei@apache.org> wrote:
> 
> Greetings folks:
> 
> Please consider the following proposal, which is also on the wiki
> <https://cwiki.apache.org/confluence/display/INCUBATOR/YuniKornProposal>
> [1].
> I look forward to hearing feedback from you.
> 
> 
> *YuniKorn - An Unified Resource Scheduler*
> 
> 1. Abstract
> YuniKorn is a standalone resource scheduler responsible for scheduling
> batch jobs and long-running services on large scale distributed systems
> running in on-premises environments as well as different public clouds.
> 
> 2. Proposal
> YuniKorn ['ju:nikɔ:n] is a unified resource scheduler aiming to achieve
> fine-grained resource sharing for various workloads efficiently on a large
> scale, multi-tenant and cloud-native environments. YuniKorn brings a
> unified, cross-platform scheduling experience for mixed workloads, with
> support for but not limited to, Apache™ Hadoop® YARN and Kubernetes.
> 
> Currently, YuniKorn is an open-source project with Apache 2.0 license. The
> source code is hosted as a git-repo under the github.com/cloudera domain.
> We would like to share it with the ASF and expand the community to a wider
> range of users and contributors.
> 
> 2.1 Background
> Enterprise users run their workloads on different platforms such as Apache™
> Hadoop® YARN and Kubernetes. They need to work with different resource
> schedulers in order to plan their workloads to run on these platforms
> efficiently. The scheduler implementations are fragmented, and not
> optimized to balance existing use-cases like batch workloads along with new
> needs such as cloud-native architecture, autoscaling, etc. We need a single
> resource planning/management framework to manage resources on different
> platforms using the same semantics, in order to address all the important
> resource management requirements.
> 
> 2.2 Rationale
> There is no solution that exists now to address the needs of having a
> unified resource scheduling experiences across platforms. That makes it
> difficult to manage workloads running on different environments, from
> on-premise to Cloud. YuniKorn aims to satisfy these needs. YuniKorn is
> designed around the following principles:
> 
> 1) Support different environments
> As the compute platforms are evolving quickly, there are more and more
> challenges appear in on-prem, cloud or hybrid environments. YuniKorn aims
> to bring unified scheduling experiences across multiple environments with
> enhanced scheduling capabilities.
> 
> 2) Support extensive types of workloads
> To improve the efficiency of the computing platform, a key idea is to run
> different types of applications, like long-running services and batch jobs,
> on shared resources. YuniKorn is an effort to address all the scheduling
> features needed for such mixed workload environments.
> 
> 3) Benefit both big-data and cloud-native communities
> A resource scheduler needs to be capable of supporting mixed workloads,
> both batch, and long-running services. This is the key to improving cluster
> utilization, and to reduce the complexity of dev-ops. By creating a common
> scheduler that is decoupled from the container platforms underneath, it can
> benefit both Apache™ Hadoop® YARN and the Kubernetes communities.
> 
> 2.3 Initial Goals
> Initial goals are:
> - Move the existing codebase, documentation to Apache hosted repo
> - Set up mailing lists, website, CI/CD pipeline under Apache infrastructure
> - Setup JIRA for issue tracking
> - Incremental development and releases according to Apache guidelines
> - Expand the community and bring more diversified contributors/users to
> the community
> 
> 2.4 Current Status
> 
> 2.4.1 Meritocracy
> Many of the initial developers of YuniKorn are already Apache committers
> and PMC members from other Apache projects, such as Apache Hadoop and
> Apache Submarine. Many of us have worked in the Apache Hadoop community for
> years and know the Apache way well. We believe strongly in meritocracy in
> electing committers and PMC members. We believe that contributions can come
> in forms other than just code: for example, one of our initial proposed
> committers has contributed solely in the area of project documentation. We
> will encourage contributions and participation of all types, and ensure
> that contributors are appropriately recognized.
> 
> 2.4.2 Community
> YuniKorn is a relatively new open source project, Cloudera is the original
> development sponsor for YuniKorn. From the beginning of the project itself,
> we had clearly aimed to have this as an open-source project, so we started
> to build the community from the very early stages. We received a lot of
> feedback and valuable suggestions from other community members while the
> project was hosted as an open-source project on GitHub. This feedback has
> greatly influenced some of our designs. For e.g, developers from Alibaba
> had been involved in the very early stage of development, lots of effort
> related to performance/throughput enhancement were contributed by them.
> Lots of other organizations further showed their interest to join the
> community once we started talking about it in meetups, conferences, etc.
> 
> 2.4.3 Core developers
> The project was initiated in Cloudera and so the core developers are
> heavily from this organization. Tao Yang from Alibaba joined the
> development at a very early stage. The core developers of YuniKorn are
> (listed in alphabetical order):
> 
> - Akhil PB (Cloudera)
> - Sunil Govindan (Cloudera)
> - Tao Yang (Alibaba)
> - Vinod Vavilapalli (Cloudera)
> - Wangda Tan (Cloudera)
> - Weiwei Yang (Cloudera)
> - Wilfred Spiegelenburg (Cloudera)
> 
> Given the origin history, the core development team so far has not been
> very diverse, but we’ve been attempting to grow that diversity. We have
> every hope to continue building a diverse and sustainable community if the
> project gets accepted into Apache.
> 
> 2.4.4 Alignment
> The motivation of YuniKorn project is to resolve common resource scheduling
> problems for various workloads, on large scale distributed systems. Apache
> is home to one of these systems in the form of Apache Hadoop YARN. Many of
> thee workloads that we expect to leverage YuniKorn are computing engines
> like Apache Spark, Apache Flink whether they run on top of YARN or on
> Kubernetes.
> 
> 2.5 Known Risks
> 
> 2.5.1 Orphaned products
> The core developers of YuniKorn project from different companies plan to
> work full time on this project. Currently, the initial team intends to
> continue the investments on the YuniKorn project, it will be integrated
> into the solutions to the customers. Several other organizations (like
> Alibaba) have also started to evaluate the project, and plan to adopt it in
> their production environments. We anticipate the adoption will be further
> improved once it becomes an Apache project.
> 
> We have also got support from core-platform developers and Apache
> committers who are interested in contributing to YuniKorn project from
> different companies like Microsoft, Nvidia, Tencent, etc. We’re expecting
> to see more contributions from these committers and usage by their internal
> platforms. So overall, the risk of YuniKorn being an orphaned project is
> low.
> 
> 2.5.2 Inexperience with Open source
> Most of the core developers in YuniKorn project are experienced open source
> veterans, several developers are Apache committers and PMC members of other
> projects, such as Apache™ Hadoop®. And the development style is already
> very likely the Apache way
> - We have open community meetings to discuss designs, problems and roadmaps
> - We publish all patches and issue related discussions on Github
> - We enforce the code review and log all comments in GitHub issues
> 
> 2.5.3 Length of Incubation
> We started the work 10 months ago, so far the groundwork for YuniKorn is
> done and the initial version can work with K8s seamlessly. Based on the
> initial contributors’ experience in ASF projects, we don’t expect that
> there will be huge gaps before YuniKorn can graduate with regarding to
> ASF’s policies on software and releases. The goal is to grow the community
> quickly and increase the user base within a few months while making
> releases that adhere to the ASF standards. When it reaches a reasonable
> size of adoption and a strong community with a good number of
> committers/PMC members, we can prompt the graduation. We expect the length
> of incubation to be approximately 6 to 12 months.
> 
> 2.5.4 Homogenous Development
> The initial proposed list of committers and contributors includes
> developers from several institutions and industry participants. The
> developers are also from different regions like U.S, Australia, India, and
> the development team leverages slack, community mailing list, weekly
> community calls to collaborate efficiently.
> 
> 2.5.5 Reliance on Salaried Developers
> Clearly, Cloudera has contributed most of the initial development through
> salaried developers. But since the very beginning, YuniKorn is built as a
> community effort project. We have people from other organizations that are
> already collaborating with us on Github. This includes both at the source
> code level, as well as participating in designs and providing feedback
> through community calls. We expect our reliance on salaried developers to
> decrease drastically during the incubation process itself.
> 
> 2.5.6 Relationship to Other Apache Products
> 
> YuniKorn is very closely related to other Big-Data projects in Apache, such
> as Hadoop YARN, Spark, Hive, Flink, etc.
> 
> YuniKorn’s core idea is to support both long-running and batch workloads
> like Spark, Hive, Flink, etc, and provide a consistent, unified way to
> manage and schedule resources for Big Data workloads across resource
> managers like Apache™ Hadoop® YARN / Kubernetes and on-premise and cloud
> environments.
> 
> Many of the core ideas for YuniKorn come from the experience of the initial
> team building Apache Hadoop YARN’s schedulers - Capacity Scheduler and Fair
> Scheduler.
> 
> 2.5.7 An Excessive Fascination with the Apache Brand
> Many of the initial developers in YuniKorn project are already experienced
> Apache committers, PMC members. We understand the value of the Apache way,
> and how to operate the project development on a day to day basis. The
> reason for proposing YuniKorn as an Apache project is to build a healthy
> community, increasing adoption & the size of the community and end-users,
> because we believe the only way to build a highly valuable infrastructure
> layer software is to have wide adoption and cater to common use cases.
> 
> 2.6 Documentation
> Project summary:
> https://github.com/cloudera/yunikorn-core/blob/master/README.md
> User guides:
> https://github.com/cloudera/yunikorn-core/blob/master/docs/user-guide.md
> Developer guides:
> https://github.com/cloudera/yunikorn-core/blob/master/docs/developer-guide.md
> Roadmap:
> https://github.com/cloudera/yunikorn-core/blob/master/docs/roadmap.md
> 
> 2.7 Initial Source
> Currently, YuniKorn source code is hosted in several GitHub repositories
> - Scheduler interface:
> https://github.com/cloudera/yunikorn-scheduler-interface
> - Scheduler core: https://github.com/cloudera/yunikorn-core
> - K8s Shim: https://github.com/cloudera/yunikorn-k8shim
> - Scheduler Web UI: https://github.com/cloudera/yunikorn-web
> 
> 2.8 Source and Intellectual Property Submission Plan
> 
> 2.8.1External Dependencies
> External dependencies are listed in below table
> - k8s.io/api, K8s API,  Apache License 2.0
> - k8s.io/apimachinery, K8s API, Apache License 2.0
> - k8s.io/client-go, K8s client library, Apache License 2.0
> - github.com/looplab/fsm, Go state machine library, MIT License
> - github.com/satori/go.uuid, Go UUID library, MIT License
> - github.com/uber-go/zap, Go logging library, MIT License
> - github.com/golang/protobuf, Go protobuf library, BSD 3-Clause License
> - github.com/gorilla/mux, Go network library, BSD 3-Clause License
> - google.golang.org/grpc, Go RPC library, Apache License 2.0
> - gopkg.in/yaml.v2, Go YAML library, Apache License 2.0
> - github.com/prometheus/client_golang, Prometheus Client Library, Apache
> License 2.0
> - Angular v6.1.x, Angular UI Framework Libraries, MIT License
> - TypeScript, TypeScript Language Compiler, Apache License 2.0
> - Chart.js, JavaScript Charting Library, MIT License
> - Moment.js, JavaScript Date & Time Library, MIT License
> 
> Build and test only:
> - gotest.tools, Test library, Apache License 2.0
> - github.com/stretchr/testify, Test library, MIT License
> - Karma, Unit test library, MIT License
> - Protactor, End2End test library, MIT License
> - Json-server, Test server, MIT License
> - Yarn, Dependency manager, BSD 2-Clause License
> 
> 2.8.2 Cryptography
> YuniKorn does not currently include any cryptography-related code.
> 
> 2.9 Required Resources
> 
> 2.9.1 Mailing lists:
> - private@yunikorn.incubator.apache.org (PMC list)
> - commits@yunikorn.incubator.apache.org (git push emails)
> - issues@yunikorn.incubator.apache.org (JIRA issue feed)
> - dev@yunikorn.incubator.apache.org (Dev discussion)
> - user@yunikorn.incubator.apache.org (User questions)
> 
> 2.9.2 Git Repositories
> Git is the preferred source control system
> - git://git.apache.org/yunikorn-* (We have multiple git repositories)
> 
> 2.9.3 Issue Tracking
> JIRA YuniKorn (*YUNIKORN-*)
> 
> 2.9.4 Other Resources
> We have published a series of demo videos on the Youtube channel:
> https://www.youtube.com/channel/UCDSJ2z-lEZcjdK27tTj_hGw
> 
> 2.10 Initial Committers and Affinities
> Initial committers and affinities are listed as below:
> - Akhil PB (apb@cloudera.com) (Cloudera)
> - Sunil Govindan (sunilg@apache.org) (Cloudera)
> - Vinod Kumar Vavilapalli (vinodkv@apache.org) (Cloudera)
> - Wangda Tan (wangda@apache.org) (Cloudera)
> - Weiwei Yang (wwei@apache.org) (Cloudera)
> - Wilfred Spiegelenburg (wspiegelenburg@cloudera.com) (Cloudera)
> - Carlo Curino (curino@apache.org) (Microsoft)
> - Subramaniam Krishnan (subru@apache.org) (Microsoft)
> - Arun Suresh (asuresh@apache.org) (Microsoft)
> - Konstantinos Karanasos (kkaranasos@apache.org) (Microsoft)
> - Jonathan Hung (jhung@apache.org) (LinkedIn)
> - DB Tsai (dbtsai@apache.org) (Apple)
> - Junping Du (junping_du@apache.org) (Tencent)
> - Tao Yang (taoyang@apache.org) (Alibaba)
> - Jason Lowe (jlowe@apache.org) (Nvidia)
> 
> 2.11 Sponsors
> Champion
> - Vinod Kumar Vavilapalli (vinodkv@apache.org)
> 
> Nominated Mentors
> - Junping Du (Tencent), (junping_du@apache.org)
> - Felix Cheung (Uber), (felixcheung@apache.org)
> - Jason Lowe (Nvidia), (jlowe@apache.org)
> - Holden Karau (Apple), (holden@apache.org)
> 
> Sponsoring Entity
> - The Apache Incubator
> 
> [1] https://cwiki.apache.org/confluence/display/INCUBATOR/YuniKornProposal
> 
> -------------------------------- END OF THE PROPOSAL
> -------------------------------
> 
> Thanks
> Weiwei


---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message