incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixche...@apache.org>
Subject Re: [DISCUSS] YuniKorn Proposal
Date Wed, 11 Dec 2019 20:15:34 GMT
+1 ;)


On Wed, Dec 11, 2019 at 12:08 PM Dave Fisher <wave@apache.org> wrote:

> Hi -
>
> Very interesting proposal. +1.
>
> Please note that two of your nominated mentors will need to ask to join
> the IPMC by request at private@i.a.o
>
> Regards,
> Dave
>
> > On Dec 11, 2019, at 11:50 AM, Weiwei Yang <wwei@apache.org> wrote:
> >
> > Greetings folks:
> >
> > Please consider the following proposal, which is also on the wiki
> > <https://cwiki.apache.org/confluence/display/INCUBATOR/YuniKornProposal>
> > [1].
> > I look forward to hearing feedback from you.
> >
> >
> > *YuniKorn - An Unified Resource Scheduler*
> >
> > 1. Abstract
> > YuniKorn is a standalone resource scheduler responsible for scheduling
> > batch jobs and long-running services on large scale distributed systems
> > running in on-premises environments as well as different public clouds.
> >
> > 2. Proposal
> > YuniKorn ['ju:nikɔ:n] is a unified resource scheduler aiming to achieve
> > fine-grained resource sharing for various workloads efficiently on a
> large
> > scale, multi-tenant and cloud-native environments. YuniKorn brings a
> > unified, cross-platform scheduling experience for mixed workloads, with
> > support for but not limited to, Apache™ Hadoop® YARN and Kubernetes.
> >
> > Currently, YuniKorn is an open-source project with Apache 2.0 license.
> The
> > source code is hosted as a git-repo under the github.com/cloudera
> domain.
> > We would like to share it with the ASF and expand the community to a
> wider
> > range of users and contributors.
> >
> > 2.1 Background
> > Enterprise users run their workloads on different platforms such as
> Apache™
> > Hadoop® YARN and Kubernetes. They need to work with different resource
> > schedulers in order to plan their workloads to run on these platforms
> > efficiently. The scheduler implementations are fragmented, and not
> > optimized to balance existing use-cases like batch workloads along with
> new
> > needs such as cloud-native architecture, autoscaling, etc. We need a
> single
> > resource planning/management framework to manage resources on different
> > platforms using the same semantics, in order to address all the important
> > resource management requirements.
> >
> > 2.2 Rationale
> > There is no solution that exists now to address the needs of having a
> > unified resource scheduling experiences across platforms. That makes it
> > difficult to manage workloads running on different environments, from
> > on-premise to Cloud. YuniKorn aims to satisfy these needs. YuniKorn is
> > designed around the following principles:
> >
> > 1) Support different environments
> > As the compute platforms are evolving quickly, there are more and more
> > challenges appear in on-prem, cloud or hybrid environments. YuniKorn aims
> > to bring unified scheduling experiences across multiple environments with
> > enhanced scheduling capabilities.
> >
> > 2) Support extensive types of workloads
> > To improve the efficiency of the computing platform, a key idea is to run
> > different types of applications, like long-running services and batch
> jobs,
> > on shared resources. YuniKorn is an effort to address all the scheduling
> > features needed for such mixed workload environments.
> >
> > 3) Benefit both big-data and cloud-native communities
> > A resource scheduler needs to be capable of supporting mixed workloads,
> > both batch, and long-running services. This is the key to improving
> cluster
> > utilization, and to reduce the complexity of dev-ops. By creating a
> common
> > scheduler that is decoupled from the container platforms underneath, it
> can
> > benefit both Apache™ Hadoop® YARN and the Kubernetes communities.
> >
> > 2.3 Initial Goals
> > Initial goals are:
> > - Move the existing codebase, documentation to Apache hosted repo
> > - Set up mailing lists, website, CI/CD pipeline under Apache
> infrastructure
> > - Setup JIRA for issue tracking
> > - Incremental development and releases according to Apache guidelines
> > - Expand the community and bring more diversified contributors/users to
> > the community
> >
> > 2.4 Current Status
> >
> > 2.4.1 Meritocracy
> > Many of the initial developers of YuniKorn are already Apache committers
> > and PMC members from other Apache projects, such as Apache Hadoop and
> > Apache Submarine. Many of us have worked in the Apache Hadoop community
> for
> > years and know the Apache way well. We believe strongly in meritocracy in
> > electing committers and PMC members. We believe that contributions can
> come
> > in forms other than just code: for example, one of our initial proposed
> > committers has contributed solely in the area of project documentation.
> We
> > will encourage contributions and participation of all types, and ensure
> > that contributors are appropriately recognized.
> >
> > 2.4.2 Community
> > YuniKorn is a relatively new open source project, Cloudera is the
> original
> > development sponsor for YuniKorn. From the beginning of the project
> itself,
> > we had clearly aimed to have this as an open-source project, so we
> started
> > to build the community from the very early stages. We received a lot of
> > feedback and valuable suggestions from other community members while the
> > project was hosted as an open-source project on GitHub. This feedback has
> > greatly influenced some of our designs. For e.g, developers from Alibaba
> > had been involved in the very early stage of development, lots of effort
> > related to performance/throughput enhancement were contributed by them.
> > Lots of other organizations further showed their interest to join the
> > community once we started talking about it in meetups, conferences, etc.
> >
> > 2.4.3 Core developers
> > The project was initiated in Cloudera and so the core developers are
> > heavily from this organization. Tao Yang from Alibaba joined the
> > development at a very early stage. The core developers of YuniKorn are
> > (listed in alphabetical order):
> >
> > - Akhil PB (Cloudera)
> > - Sunil Govindan (Cloudera)
> > - Tao Yang (Alibaba)
> > - Vinod Vavilapalli (Cloudera)
> > - Wangda Tan (Cloudera)
> > - Weiwei Yang (Cloudera)
> > - Wilfred Spiegelenburg (Cloudera)
> >
> > Given the origin history, the core development team so far has not been
> > very diverse, but we’ve been attempting to grow that diversity. We have
> > every hope to continue building a diverse and sustainable community if
> the
> > project gets accepted into Apache.
> >
> > 2.4.4 Alignment
> > The motivation of YuniKorn project is to resolve common resource
> scheduling
> > problems for various workloads, on large scale distributed systems.
> Apache
> > is home to one of these systems in the form of Apache Hadoop YARN. Many
> of
> > thee workloads that we expect to leverage YuniKorn are computing engines
> > like Apache Spark, Apache Flink whether they run on top of YARN or on
> > Kubernetes.
> >
> > 2.5 Known Risks
> >
> > 2.5.1 Orphaned products
> > The core developers of YuniKorn project from different companies plan to
> > work full time on this project. Currently, the initial team intends to
> > continue the investments on the YuniKorn project, it will be integrated
> > into the solutions to the customers. Several other organizations (like
> > Alibaba) have also started to evaluate the project, and plan to adopt it
> in
> > their production environments. We anticipate the adoption will be further
> > improved once it becomes an Apache project.
> >
> > We have also got support from core-platform developers and Apache
> > committers who are interested in contributing to YuniKorn project from
> > different companies like Microsoft, Nvidia, Tencent, etc. We’re expecting
> > to see more contributions from these committers and usage by their
> internal
> > platforms. So overall, the risk of YuniKorn being an orphaned project is
> > low.
> >
> > 2.5.2 Inexperience with Open source
> > Most of the core developers in YuniKorn project are experienced open
> source
> > veterans, several developers are Apache committers and PMC members of
> other
> > projects, such as Apache™ Hadoop®. And the development style is already
> > very likely the Apache way
> > - We have open community meetings to discuss designs, problems and
> roadmaps
> > - We publish all patches and issue related discussions on Github
> > - We enforce the code review and log all comments in GitHub issues
> >
> > 2.5.3 Length of Incubation
> > We started the work 10 months ago, so far the groundwork for YuniKorn is
> > done and the initial version can work with K8s seamlessly. Based on the
> > initial contributors’ experience in ASF projects, we don’t expect that
> > there will be huge gaps before YuniKorn can graduate with regarding to
> > ASF’s policies on software and releases. The goal is to grow the
> community
> > quickly and increase the user base within a few months while making
> > releases that adhere to the ASF standards. When it reaches a reasonable
> > size of adoption and a strong community with a good number of
> > committers/PMC members, we can prompt the graduation. We expect the
> length
> > of incubation to be approximately 6 to 12 months.
> >
> > 2.5.4 Homogenous Development
> > The initial proposed list of committers and contributors includes
> > developers from several institutions and industry participants. The
> > developers are also from different regions like U.S, Australia, India,
> and
> > the development team leverages slack, community mailing list, weekly
> > community calls to collaborate efficiently.
> >
> > 2.5.5 Reliance on Salaried Developers
> > Clearly, Cloudera has contributed most of the initial development through
> > salaried developers. But since the very beginning, YuniKorn is built as a
> > community effort project. We have people from other organizations that
> are
> > already collaborating with us on Github. This includes both at the source
> > code level, as well as participating in designs and providing feedback
> > through community calls. We expect our reliance on salaried developers to
> > decrease drastically during the incubation process itself.
> >
> > 2.5.6 Relationship to Other Apache Products
> >
> > YuniKorn is very closely related to other Big-Data projects in Apache,
> such
> > as Hadoop YARN, Spark, Hive, Flink, etc.
> >
> > YuniKorn’s core idea is to support both long-running and batch workloads
> > like Spark, Hive, Flink, etc, and provide a consistent, unified way to
> > manage and schedule resources for Big Data workloads across resource
> > managers like Apache™ Hadoop® YARN / Kubernetes and on-premise and cloud
> > environments.
> >
> > Many of the core ideas for YuniKorn come from the experience of the
> initial
> > team building Apache Hadoop YARN’s schedulers - Capacity Scheduler and
> Fair
> > Scheduler.
> >
> > 2.5.7 An Excessive Fascination with the Apache Brand
> > Many of the initial developers in YuniKorn project are already
> experienced
> > Apache committers, PMC members. We understand the value of the Apache
> way,
> > and how to operate the project development on a day to day basis. The
> > reason for proposing YuniKorn as an Apache project is to build a healthy
> > community, increasing adoption & the size of the community and end-users,
> > because we believe the only way to build a highly valuable infrastructure
> > layer software is to have wide adoption and cater to common use cases.
> >
> > 2.6 Documentation
> > Project summary:
> > https://github.com/cloudera/yunikorn-core/blob/master/README.md
> > User guides:
> > https://github.com/cloudera/yunikorn-core/blob/master/docs/user-guide.md
> > Developer guides:
> >
> https://github.com/cloudera/yunikorn-core/blob/master/docs/developer-guide.md
> > Roadmap:
> > https://github.com/cloudera/yunikorn-core/blob/master/docs/roadmap.md
> >
> > 2.7 Initial Source
> > Currently, YuniKorn source code is hosted in several GitHub repositories
> > - Scheduler interface:
> > https://github.com/cloudera/yunikorn-scheduler-interface
> > - Scheduler core: https://github.com/cloudera/yunikorn-core
> > - K8s Shim: https://github.com/cloudera/yunikorn-k8shim
> > - Scheduler Web UI: https://github.com/cloudera/yunikorn-web
> >
> > 2.8 Source and Intellectual Property Submission Plan
> >
> > 2.8.1External Dependencies
> > External dependencies are listed in below table
> > - k8s.io/api, K8s API,  Apache License 2.0
> > - k8s.io/apimachinery, K8s API, Apache License 2.0
> > - k8s.io/client-go, K8s client library, Apache License 2.0
> > - github.com/looplab/fsm, Go state machine library, MIT License
> > - github.com/satori/go.uuid, Go UUID library, MIT License
> > - github.com/uber-go/zap, Go logging library, MIT License
> > - github.com/golang/protobuf, Go protobuf library, BSD 3-Clause License
> > - github.com/gorilla/mux, Go network library, BSD 3-Clause License
> > - google.golang.org/grpc, Go RPC library, Apache License 2.0
> > - gopkg.in/yaml.v2, Go YAML library, Apache License 2.0
> > - github.com/prometheus/client_golang, Prometheus Client Library, Apache
> > License 2.0
> > - Angular v6.1.x, Angular UI Framework Libraries, MIT License
> > - TypeScript, TypeScript Language Compiler, Apache License 2.0
> > - Chart.js, JavaScript Charting Library, MIT License
> > - Moment.js, JavaScript Date & Time Library, MIT License
> >
> > Build and test only:
> > - gotest.tools, Test library, Apache License 2.0
> > - github.com/stretchr/testify, Test library, MIT License
> > - Karma, Unit test library, MIT License
> > - Protactor, End2End test library, MIT License
> > - Json-server, Test server, MIT License
> > - Yarn, Dependency manager, BSD 2-Clause License
> >
> > 2.8.2 Cryptography
> > YuniKorn does not currently include any cryptography-related code.
> >
> > 2.9 Required Resources
> >
> > 2.9.1 Mailing lists:
> > - private@yunikorn.incubator.apache.org (PMC list)
> > - commits@yunikorn.incubator.apache.org (git push emails)
> > - issues@yunikorn.incubator.apache.org (JIRA issue feed)
> > - dev@yunikorn.incubator.apache.org (Dev discussion)
> > - user@yunikorn.incubator.apache.org (User questions)
> >
> > 2.9.2 Git Repositories
> > Git is the preferred source control system
> > - git://git.apache.org/yunikorn-* (We have multiple git repositories)
> >
> > 2.9.3 Issue Tracking
> > JIRA YuniKorn (*YUNIKORN-*)
> >
> > 2.9.4 Other Resources
> > We have published a series of demo videos on the Youtube channel:
> > https://www.youtube.com/channel/UCDSJ2z-lEZcjdK27tTj_hGw
> >
> > 2.10 Initial Committers and Affinities
> > Initial committers and affinities are listed as below:
> > - Akhil PB (apb@cloudera.com) (Cloudera)
> > - Sunil Govindan (sunilg@apache.org) (Cloudera)
> > - Vinod Kumar Vavilapalli (vinodkv@apache.org) (Cloudera)
> > - Wangda Tan (wangda@apache.org) (Cloudera)
> > - Weiwei Yang (wwei@apache.org) (Cloudera)
> > - Wilfred Spiegelenburg (wspiegelenburg@cloudera.com) (Cloudera)
> > - Carlo Curino (curino@apache.org) (Microsoft)
> > - Subramaniam Krishnan (subru@apache.org) (Microsoft)
> > - Arun Suresh (asuresh@apache.org) (Microsoft)
> > - Konstantinos Karanasos (kkaranasos@apache.org) (Microsoft)
> > - Jonathan Hung (jhung@apache.org) (LinkedIn)
> > - DB Tsai (dbtsai@apache.org) (Apple)
> > - Junping Du (junping_du@apache.org) (Tencent)
> > - Tao Yang (taoyang@apache.org) (Alibaba)
> > - Jason Lowe (jlowe@apache.org) (Nvidia)
> >
> > 2.11 Sponsors
> > Champion
> > - Vinod Kumar Vavilapalli (vinodkv@apache.org)
> >
> > Nominated Mentors
> > - Junping Du (Tencent), (junping_du@apache.org)
> > - Felix Cheung (Uber), (felixcheung@apache.org)
> > - Jason Lowe (Nvidia), (jlowe@apache.org)
> > - Holden Karau (Apple), (holden@apache.org)
> >
> > Sponsoring Entity
> > - The Apache Incubator
> >
> > [1]
> https://cwiki.apache.org/confluence/display/INCUBATOR/YuniKornProposal
> >
> > -------------------------------- END OF THE PROPOSAL
> > -------------------------------
> >
> > Thanks
> > Weiwei
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message