incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixche...@apache.org>
Subject Re: [VOTE] Accept Amaterasu into the Apache Incubator
Date Thu, 20 Apr 2017 17:14:51 GMT
+1 (non binding)

On Wed, Apr 19, 2017 at 4:59 PM Olivier Lamy <olamy@apache.org> wrote:

> +1 (binding)
>
> On 17 April 2017 at 04:20, Jean-Baptiste Onofré <jb@nanthrax.net> wrote:
>
> > Hi all,
> >
> > following the discussion thread, I think we can start the vote on
> > accepting Amaterasu into the Apache Incubator.
> >
> > The ASF voting rules are described:
> >
> >         http://www.apache.org/foundation/voting.html
> >
> > A vote for accepting a new Apache Incubator podling is a majority vote
> for
> > which only Incubator PMC member votes are binding.
> >
> > This vote will run for at least 72 hours. Please VOTE as follows
> > [] +1 Accept Amaterasu into the Apache Incubator
> > [] +0 Abstain.
> > [] -1 Do not accept Amaterasu into the Apache Incubator because ...
> >
> > The proposal is listed below, but you can also access it on the wiki:
> >
> >    https://wiki.apache.org/incubator/AmaterasuProposal
> >
> > Note that we are looking for one more mentor on the proposal.
> > One concern has been raised about the "Amaterasu" name. We want to move
> > forward with Amaterasu name but open to change if needed.
> >
> > Thanks
> > Regards
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> > = Apache Amaterasu =
> >
> > == Abstract ==
> >
> > Apache Amaterasu is a framework providing continuous deployment for Big
> > Data pipelines.
> >
> > It provides the following capabilities:
> >
> >  * '''Continuous integration''' tools to '''package pipelines and run
> > tests'''.
> >  * A repository to store those packaged applications: the '''applications
> > repository'''.
> >  * A repository to store the pipelines, and engine configuration (for
> > instance, location of the Spark master, etc.): per environment - the
> > '''configuration repository'''.
> >  * A '''dashboard''' to monitor the pipelines.
> >  * A '''DSL and integration hooks''' allowing third parties to easily
> > integrate.
> >
> > == Proposal ==
> >
> > Amaterasu is a simple and powerful framework to build and dispense
> > pipelines. It aims to help data engineers and data scientists to compose,
> > configure, test, package, deploy and execute data pipelines written using
> > multiple tools, languages and frameworks. Amaterasu provides a standard
> > repo structure to package big data pipelines, a YAML based Domain
> Specific
> > Languages (DSL) for data engineers, data scientists and operations
> > engineers to manage complex pipelines throughout their entire lifecycle
> > (Dev, UAT, Prod, etc.).
> >
> > == Background ==
> >
> > Amaterasu is a relatively new project that was created to deal with some
> > of the issues that as Consultants, we have seen recurring at different
> > client sites. Mainly the need to continuously deploy complex pipelines
> > built in multiple tools and languages.
> > Amaterasu started as a pet project and is currently being evaluated by a
> > couple of organizations, supported by the contributors, on a personal
> time
> > and voluntary bases.
> >
> > == Rational ==
> >
> > As software engineers working on big data projects we have straggled for
> a
> > long time to apply the same CI/CD practices that have become the standard
> > in the software industry for the last few years. While some of them are
> > possible, for example Apache Spark is easy to unit test. However large
> > scale pipelines are more complex and often use data, which might be
> > un-structured as integration point, which requires heavy integration
> tests.
> >
> > To automate such tests and complex deployments, we have found the need to
> > often handcraft scripts and use a mixture tools, so we have decided to
> > finally build a tool we can apply in a general way and not on a project
> by
> > project basis.
> >
> > Another issue Amaterasu is trying to tackle is the Integrating between
> the
> > work of software engineers, data scientists, and sometimes operations
> > engineers. The approach Amaterasu takes to integrate between those three
> > schools of thought it to provide a simple YAML based DSL that provides a
> > simple way to integrate different pipeline written in the native tools
> for
> > each task (R, Spark in different languages, etc.).
> >
> > == Initial Goals ==
> >
> > Our initial goals are to bring Amaterasu into the ASF, transition
> internal
> > engineering processes into the open, and foster a collaborative
> development
> > model according to the "Apache Way".
> >
> > In addition, we intend to continue the development of Amaterasu, add new
> > features as well as  integrate better with other frameworks, including:
> >
> >  * Apache Arrow
> >  * Apache Hive
> >  * Apache Drill
> >  * Apache Beam
> >  * Apache YARN
> >  * Farther and more complete integration with Apache Spark
> >
> > Other frameworks will be evaluated after those initial goals are reached.
> >
> > == Current Status ==
> >
> > Amaterasu is preview state but provide a large set of features. We plan
> to
> > stabilize and head to a first production ready release during the
> > incubation process. The current license is already Apache 2.0.
> >
> > === Meritocracy ===
> >
> > We intend to radically expand the initial developer and user community by
> > running the project in accordance with the "Apache Way". Users and new
> > contributors will be treated with respect and welcomed. By participating
> in
> > the community and providing quality patches/support that move the project
> > forward, they will earn merit. They also will be encouraged to provide
> > non-code contributions (documentation, events, community management,
> etc.)
> > and will gain merit for doing so. Those with a proven support and quality
> > track record will be encouraged to become committers.
> >
> > === Community ===
> >
> > As a relatively new project, Amaterasu has a small, but growing
> community.
> > Amaterasu is an open project, not just with it’s source code but also
> with
> > our discussions which are held openly in our slack
> > https://shintoio.slack.com which contains channels for design, tech and
> > future directions discussions.
> >
> > If Amaterasu is accepted for incubation, the primary initial goal is to
> > build a large and strong community. We are confident that Amaterasu can
> > become a key project for big data operations, which hopefully will
> create a
> > large community of users and developers.
> >
> > === Known Risks ===
> >
> > Development has been sponsored mostly by a one company. For the project
> to
> > fully transition to the Apache Way governance model, development must
> shift
> > towards the meritocracy-centric model of growing a community of
> > contributors balanced with the needs for extreme stability and core
> > implementation coherency.
> >
> > === Orphaned products ===
> >
> > We are fully committed on Amaterasu. A few organizations have expressed
> > their interest in using Amaterasu.
> >
> > === Inexperience with Open Source ===
> >
> > We have been developing and using open source software for a long time.
> > Additionally, several ASF veterans have agreed to mentor the project and
> > they are listed in this proposal. The project will rely on their guidance
> > and collective wisdom to quickly transition the entire team of initial
> > committers towards practicing the Apache Way.
> >
> > === Reliance on Salaried Developers ===
> >
> > Most of the current contributors are employed in the Big Data space.
> While
> > they might wander from their current employers, they are unlikely to
> > venture far from their core expertises and thus will continue to be
> engaged
> > with the project regardless of their current employers.
> >
> > === An Excessive Fascination with the Apache Brand ===
> >
> > While we intend to leverage the Apache ‘branding’ when talking to other
> > projects as testament of our project’s ‘neutrality’, we have no plans for
> > making use of Apache brand in press releases nor posting billboards
> > advertising acceptance of Amaterasu into Apache Incubator.
> >
> > The main purpose in applying for Apache incubation is due to the fact
> that
> > Amaterasu is built with integration already in mind for many tools which
> > are Apache projects, and we see Amaterasu as an extension of these
> > projects. We hope that by being an Apache project, we can integrate
> better,
> > and collaborate more effectively with the relevant projects. As Amaterasu
> > matures, we see mutual benefits for all involved.
> >
> > === Initial Source ===
> >
> > https://github.com/shintoio/amaterasu
> >
> > === External Dependencies ===
> >
> > All external dependencies are licensed under an Apache 2.0 license or
> > Apache-compatible license. As we grow the Amaterasu community we will
> > configure our build process to require and validate all contributions and
> > dependencies are licensed under the Apache 2.0 license or are under an
> > Apache-compatible license.
> >
> >  * Apache Spark
> >  * Apache Hadoop
> >  * Apache Maven (maven-core)
> >  * Apache Commons
> >  * Apache Log4j
> >  * Apache Mesos
> >  * Apache Zookeeper
> >  * Apache Curator
> >  * Scala
> >  * Junit
> >  * Py4j
> >
> > Future versions are planned to integrate with:
> >
> >  * Apache YARN
> >  * Apache Hive
> >  * Apache Drill
> >
> > === Required Resources ===
> >
> > ==== Mailing lists ====
> >
> >  * private@amaterasu.incubator.apache.org (moderated subscriptions)
> >  * commits@amaterasu.incubator.apache.org
> >  * dev@amaterasu.incubator.apache.org
> >  * issues@amaterasu.incubator.apache.org
> >
> > ==== Git Repository ====
> >
> >  * https://git-wip-us.apache.org/repos/asf/incubator-amaterasu.git
> >
> > ==== Issue Tracking ====
> >
> >  * JIRA Project Amaterasu
> >
> > ==== Initial Committers ====
> >
> >  * Yaniv Rodenski
> >  * Jean-Baptiste Onofré
> >  * Eyal Ben Ivri
> >  * Karel Alfonso
> >  * Kirupagaran (Kirupa) Devarajan
> >  * Nadav Har Tzvi
> >
> > ==== Affiliations ====
> >
> >  * Yaniv Rodenski - Shinto
> >  * Jean-Baptiste Onofré - Talend
> >  * Olivier Lamy - Webtide
> >
> > ==== Sponsors ====
> >
> > ==== Champion ====
> >
> >  * Jean-Baptiste Onofré
> >
> > ==== Mentors ====
> >
> >  * Jean-Baptiste Onofré
> >  * Olivier Lamy
> >
> > ==== Sponsoring Entity ====
> >
> > The Apache Incubator
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message