From general-return-52983-apmail-incubator-general-archive=incubator.apache.org@incubator.apache.org Wed Nov 25 17:20:23 2015 Return-Path: X-Original-To: apmail-incubator-general-archive@www.apache.org Delivered-To: apmail-incubator-general-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 322FB18CC2 for ; Wed, 25 Nov 2015 17:20:23 +0000 (UTC) Received: (qmail 67768 invoked by uid 500); 25 Nov 2015 17:20:22 -0000 Delivered-To: apmail-incubator-general-archive@incubator.apache.org Received: (qmail 67587 invoked by uid 500); 25 Nov 2015 17:20:22 -0000 Mailing-List: contact general-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@incubator.apache.org Delivered-To: mailing list general@incubator.apache.org Received: (qmail 67576 invoked by uid 99); 25 Nov 2015 17:20:22 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Nov 2015 17:20:22 +0000 Received: from mail-qg0-f42.google.com (mail-qg0-f42.google.com [209.85.192.42]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id F19251A0513 for ; Wed, 25 Nov 2015 17:20:21 +0000 (UTC) Received: by qgec40 with SMTP id c40so36707295qge.2 for ; Wed, 25 Nov 2015 09:20:20 -0800 (PST) MIME-Version: 1.0 X-Received: by 10.140.43.9 with SMTP id d9mr19310499qga.51.1448472020785; Wed, 25 Nov 2015 09:20:20 -0800 (PST) Received: by 10.55.128.197 with HTTP; Wed, 25 Nov 2015 09:20:20 -0800 (PST) In-Reply-To: References: Date: Wed, 25 Nov 2015 09:20:20 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [VOTE] Accept Impala into the Apache Incubator From: Doug Cutting To: "general@incubator.apache.org" Content-Type: multipart/alternative; boundary=001a1139bea44422f5052560ac70 --001a1139bea44422f5052560ac70 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable +1 (binding) Doug On Tue, Nov 24, 2015 at 1:03 PM, Henry Robinson wrote: > Hi - > > The [DISCUSS] thread has been quiet for a few days, so I think there's be= en > sufficient opportunity for discussion around our proposal to bring Impala > to the ASF Incubator. > > I'd like to call a VOTE on that proposal, which is on the wiki at > https://wiki.apache.org/incubator/ImpalaProposal, and which I've pasted > below. > > During the discussion period, the proposal has been amended to add Brock > Noland as a new mentor, to add one missed committer from the list and to > correct some issues with the dependency list. > > Please cast your votes as follows: > > [] +1, accept Impala into the Incubator > [] +/-0, non-counted vote to express a disposition > [] -1, do not accept Impala into the Incubator (please give your reason(s= )) > > As with the concurrent Kudu vote, I propose leaving the vote open for a > full seven days (to close at Tuesday, December 1st at noon PST), due to t= he > upcoming US holiday. > > Thanks, > Henry > > -------- > > =3D Abstract =3D > Impala is a high-performance C++ and Java SQL query engine for data store= d > in Apache Hadoop-based clusters. > > =3D Proposal =3D > > We propose to contribute the Impala codebase and associated artifacts (e.= g. > documentation, web-site content etc.) to the Apache Software Foundation > with the intent of forming a productive, meritocratic and open community > around Impala=E2=80=99s continued development, according to the =E2=80=98= Apache Way=E2=80=99. > > Cloudera owns several trademarks regarding Impala, and proposes to transf= er > ownership of those trademarks in full to the ASF. > > =3D Background =3D > Engineers at Cloudera developed Impala and released it as an > Apache-licensed open-source project in Fall 2012. Impala was written as a > brand-new, modern C++ SQL engine targeted from the start for data stored = in > Apache Hadoop clusters. > > Impala=E2=80=99s most important benefit to users is high-performance, mak= ing it > extremely appropriate for common enterprise analytic and business > intelligence workloads. This is achieved by a number of software > techniques, including: native support for data stored in HDFS and related > filesystems, just-in-time compilation and optimization of individual quer= y > plans, high-performance C++ codebase and massively-parallel distributed > architecture. In benchmarks, Impala is routinely amongst the very highest > performing SQL query engines. > > =3D Rationale =3D > > Despite the exciting innovation in the so-called =E2=80=98big-data=E2=80= =99 space, SQL > remains by far the most common interface for interacting with data in bot= h > traditional warehouses and modern =E2=80=98big-data=E2=80=99 clusters. Th= ere is clearly a > need, as evidenced by the eager adoption of Impala and other SQL engines = in > enterprise contexts, for a query engine that offers the familiar SQL > interface, but that has been specifically designed to operate in massive, > distributed clusters rather than in traditional, fixed-hardware, > warehouse-specific deployments. Impala is one such query engine. > > We believe that the ASF is the right venue to foster an open-source > community around Impala=E2=80=99s development. We expect that Impala will= benefit > from more productive collaboration with related Apache projects, and unde= r > the auspices of the ASF will attract talented contributors who will push > Impala=E2=80=99s development forward at pace. > > We believe that the timing is right for Impala=E2=80=99s development to m= ove > wholesale to the ASF: Impala is well-established, has been Apache-license= d > open-source for more than three years, and the core project is relatively > stable. We are excited to see where an ASF-based community can take Impal= a > from this strong starting point. > > =3D Initial Goals =3D > Our initial goals are as follows: > > * Establish ASF-compatible engineering practices and workflows > * Refactor and publish existing internal build scripts and test > infrastructure, in order to make them usable by any community member. > * Transfer source code, documentation and associated artifacts to the AS= F. > * Grow the user and developer communities > > =3D Current Status =3D > > Impala is developed as an Apache-licensed open-source project. The source > code is available at http://github.com/cloudera/Impala, and developer > documentation is at https://github.com/cloudera/Impala/wiki. The majority > of commits to the project have come from Cloudera-employed developers, bu= t > we have accepted some contributions from individuals from other > organizations. > > All code reviews are done via a public instance of the Gerrit review tool > at http://gerrit.cloudera.org:8080/, and discussed on a public mailing > list. All patches must be reviewed before they are accepted into the > codebase, via a voting mechanism that is similar to that used on Apache > projects such as Hadoop and HBase. > > Before a patch is committed, it must pass a suite of pre-commit tests. > These tests are currently run on Cloudera=E2=80=99s internal infrastructu= re. One of > our initial goals will be to work with the ASF Infrastructure team to fin= d > a way to run these tests in an acceptable way on publicly accessible > machines. > > Issues are tracked in JIRA at https://issues.cloudera.org/projects/IMPALA= , > in a way that is extremely similar to existing practices at other ASF > projects. > > =3D Meritocracy =3D > > We understand the central importance of meritocracy to the Apache Way. We > will work to establish a welcoming, fair and meritocratic community, in > part by expanding the set of committers on the project. Although Impala= =E2=80=99s > committer list will initially be dominated by members of the Impala > engineering team at Cloudera, we look forward to growing a rich user and > developer community. > > =3D Community =3D > Impala has a strong user community (see > https://groups.google.com/a/cloudera.org/forum/#!forum/impala-user), and = a > growing developer community (see > https://groups.google.com/a/cloudera.org/forum/#!forum/impala-dev). We > wish > to attract more developers to the project, and we believe that the ASF=E2= =80=99s > open and meritocratic philosophy will help us with this. We note the > success of other, similar projects already part of the ASF. > > =3D Core Developers =3D > Most - but not all - of Impala=E2=80=99s core developers are not currentl= y > affiliated with the ASF, and will require new ICLAs. > > =3D Alignment =3D > Impala is related to several other Apache projects: > > * Data that is read by Impala is very often stored in Apache Hadoop > clusters powered by the HDFS filesystem. > * Impala can also read data stored in Apache HBase > * Metadata for databases, tables and so on is read by Impala from Apache > Hive. > * The preferred data format for HDFS-based tables is Apache Parquet, and > Apache Avro is also a supported data format. > * Impala is closely integrated with Kudu, which is also being proposed t= o > the Incubator. > * Impala uses Apache Thrift as its RPC and serialization framework of > choice. > > =3D Known Risks =3D > > =3D=3D Orphaned Products =3D=3D > Impala is used by most of Cloudera=E2=80=99s customers, and Cloudera rema= ins > committed to developing and supporting the project. Cloudera has a strong > track record in standing behind projects that were contributed to the ASF > by its employees, including Apache Flume, Apache Sqoop, and others. Other > companies both ship and support Impala, lending credence to the idea that > Impala is not at risk of being suddenly orphaned. > > =3D=3D Inexperience with Open Source =3D=3D > Although all committers on the initial list have significant experience > with at least one open-source project - namely Impala - fewer have much > experience with ASF-based software projects as contributors and community > members. However, with the guidance of our mentors, committers who do hav= e > ASF experience, and time to learn during Incubation, we are confident tha= t > the project can be run in accordance with Apache principles on an ongoing > basis. > > =3D=3D Homogeneous Developers =3D=3D > > The initial committers are employees of Cloudera. > > The project has received some contributions from developers outside of > Cloudera, from individuals belonging to organizations such as Intel and > Google, from hobbyists and from students using Impala to advance their > understanding of distributed databases. The project attracted an active > user community as well. We hope to continue to encourage contributions fr= om > these developers and community members and grow them into committers afte= r > they have had time to continue their contributions. > > =3D=3D Reliance on Salaried Developers =3D=3D > > Many of Impala=E2=80=99s initial set of committers work full-time on Impa= la, and > are paid to do so. However, as mentioned elsewhere, we anticipate growth = in > the developer community which we hope will include hobbyists and academic= s > who have an interested in distributed data systems. > > =3D=3D An Excessive Fascination with the Apache Brand =3D=3D > Although we hope that Impala benefits from the Apache Brand, any reflecte= d > goodwill to Cloudera as the contributing entity is not the goal of > establishing Impala as an Apache project. We will work with the Incubator > PMC and the PRC to ensure that the Apache Brand is respected. > > =3D Documentation =3D > Impala: A Modern, Open-Source SQL Engine for Hadoop ( > http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper28.pdf) > > Impala=E2=80=99s developer wiki (https://github.com/cloudera/Impala/wiki) > > Impala=E2=80=99s auto-generated API documentation ( > http://impala.io/doc/html/index.html) > > =3D Initial Source =3D > Impala=E2=80=99s initial source contribution will come from > http://github.com/cloudera/Impala/. > > =3D External Dependencies =3D > > Impala depends upon a number of third-party libraries, which we list belo= w. > We intend to compile a LICENSE.txt file in the very short term (see > https://issues.cloudera.org/browse/IMPALA-2670). > > * Google gflags (BSD) > * Google glog (BSD) > * Apache Thrift (Apache Software License v2.0) > * Apache Commons (Apache Software License v2.0) > * Apache Hadoop (Apache Software License v2.0) > * Apache HBase (Apache Software License v2.0) > * Apache Hive (Apache Software License v2.0) > * Boost (Boost Software License) > * OpenLdap (OpenLDAP Software License) > * rapidjson (MIT) > * Google RE2 (BSD-style) > * lz4 (BSD) > * snappy (BSD) > * cyrus-sasl (CMU License) > * Apache Avro (Apache Software License v2.0) > * Cloudera squeasel (Apache Software License v2.0) > * Apache htrace (Incubating) (Apache Software License v2.0) > * Apache Sentry (Incubating) (Apache Software License v2.0) > * Apache Shiro (Apache Software License v2.0) > * Twitter Bootstrap (Apache Software License v2.0) > * d3 (BSD) > * LLVM (BSD-like) > > Build and test dependencies: > > * ant (Apache Software License v2.0) > * Apache Maven (Apache Software License v2.0) > * cmake (BSD) > * clang (BSD) > * Google gtest (Apache Software License v2.0) > > =3D Required Resources =3D > > We request that following resources be created for the project to use: > > =3D=3D Mailing lists =3D=3D > > * private@impala.incubator.apache.org (moderated subscriptions) > * commits@impala.incubator.apache.org > * dev@impala.incubator.apache.org > * issues@impala.incubator.apache.org > * user@impala.incubator.apache.org > > =3D=3D Git repository =3D=3D > https://git.apache.org/impala.git > > =3D=3D JIRA instance =3D=3D > JIRA project IMPALA (IMPALA or IMP) > > =3D=3D Other Resources =3D=3D > We hope to continue using Gerrit for our code review and commit workflow. > We are involved with discussions that the Kudu team at Cloudera have been > having with Jake Farrell to start discussions on how Gerrit can fit into > the ASF. We know that several other ASF projects or podlings are also > interested in Gerrit. > > If the Infrastructure team does not have the bandwidth to support gerrit, > we will continue to support our own instance of gerrit for Impala, and ma= ke > the necessary integrations such that commits are properly authenticated a= nd > maintain sufficient provenance to uphold the ASF standards (e.g. via the > solution adopted by the AsterixDB podling). > > =3D Initial Committers =3D > > * Tim Armstrong > * Alex Behm > * Taras Bobrovytsky > * Casey Ching > * Martin Grund > * Daniel Hecht > * Michael Ho > * Matthew Jacobs > * Ishaan Joshi > * Lenni Kuff > * Marcel Kornacker > * Sailesh Mukil > * Henry Robinson > * John Russell > * Dimitris Tsirogiannis > * Skye Wanderman-Milne > * Juan Yu > > =3D=3D Affiliations =3D=3D > All: Cloudera Inc. > > =3D Sponsors =3D > > =3D=3D Champion =3D=3D > Tom White > > =3D=3D Nominated Mentors =3D=3D > * Tom White (Cloudera) > * Todd Lipcon (Cloudera) > * Carl Steinbach (LinkedIn) > * Brock Noland (StreamSets) > > > =3D Sponsoring Entity =3D > We ask that the Incubator PMC sponsor this proposal. > --001a1139bea44422f5052560ac70--