incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: [PROPOSAL] Apache AsterixDB Incubator
Date Tue, 20 Jan 2015 19:06:55 GMT
Added my name to the mentor list.



On Tue, Jan 20, 2015 at 8:37 AM, Mike Carey <dtabass@gmail.com> wrote:

>  Wonderful; thanks, Ted!!
> Cheers,
> Mike
>
>  On 1/19/15 11:29 PM, Ted Dunning wrote:
>
>
> Chris just asked me under separate cover.
>
>  I am happy to help out as mentor.
>
>
>
> On Mon, Jan 19, 2015 at 8:17 PM, Henry Saputra <henry.saputra@gmail.com>
> wrote:
>
>> Thanks Till,
>>
>> Will try to solicit more mentors to help.
>> Especially with initial committers mostly have not been exposed to
>> contributing the Apache way.
>>
>> - Henry
>>
>> On Mon, Jan 19, 2015 at 5:28 PM, Till Westmann <till@westmann.org> wrote:
>> > Hi Henry,
>> >
>> > thanks! It’s great that you’ve seen (and liked) AsterixDB before.
>> >
>> > Even if your time is very limited we would be very happy to have you on
>> board as a mentor.
>> > I’ll add you to the proposal.
>> >
>> > Cheers,
>> > Till
>> >
>> >> On Jan 19, 2015, at 10:26 AM, Henry Saputra <henry.saputra@gmail.com>
>> wrote:
>> >>
>> >> +1 This is GREAT News!
>> >>
>> >> Was watching and trying AsterixDB last year and looked in awesome
>> shape.
>> >>
>> >> I have my plate full but would love to help mentor this project to get
>> >> it going to ASF if needed!
>> >>
>> >> - Henry
>> >>
>> >> On Wed, Jan 14, 2015 at 6:21 PM, Mattmann, Chris A (3980)
>> >> <chris.a.mattmann@jpl.nasa.gov> wrote:
>> >>> Hi Folks,
>> >>>
>> >>> I am pleased to bring forth the Apache AsterixDB proposal to the
>> >>> Apache Incubator as Champion, working in collaboration with the
>> >>> team. Please find the wiki proposal here:
>> >>>
>> >>> https://wiki.apache.org/incubator/AsterixDBProposal
>> >>>
>> >>>
>> >>> Full text of the proposal is below. Please discuss and enjoy. I’ll
>> >>> leave the discussion open for a week, and then look to call a VOTE
>> >>> hopefully end of next week if all is well.
>> >>>
>> >>> Cheers!
>> >>> Chris Mattmann
>> >>>
>> >>> =============================================================
>> >>> Apache AsterixDB Proposal
>> >>>
>> >>> Abstract
>> >>>
>> >>> Apache AsterixDB is a scalable big data management system (BDMS) that
>> >>> provides storage, management, and query capabilities for large
>> >>> collections of semi-structured data.
>> >>>
>> >>> Proposal
>> >>>
>> >>> AsterixDB is a big data management system (BDMS) that makes it
>> >>> well-suited to needs such as web data warehousing and social data
>> >>> storage and analysis. Feature-wise, AsterixDB has:
>> >>>
>> >>> * A NoSQL style data model (ADM) based on extending JSON with object
>> >>>  database concepts.
>> >>> * An expressive and declarative query language (AQL) for querying
>> >>>  semi-structured data.
>> >>> * A runtime query execution engine, Hyracks, for partitioned-parallel
>> >>>  execution of query plans.
>> >>> * Partitioned LSM-based data storage and indexing for efficient
>> >>>  ingestion of newly arriving data.
>> >>> * Support for querying and indexing external data (e.g., in HDFS) as
>> >>>  well as data stored within AsterixDB.
>> >>> * A rich set of primitive data types, including support for spatial,
>> >>>  temporal, and textual data.
>> >>> * Indexing options that include B+ trees, R trees, and inverted
>> >>>  keyword index support.
>> >>> * Basic transactional (concurrency and recovery) capabilities akin to
>> >>>  those of a NoSQL store.
>> >>>
>> >>>
>> >>> Background and Rationale
>> >>>
>> >>> In the world of relational databases, the need to tackle data volumes
>> >>> that exceed the capabilities of a single server led to the
>> >>> development of “shared-nothing” parallel database systems several
>> >>> decades ago. These systems spread data over a cluster based on a
>> >>> partitioning strategy, such as hash partitioning, and queries are
>> >>> processed by employing partitioned-parallel divide-and-conquer
>> >>> techniques. Since these systems are fronted by a high-level,
>> >>> declarative language (SQL), their users are shielded from the
>> >>> complexities of parallel programming. Parallel database systems have
>> >>> been an extremely successful application of parallel computing, and
>> >>> quite a number of commercial products exist today.
>> >>>
>> >>> In the distributed systems world, the Web brought a need to index and
>> >>> query its huge content. SQL and relational databases were not the
>> >>> answer, though shared-nothing clusters again emerged as the hardware
>> >>> platform of choice. Google developed the Google File System (GFS) and
>> >>> MapReduce programming model to allow programmers to store and process
>> >>> Big Data by writing a few user-defined functions. The MapReduce
>> >>> framework applies these functions in parallel to data instances in
>> >>> distributed files (map) and to sorted groups of instances sharing a
>> >>> common key (reduce) -- not unlike the partitioned parallelism in
>> >>> parallel database systems. Apache's Hadoop MapReduce platform is the
>> >>> most prominent implementation of this paradigm for the rest of the
>> >>> Big Data community. On top of Hadoop and HDFS sit declarative
>> >>> languages like Pig and Hive that each compile down to Hadoop
>> >>> MapReduce jobs.
>> >>>
>> >>> The big Web companies were also challenged by extreme user bases
>> >>> (100s of millions of users) and needed fast simple lookups and
>> >>> updates to very large keyed data sets like user profiles. SQL
>> >>> databases were deemed either too expensive or not scalable, so the
>> >>> “NoSQL movement” was born. The ASF now has HBase and Cassandra,
two
>> >>> popular key-value stores, in this space. MongoDB and Couchbase are
>> >>> other open source alternatives (document stores).
>> >>>
>> >>> It is evident from the rapidly growing popularity of "NoSQL" stores,
>> >>> as well as the strong demand for Big Data analytics engines today,
>> >>> that there is a strong (and growing!) need to store, process, *and*
>> >>> query large volumes of semi-structured data in many application
>> >>> areas. Until very recently, developers have had to ``choose'' between
>> >>> using big data analytics engines like Apache Hive or Apache Spark,
>> >>> which can do complex query processing and analysis over HDFS-resident
>> >>> files, and flexible but low-function data stores like MongoDB or
>> >>> Apache HBase. (The Apache Phoenix project,
>> >>> http://phoenix.apache.org/, is a recent SQL-over-HBase effort that
>> >>> aims to bridge between these choices.)
>> >>>
>> >>> AsterixDB is a highly scalable data management system that can store,
>> >>> index, and manage semi-structured data, e.g., much like MongoDB, but
>> >>> it also supports a full-power query language with the expressiveness
>> >>> of SQL (and more). Unlike analytics engines like Hive or Spark, it
>> >>> stores and manages data, so AsterixDB can exploit its knowledge of
>> >>> data partitioning and the availability of indexes to avoid always
>> >>> scanning data set(s) to process queries. Somewhat surprisingly, there
>> >>> is no open source parallel database system (relational or otherwise)
>> >>> available to developers today -- AsterixDB aims to fill this need.
>> >>> Since Apache is where the majority of the today's most important Big
>> >>> Data technologies live, the ASF seems like the obvious home for a
>> >>> system like AsterixDB.
>> >>>
>> >>> Current Status
>> >>>
>> >>> The current version of AsterixDB was co-developed by a team of
>> >>> faculty, staff, and students at UC Irvine and UC Riverside. The
>> >>> project was initiated as a large NSF-sponsored project in 2009, the
>> >>> goal of which was to combine the best ideas from the parallel
>> >>> database world, the then new Hadoop world, and the semi-structured
>> >>> (e.g., XML/JSON) data world in order to create a next-generation
>> >>> BDMS. A first informal open source release was made four years later,
>> >>> in June of 2013, under the Apache Software License 2.0.
>> >>>
>> >>>
>> >>> Meritocracy
>> >>>
>> >>> The current developers are familiar with meritocratic open source
>> >>> development at Apache. Apache was chosen specifically because we want
>> >>> to encourage this style of development for the project.
>> >>>
>> >>>
>> >>> Community
>> >>>
>> >>> While AsterixDB started as a university project it has developed into
>> >>> a community. A number of the initial committers started contributing
>> >>> in academia and continue to actively participate and contribute after
>> >>> graduation. And we seek to further develop developer and user
>> >>> communities. One way to broaden the community that is ongoing is
>> >>> through academic collaborations (currently with IIT Mumbai in India
>> >>> and TU Berlin in Germany). During incubation we will also explicitly
>> >>> seek increased industrial participation.
>> >>>
>> >>> Some indicators of the effort's development community and history can
>> >>> be
>> >>> found at:
>> >>>
>> https://www.openhub.net/p/asterixdb/contributors?query=&sort=commits_12_mo
>> ,
>> >>>
>> https://www.openhub.net/p/hyracks/contributors?query=&sort=commits_12_mo
>> >>>
>> >>>
>> >>> Core Developers
>> >>>
>> >>> The core developers of the project are diverse, although initially UC
>> >>> Irvine heavy (roughly 50) due to the project's origins at UCI. The
>> >>> other 50 are from other academic institutions (UC Riverside and the
>> >>> Hebrew University in Jerusalem) and companies (Couchbase, Facebook,
>> >>> IBM, KACST Saudi Arabia, Oracle, Saudi Aramco, X15 Software).
>> >>>
>> >>>
>> >>> Alignment
>> >>>
>> >>> Apache is, by far, the most natural home for taking the AsterixDB
>> >>> project forward. A large fraction of today's top Big Data
>> >>> technologies have their homes in Apache, including Hadoop, YARN, Pig,
>> >>> Hive, Spark, Flink, HBase, Cassandra and others. AsterixDB fills a
>> >>> significant gap -- the parallel data management system gap -- that
>> >>> exists in the Big Data open source world. It is well-aligned with a
>> >>> number of the Apache projects, e.g., it has strong support for
>> >>> accessing and indexing external data in HDFS, and it uses YARN as an
>> >>> answer to basic cluster resource management. AsterixDB also seeks to
>> >>> achieve an Apache-style development model; it is seeking a broader
>> >>> community of contributors and users in order to achieve its full
>> >>> potential and value to the Big Data community.
>> >>>
>> >>> There are also a number of related Apache projects and dependencies
>> >>> that will be mentioned below in the Relationships with Other Apache
>> >>> products section.
>> >>>
>> >>>
>> >>> Known Risks
>> >>>
>> >>> Orphaned products
>> >>>
>> >>> Given the current level of intellectual investment in AsterixDB, the
>> >>> risk of the project being abandoned is very small. The UCI/UCR
>> >>> faculty team leads are highly incentivized to continue development
>> >>> since the database groups at UC Irvine and UC Riverside are both
>> >>> reliant on AsterixDB as a platform for long-term graduate research
>> >>> projects. UC San Diego is also beginning to contribute to the code
>> >>> base, and a collaboration involving public health applications is
>> >>> forming with UCLA. The work on AsterixDB is managed via a mix of
>> >>> mailing list discussions supplemented by weekly project status
>> >>> meetings which are summarized on the mailing list. Typical (local
>> >>> plus Skype-in) attendance to the weekly status meetings runs at about
>> >>> 20 active contributors.
>> >>>
>> >>>
>> >>> Inexperience with Open Source
>> >>>
>> >>> AsterixDB and Hyracks were completely developed in Open Source under
>> >>> the ASL 2.0. The source code repositories, issue tracker, and mailing
>> >>> lists are available on Google Code and discussions and decisions
>> >>> happen on the mailing lists (which is necessary due to the geographic
>> >>> distribution of the current developers).
>> >>>
>> >>> Also a few of the initial committers have contributed to Apache
>> >>> projects. Vinayak Borkar is a committer on the Apache Helix and
>> >>> Apache VXQuery projects. Till Westmann is the VP VXQuery at the ASF
>> >>> and an IPMC member. Preston Carman and Steven Jacobs are committers
>> >>> on the Apache VXQuery project.
>> >>>
>> >>>
>> >>> Relationships with Other Apache Products
>> >>>
>> >>> Apache VXQuery is based on the Hyracks data-parallel runtime, which
>> >>> is also included in the AsterixDB code base.
>> >>>
>> >>> AsterixDB is closely related to Apache Hadoop. Included in AsterixDB
>> >>> is support for accessing external data in HDFS (and Hive formats),
>> >>> and resource management and system administration features are in the
>> >>> process of being migrated to YARN.
>> >>>
>> >>> AsterixDB's AQL query facilities offer comparable query power to
>> >>> Apache's Pig and Hive systems for big data analytics. AsterixDB
>> >>> differs in storing and indexing data and thus being able to quickly
>> >>> answer small and medium queries without large HDFS data scans -
>> >>> thereby targeting a different class of use cases.
>> >>>
>> >>> AsterixDB's data storage and indexing facilities are similar to those
>> >>> of HBase, but AsterixDB differs in being a much more complete and
>> >>> queryable BDMS (not just a key-value style store).
>> >>>
>> >>> AsterixDB's target use cases are not in-memory processing or
>> >>> iterative algorithm support, making AsterixDB complementary to the
>> >>> Apache Spark platform. (Spark interoperability is on our longer-term
>> >>> to-do wishlist.)
>> >>>
>> >>>
>> >>> Homogeneous Developers
>> >>>
>> >>> As mentioned before the current community is already organizationally
>> >>> and geographically distributed - and we would like to increase the
>> >>> heterogeneity.
>> >>>
>> >>>
>> >>> Reliance on Salaried Developers
>> >>>
>> >>> Of the initial committers only 3 are full-time UCI staff. The other
>> >>> committers are a mix of students, alumni who continue to contribute
>> >>> to the effort, and individuals working with permission part-time (or
>> >>> in spare time) on this project.
>> >>>
>> >>>
>> >>> A Excessive Fascination with the Apache Brand
>> >>>
>> >>> We believe in the processes, systems, and framework Apache has put in
>> >>> place. Apache is also known to foster a great community around their
>> >>> projects and provide exposure. While brand is important, our
>> >>> fascination with it is not excessive. We believe that the ASF is the
>> >>> right home for AsterixDB and that having AsterixDB inside of the ASF
>> >>> will lead to a better long-term outcome for the Big Data community.
>> >>>
>> >>>
>> >>> Documentation
>> >>>
>> >>> Documentation and publications related to AsterixDB can be found at
>> >>> http://asterixdb.ics.uci.edu/.
>> >>>
>> >>>
>> >>> Initial Source
>> >>>
>> >>> Current source resides in Google code:
>> >>> https://code.google.com/p/asterixdb/ (query language and upper system
>> >>> layers) and https://code.google.com/p/hyracks/ (dataflow runtime
>> >>> system and storage management libraries).
>> >>>
>> >>>
>> >>> External Dependencies
>> >>>
>> >>> AsterixDB depends on a number of Apache projects:
>> >>>
>> >>> - Ant
>> >>> - Avro
>> >>> - ApacheDB JDO
>> >>> - Commons
>> >>> - Derby
>> >>> - Hadoop
>> >>> - Hive
>> >>> - HTTPComponents
>> >>> - Jakarta ORO
>> >>> - Maven
>> >>> - Tomcat
>> >>> - Thrift
>> >>> - Velocity
>> >>> - Wicket
>> >>> - Xerces
>> >>>
>> >>> and other open source projects (organized by license):
>> >>>
>> >>> -- ASL 2.0:
>> >>> - Jackson
>> >>> - Google Guava
>> >>> - Google Guice
>> >>> - JSON-simple
>> >>> - BoneCP
>> >>> - Microsoft Azure SDK
>> >>> - Netty
>> >>> - Rome
>> >>> - JetS3t
>> >>> - Groovy
>> >>> - Jettison
>> >>> - Plexus
>> >>> - Datanucleus (JDO)
>> >>> - Jetty
>> >>> - Twitter4J
>> >>> - Snappy-java
>> >>>
>> >>> -- BSD:
>> >>> - Antlr
>> >>> - ObjectWeb ASM
>> >>> - Protobuf
>> >>> - JSCH
>> >>> - JavaCC
>> >>> - Paranamer
>> >>> - JLine
>> >>> - Stax
>> >>> - StringTemplate
>> >>> - xmlEnc
>> >>>
>> >>> -- MIT
>> >>> - AppAssembler
>> >>> - SimpleLog4J
>> >>>
>> >>> -- CDDL 1.0
>> >>> - Java Activation Framework
>> >>> - Java Transactions
>> >>> - Java Servlet API
>> >>> - Grizzly
>> >>> - gmbal
>> >>> - Glassfish
>> >>>
>> >>> -- CDDL 1.1
>> >>> - Jersey
>> >>> - JAXB Reference Implementation
>> >>>
>> >>> -- JSON License
>> >>> - JSON
>> >>>
>> >>> -- EPL 1.0
>> >>> - JUnit
>> >>>
>> >>> -- JDOM License
>> >>> - JDOM
>> >>>
>> >>> -- Public Domain
>> >>> - xz
>> >>> - AOPAlliance
>> >>>
>> >>> As all dependencies are managed using Apache Maven, none of the
>> >>> external libraries need to be packaged in a source distribution.
>> >>>
>> >>>
>> >>> Required Resources
>> >>>
>> >>> Developer and user mailing lists
>> >>>
>> >>> private@asterixdb.incubator.apache.org (with moderated subscriptions)
>> >>> commits@asterixdb.incubator.apache.org
>> >>> dev@asterixdb.incubator.apache.org
>> >>> users@asterixdb.incubator.apache.org
>> >>>
>> >>>
>> >>> A git repository
>> >>>
>> >>> https://git-wip-us.apache.org/repos/asf/incubator-asterixdb.git
>> >>>
>> >>>
>> >>> A JIRA issue tracker
>> >>>
>> >>> https://issues.apache.org/jira/browse/ASTERIXDB
>> >>>
>> >>>
>> >>> Initial Committers
>> >>>
>> >>> The following is a list of the planned initial Apache committers (the
>> >>> active subset of the committers for the current repository at Google
>> >>> code).
>> >>>
>> >>> Abdullah Alamoudi (bamousaa@gmail.com)
>> >>> Cameron Samak (eufery@gmail.com)
>> >>> Chen Li (chenli@gmail.com)
>> >>> Ian Maxon (imaxon@uci.edu)
>> >>> Ildar Absalyamov (ildar.absalyamov@gmail.com)
>> >>> Jianfeng Jia (jianfeng.jia@gmail.com)
>> >>> Karen Ouaknine (kereno@gmail.com)
>> >>> Markus Dreseler (apache@dreseler.de)
>> >>> Mike Carey (dtabass@apache.org)
>> >>> Murtadha Hubail (hubailmor@gmail.com)
>> >>> Pouria Pirzadeh (pouria.pirzadeh@gmail.com)
>> >>> Preston Carman (prestonc@apache.org)
>> >>> Raman Grover (RamanGrover29@gmail.com)
>> >>> Sattam Alsubaiee (salsubaiee@gmail.com)
>> >>> Steven Jacobs (sjaco002@apache.org)
>> >>> Taewoo Kim (wangsaeu@gmail.com)
>> >>> Till Westmann (tillw@apache.org)
>> >>> Vinayak Borkar (vinayakb@apache.org)
>> >>> Yingyi Bu (buyingyi@gmail.com)
>> >>> Young-Seok Kim (kisskys@gmail.com)
>> >>> Zach Heilbron (zheilbron@gmail.com)
>> >>>
>> >>>
>> >>> Affiliations
>> >>>
>> >>> UC Irvine
>> >>> - Mike Carey
>> >>> - Chen Li
>> >>> - Ian Maxon
>> >>> - Yingyi Bu
>> >>> - Raman Grover
>> >>> - Pouria Pirzadeh
>> >>> - Young-Seok Kim
>> >>> - Cameron Samak
>> >>> - Taewoo Kim
>> >>> - Jianfeng Jia
>> >>> - Murtadha Hubail
>> >>> - Markus Dreseler
>> >>>
>> >>> UC Riverside
>> >>> - Ildar Absalyamov
>> >>> - Preston Carman
>> >>> - Steven Jacobs
>> >>>
>> >>> Hebrew University
>> >>> - Keren Ouaknine
>> >>>
>> >>> Oracle
>> >>> - Till Westmann
>> >>>
>> >>> X15 Software
>> >>> - Vinayak Borkar
>> >>> - Zach Heilbron
>> >>>
>> >>> KACST Saudi Arabia
>> >>> - Sattam Alsubaiee
>> >>>
>> >>> Saudi Aramco
>> >>> - Abdullah Alamoudi
>> >>>
>> >>> Carey, Li, and Maxon are full-time UCI staff, with the remaining UCI
>> >>> (UC Irvine) and UCR (UC Riverside) affiliates being students. The
>> >>> non-UC committers are a mix of alumni who continue to contribute to
>> >>> the effort and individuals working with permission part-time (or in
>> >>> spare time) on this project.
>> >>>
>> >>>
>> >>> Sponsors
>> >>>
>> >>> Champion
>> >>>
>> >>> Chris Mattmann (NASA/JPL)
>> >>>
>> >>> Nominated Mentors
>> >>>
>> >>> TBD
>> >>>
>> >>> Sponsoring Entity
>> >>>
>> >>> The Apache Incubator
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Chris Mattmann, Ph.D.
>> >>> Chief Architect
>> >>> Instrument Software and Science Data Systems Section (398)
>> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>> Office: 168-519, Mailstop: 168-527
>> >>> Email: chris.a.mattmann@nasa.gov
>> >>> WWW:  http://sunset.usc.edu/~mattmann/
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Adjunct Associate Professor, Computer Science Department
>> >>> University of Southern California, Los Angeles, CA 90089 USA
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>
>> >>>
>> >>>
>> >>>
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message