incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <>
Subject [PROPOSAL] Phoenix for Incubation
Date Wed, 13 Nov 2013 20:43:31 GMT
Hi All,

We're pleased to share a draft ASF incubation proposal for Phoenix, a
SQL layer over HBase, initially developed at and
subsequently open sourced on github
( Instead of using Map-reduce
to processes queries, it compiles SQL directly into native HBase
calls. The complete proposal can be found here:, and is also pasted

Your feedback is greatly appreciated.


== Abstract ==
Phoenix is an open source SQL query engine for Apache HBase, a NoSQL
data store.  It is accessed as a JDBC driver and enables querying and
managing HBase tables using SQL.

== Proposal ==
Phoenix is an open source SQL skin over HBase delivered as a
client-embedded JDBC driver targeting low latency queries over HBase
data. Phoenix takes your SQL query, compiles it into a series of HBase
scans, and orchestrates the running of those scans to produce regular
JDBC result sets. The table metadata is stored in an HBase table and
versioned, such that snapshot queries over prior versions will
automatically use the correct schema. Direct use of the HBase API,
along with coprocessors and custom filters, results in performance on
the order of milliseconds for small queries, or seconds for tens of
millions of rows. Phoenix interfaces with both Pig and Map-reduce for
the input and output of data.

== Background ==
Phoenix initially started as an internal project at to
efficiently analyze big data stored in HBase. It was open sourced on
Github about a year ago in Jan 2013. Over time Phoenix, together with
HBase as the storage tier, has begun to evolve into a general SQL
database with support for metadata management, secondary indexes,
joins, query optimization, and multi-tenancy. This is expected to
continue as Phoenix implements a cost-based query optimizer and
potentially transaction support, and surfaces new HBase security
features such as encryption and cell-level security. Phoenix's
developer community has also grown to include additional companies
such as Intel, who have contributed join support to Phoenix, as well
as Hortonworks, who are in the process of porting Phoenix to the 0.96
release of HBase.

== Rationale ==
As usage and the number of contributors to Phoenix has grown, we have
sought for a long-term home for the project, and we believe the Apache
foundation would be a great fit. Joining Apache would ensure that
tried and true processes and procedures are in place for the growing
number of organizations interested in contributing to Phoenix. Phoenix
is also a good fit for the Apache foundation: Phoenix already
interoperates with several existing Apache projects (HBase, Hadoop,
Pig). The Phoenix team is familiar with the Apache process and and
believes in the Apache mission - the team already includes multiple
Apache committers.

== Initial Goals ==
The initial goals will be to move the existing codebase to Apache and
integrate with the Apache development process. Once this is
accomplished, we plan for incremental development and releases that
follow the Apache guidelines.

== Current Status ==
Phoenix has undergone two major and three minor releases (1.0, 1.1,
1.2, 2.0, and 2.1) as well as many patch releases. Phoenix is being
used in production by as well as at other
organizations. The Phoenix codebase is currently hosted at,
which will form the basis of the Apache git repository.

=== Meritocracy ===
The Phoenix project already operates on meritocratic principles.
Phoenix has several developers from various organizations outside of who have contributed major new features. While this
process has remained mostly informal, as we do not have an official
committer list, an implicit organization exists in which individuals
who contribute major components act as maintainers for those modules.
If accepted, the Phoenix project would include several of these
participants as initial committers. We will work to identify all
committers and PPMC members for the project and to operate under the
ASF meritocratic principles.

=== Community ===
Acceptance into the Apache foundation would bolster the already strong
user and developer community around Phoenix. That community includes
many contributors from various other companies, and an active mailing
list composed of hundreds of users.

=== Core Developers ===
The core developers of our project are listed in our contributors and
initial PPMC below. Though many are employed at, there
is a representative cross sampling of other organizations including
Intel, Hortonworks, Cloudera, and Twitter.

=== Alignment ===
Our proposed Phoenix effort aligns closely with Apache HBase. The
HBase project perimeter is denoted by a simple byte-array based
Create, Read, Update, Delete and Scan APIs with no current plans to
extend beyond this bounds. Phoenix complements this with a higher
level API in SQL with which many are already familiar. At first
glance, it may seem that Phoenix should just be folded into HBase as a
new module. However, the focus of the two projects will be quite
different, especially as Phoenix matures. With secondary indexing and
joins just having been introduced into Phoenix, the next big frontier
will be to implement a cost-based query optimizer. This is the
heart-and-soul of most relational databases and can can take a
lifetime to get right.

HBase is focused on being a scalable data store agnostic to types and
schema.  Phoenix would layer typing, and relational facilities on top
of this scalable store. By keeping Apache HBase and Phoenix separate,
both may evolve independently and at different rates. Though the focus
of the two projects is different, the relationship between them is
very positive and mutually beneficial. New features in HBase will be
leveraged in Phoenix as it makes sense to surface these in a SQL
paradigm. In addition, Phoenix may drive new features in HBase, as
evidenced by the new type system recently introduced into HBase. This
will enable better interoperability between Apache Hive, standalone
HBase uses case, and Phoenix by defining a standard serialization

Other projects exists that perform SQL over HBase data (such as Apache
Hive), however these products do not provide the same low latency
query capabilities as Phoenix. Instead, they are more oriented around
maximizing throughput for batched operations. Phoenix opens the door
to a completely new set of use cases for Apache HBase that demand a
more interactive user experience.

There are also a number of related Apache projects and dependencies
that are mentioned in the Relationships with Other Apache products

== Known Risks ==
=== Orphaned Products ===
Given the current level of investment in Phoenix - the risk of the
project being abandoned is minimal. All current and planned HBase use
cases at go through Phoenix. In addition, both Intel
and Hortonworks plan to include Phoenix in their distributions. Other
companies have devoted significant internal infrastructure investment
in Phoenix.

=== Inexperience with Open Source ===
Phoenix has existed as a healthy open source project for almost a
year. During that time, James, Mujtaba, and others have successfully
fostered an open-source community, attracting users and developers
from a diverse group of companies including Intel, Intuit, Bloomberg,
Tagged, and Hortonworks. Although neither are committers on other
Apache projects, both James and Mujtaba have experience working with
and contributing to other Apache projects.

=== Homogenous Developers ===
The initial list of committers includes developers from several
institutions, including Salesforce, Intel, Hortonworks, and Twitter.

=== Reliance on Salaried Developers ===
Like most open source projects, Phoenix receives substantial support
from salaried developers. A large fraction of Phoenix development is
supported by In addition, those working from within
corporations and universities often devote “after hours” or spare time
to the project. We will continue our efforts to ensure stewardship of
the project to be independent of salaried developers.

=== Relationship with Other Apache Products ===
Although Phoenix provides a higher level abstraction than Apache HBase
by hiding its client APIs, Phoenix relies on Apache HBase for both
storing and retrieving data. It also inter-operates with Apache HBase
by allowing existing data, not created by Phoenix, to be queried. In
addition, both Apache Pig and Hadoop are supported for data input and
output. Finally, the Phoenix is included and installable through
Apache Bigtop and the build and test suite are run through Apache

Phoenix offers an alternative query engine to Apache Hadoop
(MapReduce). Unlike MapReduce, Phoenix is designed for lower-latency,
OLTP, and interactive workloads. This makes the projects complimentary
as users may run MapReduce and Phoenix side-by-side.

We plan to increase the interoperability between Phoenix, Apache Hive,
and standalone Apache HBase usage by standardizing on a new type
system that has been introduced in the current major release of HBase.
By all these products adopting this new serialization format,
interoperability between them will take a big step forward.

In addition, we plan to explore providing lower level APIs for other
products such as Apache Drill to plug into when querying HBase data so
that they get the performance benefits of using Phoenix.

=== A Excessive Fascination with the Apache Brand ===
Phoenix is already a healthy and relatively well known open source
project. This proposal is not for the purpose of generating publicity.
Rather, the primary benefits to joining Apache are those outlined in
the Rationale section.

=== Documentation ===
Additional documentation on Phoenix may be found on its github website:
 * Phoenix overview:
 * Phoenix wiki:
 * Phoenix road map:
 * Phoenix issue tracking:
 * Phoenix codebase:
 * Phoenix SQL language reference:
 * Phoenix performance:
 * User group:

== Initial Source ==
The Phoenix codebase is currently hosted on Github:

=== Source and Intellectual Property Submission Plan ===
Currently, the Phoenix codebase is distributed under a BSD license.
Upon entering Apache, the Phoenix license will be migrated to the
Apache 2.0 License.

== External Dependencies ==
Beyond relying on Apache HBase, Phoenix has the following external dependencies:
 * ANTLR 3.5 (BSD license:
 * Sqlline 1.1.2 (BSD license:
 * Open CSV 2.3 (Apache 2.0 license)

Upon acceptance to the incubator, we would begin a thorough analysis
of all transitive dependencies to verify this information and
introduce license checking into the build and release process by
integrating with Apache Rat.

== Required Resources ==
=== Mailing list ===
We will migrate the existing Phoenix mailing lists as follows:

 * -->
 * -->
 * for IPMC members

The latter is to be consistent with the new PIAO naming scheme for podlings.

=== Source control ===
The Phoenix team would like to use Git for source control, due to our
current use of Git.
We request a writeable Git repo for Phoenix, and mirroring to be set
up to Github through INFRA.

=== Issue Tracking ===
Phoenix currently uses the github issue tracking system associated
with its github repo:
We will migrate to the Apache JIRA:

=== Other Resources ===
 * Jenkins/Hudson for builds and test running.
 * Wiki for documentation purposes
 * Blog to improve project dissemination

== Initial Committers ==
 * James Taylor <jtaylor at salesforce dot com>
 * Mujtaba Chohan <mchohan at salesforce dot com>
 * Jesse Yates <jyates at apache dot org>
 * Eli Levine <elevine at salesforce dot com>
 * Simon Toens <stoens at salesforce dot com>
 * Maryann Xue <wei.xue at intel dot com>
 * Anoop Sam John <anoopsamjohn at apache dot org>
 * Ramkrishna S Vasudevan <ramkrishna at apache dot org>
 * Jeffrey Zhong <jeffreyz at apache dot org>
 * Nick Dimiduk <ndimiduk at apache dot org>
 * Tony Huang <thuang at twitter dot com>

== Affiliations ==
The initial committers are from four organizations:,
Intel, Hortonworks, and Twitter.

 * James Taylor (
 * Mujtaba Chohan (
 * Jesse Yates (
 * Eli Levine (
 * Simon Toens (
 * Maryann Xue (Intel)
 * Anoop Sam John (Intel)
 * Ramkrishna S Vasudevan (Intel)
 * Jeffrey Zhong (Hortonworks)
 * Nick Dimiduk (Hortonworks)
 * Tony Huang (Twitter)

== Sponsors ==
=== Champion ===
 * Michael Stack

=== Nominated Mentors ===
 * Michael Stack
 * Lars Hofhansl
 * Andrew Purtell
 * Devaraj Das
 * Enis Soztutar

=== Sponsoring Entity ===
 The Apache Incubator

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message