incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Charles <>
Subject Re: [PROPOSAL] Curator for the Apache Incubator
Date Tue, 26 Feb 2013 08:02:31 GMT

Inputs from a current Curator user:

I followed one year ago the public thread Patrick mentionned and was 
surprised to see the lack of interest for an higher level Zk client.

If this discussion happened now, maybe the interest would be different, 
but still, I think Curator should be hosted in its own community for the 
following reasons:

1.- I would love to have a client that support multiple server types. I 
use for example Curator to take distributed locks on Zk servers, but 
other servers (Hazelcast...) also implement this functionality. Simply 
by configuration, I would love Curator to give me access to those 
servers without chaning my code. Ok, This is not in the proposal scope, 
but having a separate community opens to door to those type of 
contributions. Obviously, if Curator is part of Zookeeper, this will be 
more difficult.

2.- Zookeeper is getting more and more functionalities (Bookeeper, 
Hedwig...) but not all of these functionalities may find an client part 
in Curator. This is why (see my first point), Curator should not be 
stricly a full Zk client, but a client providing access in some way to 
some distributed server functions.

3.- Client dev (internal machinery such injection, configuration...) is 
quite different from server dev. Some client architecture decisions 
should not be depende on server dev habits and server developers tend to 
negligate clients. I feel everyday the pain of real 
hadoop/hbase/casssandra/... client (I mean without the server jars). 
This is evolving, but not having a separate project for client does not 
help. HTTP client have never been developer in server projects.

I agree that hosting in the same community can create some convergence 
and ease the communication (API, bug solving...), but I still feel 
Curator deserves a dedicated community, especially for the first reason 
I have listed (client towards multiple servers).

Thx, Eric

On 26/02/2013 06:10, Patrick Hunt wrote:
> On Mon, Feb 25, 2013 at 7:10 PM, Greg Stein <> wrote:
>> My concern is that we're looking at two "new" committers, rather than
>> a Curator community. Following normal Incubator work, Curator would
>> build a community for itself. But then we'd have a community
>> *distinct* from that of Zookeeper. And it really looks like this
>> should be part of Zookeeper itself -- a more capable and easier-to-use
>> client.
>> So I question the incubation of this. Why do we want to build a
>> new/separate project? Why isn't this just part of Zookeeper right from
>> the start?
>> I would suggest that this work is placed on a branch within Zookeeper.
>> That Jordan and Jay become committers on that branch (not necessarily
>> Zookeeper trunk). Over time, the branch can be folded into trunk,
>> along with all the various tests, doc, and other artifacts that I see
>> in the GitHub repository. And hopefully that Jordan and Jay become
>> regular committers (and PMC members!) of the Zookeeper project itself.
>> The current Zookeeper client can remain for backwards compat, and the
>> Curator work can become the next-gen client.
>> Honestly, in my opnion, incubating this project seems like it would
>> create a distinct community, and really doesn't seem like it would
>> serve the Zookeeper community.
>> All that said, I am not familiar with the Zookeeper or Curator
>> communities. But from this read, I don't think Incubation is the right
>> approach. I would rather push for a more direct incorporation of
>> Curator directly into Zookeeper. (use the short-form IP clearance) ...
>> so, unless somebody can help me understand the communities and
>> situation better, I'm -1 (binding) on this incubation. I'd rather see
>> combined, rather than distinct, communities from the start.
> The Curator library is built upon the current ZooKeeper client
> libraries. They are an extension that implements higher level use
> cases (what we call "recipes" in ZK) whereas the ZK libraries
> implement lower level primitives. An analogy might be Apache Crunch
> (recently graduated to TLP) and MapReduce. Also, not everyone agrees
> with Jordan's assessment that Curator is 'better" than (or a
> replacement for) the existing client libraries.
> The ZK community discussed incorporating the Curator code about a year
> ago. At that time there wasn't interest to adopt these libraries into
> ZK itself. My belief is that if Curator were to go through incubation
> the ZK community would be more likely to adopt it. Similar to how
> HCatalog spun off to work on the metastore and recently merged back
> into Hive. If there's significant overlap in the Curator podling
> community and the ZK community that will be a strong indication that
> they should be merged (my belief). If not it would have the
> opportunity to go TLP.
> Here's my report (ZooKeeper) to the board in Dec 2011 and the response
> I received. This is one of the reasons I advised Jordan to go the
> incubator approach. There's also the discussion thread on ZooKeeper
> dev if you want more insight.
>>> A discussion is currently under way regarding the possibility of> merging
the recently open sourced "Curator" source base from> Netflix. These are client implementations
of ZooKeeper "recipes"> (e.g. leadership election, group membership, etc...) which simplify>
the act of using ZooKeeper in client side applications.>> The authors of Curator are
unwilling to join the incubator, this is> based on their past experience as well as some
of the ongoing issues> they are seeing wrt projects entering the incubator. They have>
expressed a preference to come in either as a subproject or as a> separate release artifact
of the TLP.
>> to which the board responded:
>> Doug Cutting (chairman):
>> +               Long-term, do you think that Curator will have an
>> +               indepdendent community from Zookeeper?  If so, then it
>> +               ought to enter through the Incubator.  If not then the
>> +               code might still enter through the incubator for
>> +               resolution of IP issues, but then transfer relatively
>> +               quickly to live under the Zookeeper PMC.  Or the
>> +               Zookeeper PMC could directly adopt the code, although
>> +               you might not immediately add its committers to the
>> +               project but rather have them first contribute patches
>> +               through the normal process until their community merit
>> +               is established.
>> Sam Ruby:
>> + suggested that the "short form" of the incubator's IP clearance
>> process would be appropriate if Zookeeper directly adopts this code.
>> The short form can be found here:
> Regards,
> Patrick
>> On Mon, Feb 25, 2013 at 8:14 PM, Jordan Zimmerman
>> <> wrote:
>>> Hello,
>>> I would like to propose that Curator to be an Apache Incubator project.
>>> The proposal can be found here:
>>> I have included the contents of the proposal below.
>>> Sincerely,
>>> Jordan Zimmerman
>>> ===================
>>> = Curator - ZooKeeper client wrapper and rich ZooKeeper framework =
>>> == Abstract ==
>>> Curator is a set of Java libraries that make using Apache ZooKeeper much easier.
While ZooKeeper comes bundled with a Java client, using the client is non-trivial and error
>>> == Proposal ==
>>> Curator is a set of Java libraries that make using Apache ZooKeeper much easier.
While ZooKeeper comes bundled with a Java client, using the client is non-trivial and error
prone. It consists of three components that build on each other. Curator Client is a replacement
for the bundled ZooKeeper class that takes care of some low-level housekeeping and provides
some useful utilities. Curator Framework is a high-level API that greatly simplifies using
ZooKeeper. It adds many features that build on ZooKeeper and handles the complexity of managing
connections to the ZooKeeper cluster and retrying operations. Curator Recipes consists of
implementations of some of the common ZooKeeper ?recipes?. Additionally, Curator Test is included
which includes utilities to help with unit testing ZooKeeper-based applications.
>>> == Background ==
>>> Curator was initially developed by Netflix to make writing ZooKeeper-based applications
easier and more reliable. Curator was open-sourced by Netflix on !GitHub as an Apache 2.0
licensed project in July 2011. During this time Curator has been formally released many times
and has gained widespread adoption.
>>> == Rationale ==
>>> New users of ZooKeeper are surprised to learn that a significant amount of connection
management must be done manually. For example, when the ZooKeeper client connects to the ensemble
it must negotiate a new session, etc. This takes some time. If you use a ZooKeeper client
API before the connection process has completed, ZooKeeper will throw an exception. These
types of exceptions are referred to as ?recoverable? errors.
>>> Curator automatically handles connection management, greatly simplifying client
code. Instead of directly using the ZooKeeper APIs you use Curator APIs that internally check
for connection completion and wrap each ZooKeeper API in a retry loop. Curator uses a retry
mechanism to handle recoverable errors and automatically retry operations. The method of retry
is customizable. Curator comes bundled with several implementations (ExponentialBackoffRetry,
etc.) or custom implementations can be written.
>>> The ZooKeeper documentation describes many possible uses for ZooKeeper calling
each a ?recipe?. While the distribution comes bundled with a few implementations of these
recipes, most ZooKeeper users will need to manually implement one or more of the recipes.
Implementing a ZooKeeper recipe is not trivial. Besides the connection handling issues, there
are numerous edge cases that are not well documented that must be considered. For example,
many recipes require that an ephemeral-sequential node be created. New users of ZooKeeper
will not know that there is an edge case in ephemeral-sequential node creation that requires
you to put a special ?marker? in the node?s name so that you can search for the created node
if an I/O failure occurs. This is but one of many edge cases that are not well documented
but are handled by Curator.
>>> = Current Status =
>>> == Meritocracy ==
>>> Curator was initially developed by Jordan Zimmerman in 2011 at Netflix. Developers
external to Netflix provided feedback, suggested features and fixes and implemented extensions
of Curator. Netflix's engineering team has since maintained the project and has been dedicated
towards its improvement. Contributors to Curator include developers from multiple organizations
around the world. Curator will be a meritocracy as it enters the Incubator and beyond.
>>> == Community ==
>>> Curator is currently used by a number of organizations all over the world. Curator
has an active and growing user and developer community with active participation in the [[]]
mailing list and at its !Github home: [[]].
>>> Since open sourcing the project, there have been fifteen individuals from various
organizations who have contributed code.
>>> == Core Developers ==
>>> The core developers for Curator are:
>>>   * Jordan Zimmerman
>>>   * Jay Zarfoss
>>> Jordan has contributed towards Apache ZooKeeper and both Jordan and Jay are familiar
with Apache principles and philosophy for community driven software development.
>>> == Alignment ==
>>> Curator is a natural complement for Apache ZooKeeper. Java users of ZooKeeper
will naturally want to use Curator. When Curator graduates from Incubator it may be useful
to distribute Curator artifacts as part of ZooKeeper releases as the preferred/recommended
client side library. Further, at graduation a determination can be made as to whether Curator
should become a Top Level Project or be merged into ZooKeeper itself.
>>> = Known Risks =
>>> == Orphaned Products ==
>>> Curator is already deployed in production at multiple companies and they are
actively participating in creating new features. Curator is getting traction with developers
and thus the risks of it being orphaned are minimal.
>>> == Inexperience with Open Source ==
>>> All code developed for Curator has been open sourced by Netflix under Apache
2.0 license.  All committers to Curator are intimately familiar with the Apache model for
open-source development and are experienced with working with new contributors.
>>> == Homogeneous Developers ==
>>> The initial committers are from a single organization. However, we expect that
once approved for incubation, the project will attract new contributors from diverse organizations
and will thus grow organically. The submission of patches from developers from several different
organizations is a strong indication that Curator will be widely adopted.
>>> == Reliance on Salaried Developers ==
>>> It is expected that Curator will be developed on salaried and volunteer time,
although all of the initial developers will work on it mainly on salaried time.
>>> == Relationships with Other Apache Products ==
>>> Curator depends upon other Apache Projects: Apache ZooKeeper, Apache Log4J, and
multiple Apache Commons components. Its build depends upon Apache Maven. Notably, there is
interest from other Apache Projects such as HBase in adopting Curator as the client library
for ZooKeeper. Apache James Mailbox has already incorporated Curator.
>>> == An Excessive Fascination with the Apache Brand ==
>>> We would like Curator to become an Apache project to further foster a healthy
community of contributors and consumers around the project.  Since Curator directly interacts
with Apache ZooKeeper and solves an important problem of many ZooKeeper users, residing in
the Apache Software Foundation will increase interaction with the larger community.
>>> = Documentation =
>>>   * Curator wiki at GitHub:
>>>   * Curator issues at GitHub:
>>>   * Curator javadoc at GitHub:
>>> = Initial Source =
>>>   * git://
>>> == Source and Intellectual Property Submission Plan ==
>>>   * The initial source is already licensed under the Apache License, Version
>>> == External Dependencies ==
>>> The required external dependencies are all Apache License or compatible licenses.
Following components with non-Apache licenses are enumerated:
>>>   * org.slf4j: MIT-like License
>>>   * org.mockito: MIT-like License
>>> == Cryptography ==
>>> Curator contains no known cryptography.
>>> = Required Resources =
>>> == Mailing lists ==
>>>   * curator-private (with moderated subscriptions)
>>>   * curator-dev
>>>   * curator-commits
>>>   * curator-user
>>> == Github Repositories ==
>>> git://
>>> == Issue Tracking ==
>>> JIRA Curator (CURATOR)
>>> == Other Resources ==
>>> The existing code already has unit and integration tests so we would like a Jenkins
instance to run them whenever a new patch is submitted. This can be added after project creation.
>>> = Initial Committers =
>>>   * Jordan Zimmerman (jzimmerman at netflix dot com)
>>>   * Jay Zarfoss (jzarfoss at netflix dot com)
>>> = Affiliations =
>>>   * Jordan Zimmerman, Netflix
>>>   * Jay Zarfoss, Netflix
>>> = Sponsors =
>>> == Champion ==
>>>   * Patrick Hunt
>>> == Nominated Mentors ==
>>>   * Patrick Hunt
>>>   * Enis S÷ztutar
>>>   * Mahadev Konar
>>> == Sponsoring Entity ==
>>>   * Apache Incubator PMC
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message