incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Heneveld <>
Subject Re: [PROPOSAL] Brooklyn
Date Wed, 23 Apr 2014 17:29:56 GMT


Great!  We'd be pleased to have you.  Can you add yourself to the proposal?

+1 there are very cool possibilities integrating with both ACE and Kalumet


On 23/04/2014 17:18, Jean-Baptiste Onofré wrote:
> Hi,
> it's interesting, and I see potential interaction with Apache Kalumet 
> (and ACE also as Marcel stated).
> I would be pleased to be mentor and help on the project.
> Regards
> JB
> On 04/23/2014 04:28 PM, Chip Childers wrote:
>> The Brooklyn project [1] community is interested in moving to the 
>> ASF.  They
>> have worked within their community to create a proposal [2] for the
>> Incubator, and have managed to get agreement from 2 IPMC members to
>> mentor them if accepted (Matt Hogstrom and Alex Karasulu).
>> This is the discussion thread for this proposal.  We would also like to
>> solicit contributions from additional IPMC members willing to mentor the
>> potential podling.  Anyone interested?
>> -chip
>> [1]
>> [2]
>> Proposal follows:
>> = Apache Brooklyn =
>> == Abstract ==
>> Brooklyn is a framework for modeling, monitoring, and managing
>> applications through autonomic blueprints.
>> == Proposal ==
>> Brooklyn is about deploying and managing applications: composing a full
>> stack for an application; deploying to cloud and non-cloud targets;
>> using monitoring tools to collect key health/performance metrics;
>> responding to situations such as a failing node; and adding or removing
>> capacity to match demand.
>> A Brooklyn blueprint defines an application, using a declarative YAML
>> syntax supporting JVM plugins. A basic blueprint might comprise a single
>> process, such as a web-application server running a WAR file or a SQL
>> database and its associated DDL scripts. More complex blueprints
>> encompass combinations of processes across multiple machines and
>> services, such as a load-balancing HTTP server or SDN controller
>> fronting a cluster of J2EE application servers, in turn connected to a
>> resilient cluster of SQL database servers. Even larger clustered
>> application running in multiple regions can be described, with features
>> such as message buses with resilient brokers, cacheing tiers of NoSQL
>> key-value store servers, a high-availability database cluster and
>> multiple application components connected across these layers.
>> One main benefit of these blueprints is that they are composable:
>> best-practice blueprints for one process or pattern (e.g. a Cassandra
>> cluster) can be incorporated in other blueprints (e.g. an application
>> with a Cassandra cluster as one component). Another major benefit is
>> that the blueprints can be treated as source code as part of an
>> applications codebase: tested, tracked, versioned, and hardened as an
>> integral part of the devops process. In some ways, Brooklyn is to
>> run-time what Maven is to build-time.
>> Brooklyn knows about Chef, Salt, and similar tools, and APT and Yum and
>> plain old shell scripts, for deploying application components.
>> Blueprints are built from a mixture of both off-the-shelf packages such
>> as Tomcat, MySQL, Cassandra, and many more from our library; and
>> components that are bespoke to individual applications; together with
>> policies that allow the application to be self-managing.
>> Brooklyn is built for the cloud, and will take a blueprint and deploy it
>> to one of many supported clouds or even to multiple different clouds, or
>> to private infrastructure (bring-your-own-node), or to other platforms.
>> It will dynamically configure and connect all the different components
>> of an application, e.g. so load balancers know where the web servers are
>> and the web applications know where the database is.
>> Brooklyn collects key metrics to monitor the health of applications; for
>> example, by sending a request and measuring latency, or installing
>> monitoring tools and using those to read a server’s management interface
>> to determine the request queue length. These metrics can be fed into
>> policies, which automatically take actions such as restarting a failed
>> node, or scaling out the web tier if user demand exceeds capacity. This
>> allows an application to be self-managing: to recover itself from simple
>> failures, to scale out when demand increases and meet capacity; then
>> scale in as demand drops and stop paying for spare capacity.
>> In short, Brooklyn blueprints allow the best practices for deploying and
>> managing complex software to be codified as part of the software
>> development process.
>> === Background ===
>> Brooklyn is a product built from the ground up for application agility.
>> This includes portability across non-cloud, cloud, and PaaS targets;
>> devops-style infrastructure-as-code applied to applications; and
>> real-time autonomic management based on promise theory. Some
>> introductions to these concepts, associated tools, and open
>> specifications may be useful.
>> Cloud computing at its core is about provisioning resources on-demand.
>> The most widely known aspect is IaaS (infrastructure-as-a-service) such
>> as Amazon EC2, Softlayer, Google Cloud Platform, Apache CloudStack, or
>> OpenStack. By leveraging the Apache jclouds project (and contributing
>> heavily to it), the Brooklyn project is able to work with a large number
>> of such providers. Higher up the stack, however, there is an
>> increasingly diverse set of platform targets, from PaaS
>> (platform-as-a-service) such as Cloud Foundry and Apache Stratos,
>> through to myriad containers and runtime fabrics such as LXC/Docker,
>> Apache Mesos, Apache Karaf, Apache Hadoop, and Apache Usergrid and other
>> backend-as-a-service environments. Brooklyn is based on the premise that
>> applications may need to run in any or all of these, and the model must
>> be flexible and open enough to support this.
>> The buzzword-compliant trends of agile and devops have reinforced many
>> important lessons:
>>   * The truth is in the code (not any ancillary documents)
>>   * If it isn't tested then assume it isn't working
>>   * Toolchain integration and API's are key to a project's success
>>   * Even more critical is empowering all stakeholders to a project
>>   * Brooklyn's focus on blueprinting and modeling as code and API's
>>     serves these principles.
>> Another major influence on the design of Brooklyn are the ideas of
>> autonomic computing and promise theory. It is not necessary to have a
>> thorough understanding of these to use Brooklyn, but contributors tend
>> to become versed in these ideas quickly. Essentially, autonomics is
>> based on the concept of components looking after themselves where
>> possible (self-healing, self-optimizing, etc), and exposing a sensor
>> (data outputs) and effector (operations) API where they may need to
>> controlled by another element. Promise theory extends this approach by
>> introducing the idea that communicating intent (through promises) is a
>> more reliable basis for complex cooperating systems than
>> obligation-based effectors. Tools such as CF Engine, Chef, Puppet,
>> Ansible, and Salt apply promise theory to files and processes on
>> machines; Brooklyn can leverage all of these tools, complementing it
>> with an application-oriented model.
>> Finally we note some emerging standards in this area. OASIS CAMP (Cloud
>> Application Management for Platforms) and TOSCA (Topology and
>> Orchestration Specification for Cloud Applications) both define YAML
>> application models similar to Brooklyn's. CAMP focuses on the REST API
>> for interacting with such a management layer, and TOSCA focuses on
>> declarative support for more sophisticated orchestration. Currently
>> Brooklyn uses a YAML which complies with CAMP's syntax and exposes many
>> of the CAMP REST API endpoints. We would like to support the
>> hot-off-the-press TOSCA YAML and expand the CAMP REST API coverage.
>> === Rationale ===
>> Building and deploying applications in the cloud computing era has
>> changed many things. Provision a bare server just-in-time, and use
>> automated tools to install an application. Use APIs to add the server to
>> a load balancer. When load goes up, provision another server; when load
>> drops, kill a server to save money.
>> Many new tools have appeared that take advantage of this new era.
>> However each of them only solve part of the problem and don’t consider
>> the big picture. For example, configuration management tools such as
>> Chef can, in a single command, provision a new cloud server then install
>> and configure an application -- but they require extra programming to
>> reconfigure an load balancer whenever the pool of web servers changes.
>> Amazon Auto Scaling can provision new servers and update load balancers,
>> but it is dependent on CloudWatch -- this means either using proxy
>> metrics such as average response time, or writing more code to expose an
>> application’s real metrics. A dedicated monitoring tool may be able to
>> easily monitor the key metrics with little effort, but its alerts will
>> need to be integrated it into the server provisioning process.
>> So all the tools are there to to create and manage a cloud-scale
>> application that can adapt to demand to meet user expectations without
>> wasting money on superfluous services - but you will need several such
>> tools and it is up to you to integrate them into your deployment plan.
>> Some of these tools -- such as the Amazon Web Services web of EC2,
>> CloudWatch, AutoScaling and CloudFormation -- mean that you may suffer
>> from lock-in. Related projects in OpenStack (Heat, Ceilometer, Murano,
>> Solum, etc) provide similar functionality but again for a restricted
>> target. The most common policies (such as minimising request latency)
>> may be easy, but less common policies such as follow-the-sun and
>> follow-the-moon [1] may be up to you to implement. Your scaling policies
>> may understand that "high demand = add another server", but may not
>> understand requirements such as some clustered services requiring an odd
>> number of instances to prevent voting deadlocks.
>> In this context the advantage of Brooklyn becomes apparent: a single
>> tool is able to manage provisioning and application deployment, monitor
>> an application’s health and metrics, understand the dependencies between
>> components (such as knowing that adding a new web server means that the
>> load balancer needs reconfiguration) and apply complex policies to
>> manage the application. The tool provides a REST API and a GUI, and
>> allows the autonomic blueprints to be treated as an integral part of the
>> application. With Brooklyn, these policies become modular components
>> which can be reused and easily added to blueprints.
>> [1] Follow-the-sun refers to provisioning servers in network locations
>> close to users. User activity levels typically correlates to the time of
>> day, with most activity during daylight hours and less during the night.
>> Looking at a map showing daylight hours and server activity, demand
>> patterns will tend to “follow the sun”. Follow-the-moon refers to the
>> converse, that many compute facilities will be under-utilised during the
>> night, and cheaper pricing may be available on spot markets at these
>> times. Therefore, the location of the cheapest compute resources will
>> tend to “follow the moon”.
>> === Initial Goals ===
>> Brooklyn is a project in active development with some production
>> deployments, although it is currently at version 0.7.0-M1.
>> Our initial goals are to ensure that Brooklyn adopts the Apache Way, to
>> grow the community, and to continue development towards a 1.0 release
>> which makes a much-increased community very very happy.
>> == Current Status ==
>> === Meritocracy ===
>> We firmly believe in meritocracy. Since it was open sourced in April
>> 2012, Brooklyn has followed a published governance model based on
>> several Apache projects where Brooklyn members have been active.
>> Contributions can be made by anyone and are accepted after tests pass
>> and at least one committer has reviewed it (and checked the CLAs).
>> Contributors with a strong track record are invited to become committers
>> and join the management committee. We welcome the opportunity for
>> talented developers to become committers and join the PMC.
>> === Community ===
>> Brooklyn currently has a very active but small and homogenous community.
>> The core team is aligned to a single corporation, with just a few code
>> contributions externally. A larger and more diverse group of individuals
>> and companies are using Brooklyn, building blueprint projects with
>> Brooklyn dependenies, and discussing issues on the mailing list and in
>> pull requests. These numbers are growing slowly.
>> It is important that we foster and expand this community and attract
>> more diversity to our core team.
>> There are two routes to growing the community which are particularly
>> noteworthy, and both are rendered more attractive by moving Brooklyn to
>> Apache. ISV's and other projects can engage with the community as they
>> require blueprinting capabilities, including several Apache projects for
>> which Brooklyn blueprints already exist (see below). Secondly, some
>> enterprises and large organizations are already using Brooklyn and are
>> keen to see its future safeguarded by the Apache Foundation.
>> === Core Developers ===
>> The following core developers are proposed, being the current set of
>> committers to Brooklyn:
>> Alex Heneveld, the creator of Brooklyn, is active with several open
>> source projects (Apache jclouds, Apache Whirr, OpenStack) and standards
>> groups (OASIS CAMP & TOSCA) in the area of application management. He
>> co-founded Cloudsoft Corporation in 2009, where he currently serves as
>> CTO.
>> Aled Sage brings over a decade’s experience developing distributed
>> applications, mostly in the enterprise sector but with an extensive list
>> of open source project contributions. Particular areas of interest
>> include concurrency, fault tolerance and transactional semantics.
>> Andrew Kennedy is skilled in programming Java applications and services
>> using modern open-source technologies. He has experience analysing,
>> designing and implementing applications, middleware, messaging systems
>> and databases, and is also an accomplished penetration tester, ethical
>> hacker, network security engineer and security analyst. He is an ASF
>> committer with the Apache Qpid project.
>> Richard Downer has spent the last few years specialising in Cloud
>> Computing. He has worked extensively with the OpenStack and CloudStack
>> APIs, as well as Amazon EC2. Aside from Brooklyn, he has made numerous
>> contributions to the Apache jclouds Java cloud abstraction project
>> (
>> New core developers (initial committers) for the Brooklyn project will
>> be welcomed based on the current criteria of code contributions and
>> broad involvement in the community through code reviews, mailing lists,
>> and/or IRC.
>> === Alignment ===
>> We wish Brooklyn to follow the best practices of open source, and Apache
>> is a model of these practices. From the outset, Brooklyn has based
>> itself on the Apache model for governance including making commits and
>> voting on releases or substantive changes.
>> Brooklyn is a heavy user of many Apache projects, and Brooklyn
>> participants have been active in discussions and code contributions with
>> several of these communities. In particular, Apache jclouds is our most
>> important dependency. Apache CloudStack is a reliable and often-used
>> deployment target for Brooklyn. Apache Maven is used to build Brooklyn
>> and in some cases to resolve dependencies at runtime. In addition,
>> Brooklyn support has been built for deploying and managing many Apache
>> projects (Tomcat, Cassandra, Whirr, ActiveMQ, Kafka, and many others
>> below). These relationships are discussed further below.
>> == Known Risks ==
>> === Orphaned projects ===
>> The team behind Brooklyn firmly believe in their project and are
>> committed to its success. They will be pushing the project until, and
>> beyond, its community makes the project self-sustaining.
>> === Inexperience with Open Source ===
>> Brooklyn has been open source since April 2012. All development has
>> happened in public on GitHub[1], backed up by mailing lists on Google
>> Groups[2][3] and IRC[4]. The team of committers are committed to the
>> principles of open source, and count amongst their number existing
>> Apache committers.
>> For some of the extended group of contributors more used to commercial
>> development, the commitment to openness is a habit that still needs to
>> be learnt. For example, there is a feeling that some things need to be
>> discussed "internally" first. However this instinct is being un-learned
>> as our experience with the project in open source form continues. Under
>> the guidance of the PPMC, we do not expect this to be a risk.
>> [1]
>> [2]
>> [3]
>> [4] #brooklyncentral on Freenode
>> === Homogenous Developers ===
>> The initial committers are all affiliated with Cloudsoft Corporation Ltd
>> and the vast majority of the commits to the project to date have been by
>> Cloudsoft employees. Some of the active participants are from
>> organizations other than Cloudsoft and have expressed their support for
>> the move to Apache and interest in continuing to contribute.
>> We do have a geographically distributed list of contributors. Many are
>> in the United Kingdom, but there are a significant number of commits
>> from people from other European countries and from the United States.
>> As noted previously, our community is beginning to grow and we look
>> forward to bringing more diversity into the list of committers and PPMC
>> members. However we recognize the current lack of diversity as a known
>> risk.
>> === Reliance on Salaried Developers ===
>> The initial committers are all affiliated with Cloudsoft Corporation
>> Ltd. The vast majority of the commits to the project to date have been
>> by Cloudsoft employees in the line of their work.
>> Our community is beginning to grow, and we hope to grow the community
>> significantly and bring more diversity into the list of committers and
>> PPMC members. However we recognise the reliance on salaried developers
>> as a known risk.
>> === Relationships with Other Apache Products ===
>> Brooklyn has relationships to several other Apache projects:
>> Brooklyn has entities for many Apache projects; for example, it can
>> deploy instances of Cassandra, Tomcat, HTTP Server, ActiveMQ, Qpid,
>> Solr, Storm, Karaf, Kafka, Zookeeper and CouchDB
>> Brooklyn uses Whirr to deploy instances of Hadoop and HBase
>> jclouds is our single most important dependency, which we use for
>> virtually all our interaction with cloud provider APIs.
>> CloudStack is a first-class target for Brooklyn application deployments,
>> and we have used it on many occasions.
>> We use Maven, HttpComponents and various Commons projects.
>> There is some overlap with the Whirr project; at face value, both Whirr
>> and Brooklyn allow complex applications to be deployed into cloud
>> providers. However our emphasis is different -- Brooklyn provides
>> policy-based management to monitor applications. We consider Brooklyn to
>> be complementary rather than competitive - Brooklyn supports the use of
>> Whirr to deploy Hadoop clusters.
>> === A Excessive Fascination with the Apache Brand ===
>> We consider that Apache is a natural home for Brooklyn; we license our
>> code under the Apache License, and we make extensive use of jclouds
>> which recently graduated from the Apache incubator; we aspire to follow
>> open source best practices. Our proposal to the Apache Incubator is
>> based in pragmatism, not idolatry.
>> == Documentation ==
>> Brooklyn's primary website is available at
>> == Initial Source ==
>> There is an established existing code base for Brooklyn located at
>> GitHub:
>> == Source and Intellectual Property Submission Plan ==
>> The code is licensed under Apache License V2, and all contributions are
>> subject to an Apache-style ICLA with Cloudsoft Corporation Ltd. In
>> addition to the code, Cloudsoft has registered a trademark on the name
>> "Brooklyn", and the logo currently used for the project. The PPMC will
>> work with Cloudsoft Corporation and its stakeholders in order to
>> identify these properties and donate them to the Apache Foundation.
>> There are also a number of other assets related to the project, such as
>> its domain name, Twitter account, and IRC channel. During incubation the
>> PPMC will identify all these assets, and arrange the transfer,
>> replacement, or deprecation of these assets as appropriate.
>> == External Dependencies ==
>> The vast majority of Brooklyn's direct and indirect dependencies are
>> licensed (or dual-licensed) under The Apache License, Category A or B
>> licenses, or are dedicated to the public domain.
>> A small number dependencies give cause for concern; there are two cases
>> of LGPL dependencies, and a number of dependencies where the license is
>> either unknown, or compatibility with Apache is unknown.
>> The PPMC must therefore:
>>   * determine the license for all code where it is not currently known;
>>   * determine ASL compatibility where this is not currently known;
>>   * replace all dependencies that do not have an ASL compatible license.
>> == Cryptography ==
>> Brooklyn does not directly implement cryptography code. It may rely on
>> other projects to perform certain cryptographic tasks (e.g. calculating
>> signatures for requests to cloud providers.)
>> == Required Resources ==
>> === Mailing lists ===
>> We seek "dev" and "user" lists to replace the lists currently in use. In
>> alignment with Apache's standard practices, we would also have a
>> "private" list for the PMC members, and a "commits" list.
>> === Source Control ===
>> We would require a Git repository named "brooklyn" to hold the Brooklyn
>> source code, and "brooklyn-site" to hold the web site source code. We
>> would like these to be mirrored to GitHub repositories, where we intend
>> to follow the same model currently used by Apache jclouds.
>> === Issue Tracking ===
>> Jira, with a project name of "BROOKLYN".
>> === Other Resources ===
>> No other resources are required at this time.
>> == Initial Committers ==
>>   * Alex Heneveld
>>   * Aled Sage
>>   * Andrew Kennedy
>>   * Richard Downer
>> == Affiliations ==
>> The majority of the commits to the project so far have been made by
>> people who were employees of Cloudsoft Corporation Ltd at the time of
>> the contribution, and the vast majority of those commits were in the
>> line of their employment. All named initial committers are employees of
>> Cloudsoft.
>> As stated in the Known Risks section, we appreciate that this cannot
>> continue long term, and a key goal of the podling will be to encourage
>> people not affiliated with Cloudsoft to join the project as committers
>> and PMC members.
>> == Sponsors ==
>> Champion
>>   * Chip Childers
>> Nominated Mentors
>>   * Matt Hogstrom
>>   * Alex Karasulu
>> We appreciate that 3-5 Mentors greatly assists the incubation process,
>> so welcome more Mentors.
>> Sponsoring Entity
>> We are asking that the Incubator sponsors the Brooklyn podling.
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message