incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashvin A <aas...@gmail.com>
Subject Re: [PROPOSAL] Heron
Date Thu, 15 Jun 2017 20:46:57 GMT
Thanks William!
It will be great to work with you again.

On Thu, Jun 15, 2017 at 10:12 AM, Supun Kamburugamuve <supun06@gmail.com>
wrote:

> Thank you, William, for offering to help with the incubation process. It
> will be really helpful.
>
> Supun..
>
> On Wed, Jun 14, 2017 at 11:04 PM, William Markito Oliveira <
> william.markito@gmail.com> wrote:
>
> > Howdy!
> >
> > If Heron is looking for some help around incubation process, I'd love to
> > help while Geode experience is still fresh in my mind and given that
> it's a
> > project/space that I do have interest. Since I'm not an ASF member, I
> don't
> > think I can offer to be a mentor, but can probably still help and
> > participate on the process.
> >
> > Thanks!
> >
> > On Wed, Jun 14, 2017 at 7:54 PM, P. Taylor Goetz <ptgoetz@gmail.com>
> > wrote:
> >
> > > Hi Bill/Supun,
> > >
> > > Sorry for not being a little more clear. I was asking more about how
> the
> > > Heron community would seek to engage with Storm community at the
> > > *community* level as opposed to the technical level (i.e. “Community
> over
> > > Code”).
> > >
> > > I’ve been asked by many why this has never happened, and have always
> > > struggled to answer. Maybe you could help answer that question as well
> as
> > > if and how that might change if Heron were to incubate.
> > >
> > > Another quick question: The proposal mentions Heron being used in
> > > production at Google, but some Google employees I recently spoke to
> > seemed
> > > to contradict that. Could you explain? Note that’s nothing that would
> > > preclude the project from incubating, I’m just curious.
> > >
> > > -Taylor
> > >
> > > > On Jun 14, 2017, at 7:35 AM, Supun Kamburugamuve <supun06@gmail.com>
> > > wrote:
> > > >
> > > > Hi Taylor,
> > > >
> > > > For me, one of the interesting differences between Heron and Storm is
> > the
> > > > execution model. Storm uses a shared memory model while Heron uses a
> > > > process based model. It will be interesting to see how these two
> > evolve.
> > > >
> > > > Thanks,
> > > > Supun..
> > > >
> > > > On Mon, Jun 12, 2017 at 4:15 PM, Bill Graham <billgraham@gmail.com>
> > > wrote:
> > > >
> > > >> Hi Taylor,
> > > >>
> > > >> Thanks for the mentor offer, we'd be glad to have your help.
> > > >>
> > > >> I think the best place for collaboration would be around the
> evolution
> > > of
> > > >> the API. In addition we plan to look more into DSL solutions which
> we
> > > could
> > > >> potentially collaborate on. This could be Trident, or Beam or
> > something
> > > >> else, but there could be synergies for future development here.
> > > >>
> > > >> thanks,
> > > >> Bill
> > > >>
> > > >> On Fri, Jun 9, 2017 at 8:53 PM, P. Taylor Goetz <ptgoetz@gmail.com>
> > > wrote:
> > > >>
> > > >>> Hi Bill,
> > > >>>
> > > >>> Could you comment on how/if the Heron community would be willing
to
> > > work
> > > >>> with the Storm community? I've seen a number of new features in
> Storm
> > > >> being
> > > >>> ported to Heron, but I have yet to see any attempt by the Heron
> > > community
> > > >>> to engage with the Apache Storm community.
> > > >>>
> > > >>> I don't think it would be too far off to say that the relationship
> > > >> between
> > > >>> Heron and Apache Storm has been somewhat adversarial. The pre-
and
> > > >>> post-open sourcing marketing around Heron seemed, at least to
me,
> > > >> somewhat
> > > >>> aggressively negative toward Storm.
> > > >>>
> > > >>> As a peer to Apache Storm, how would the proposed "Apache Heron"
> > > >> community
> > > >>> work to collaborate with the Storm community? If Heron is adopting
> > API
> > > >>> changes in Storm, then it seems there is an opportunity for
> > > >> collaboration.
> > > >>>
> > > >>> Don't take any of this as an objection to incubating the project.
I
> > > would
> > > >>> support it. I would also be willing to be a mentor, if you would
> > > consider
> > > >>> taking on another.
> > > >>>
> > > >>> -Taylor
> > > >>>
> > > >>>> On Jun 8, 2017, at 1:23 PM, Bill Graham <billgraham@gmail.com>
> > wrote:
> > > >>>>
> > > >>>> Dear Apache Incubator Community,
> > > >>>>
> > > >>>> We are excited to share our proposal for discussion and feedback
> > > >>>> for entering Apache Incubation. Heron is a real-time, distributed,
> > > >>>> fault-tolerant stream processing engine.
> > > >>>>
> > > >>>> Our proposal can be found at https://wiki.apache.org/
> > > >>> incubator/HeronProposal
> > > >>>> and is included below.
> > > >>>>
> > > >>>>
> > > >>>> Thank you,
> > > >>>>
> > > >>>> Bill Graham on behalf of the Heron developers
> > > >>>>
> > > >>>>
> > > >>>> # Heron Proposal
> > > >>>>
> > > >>>> ## Abstract
> > > >>>> Heron is a real-time, distributed, fault-tolerant stream
> processing
> > > >>> engine
> > > >>>> initially developed by Twitter.
> > > >>>>
> > > >>>> ## Proposal
> > > >>>>
> > > >>>> Heron is a real-time stream processing engine built for high
> > > >> performance,
> > > >>>> ease of manageability, performance predictability and developer
> > > >>>> productivity[1]. We wish to develop a community around Heron
to
> > > >> increase
> > > >>>> contributions and see Heron thrive in an open forum.
> > > >>>>
> > > >>>> ## Background
> > > >>>>
> > > >>>> Heron provides the ability for developers to compose directed
> > acyclic
> > > >>>> graphs (DAGs) of real-time query execution logic (i.e. a topology)
> > and
> > > >>>> submit the topology to execute on a pluggable job scheduling
> system
> > > >>> (e.g.,
> > > >>>> Apache Aurora, YARN, Marathon, etc). Users can employ either
the
> > > native
> > > >>>> Heron API or the Apache Storm API to develop the topology.
Heron
> > > >> supports
> > > >>>> the Storm API for ease of migration, but beyond that Heron’s
> > > >> architecture
> > > >>>> differs considerably from Storm’s.
> > > >>>>
> > > >>>> Users submit a topology to the scheduler using the Heron client,
> > which
> > > >>> uses
> > > >>>> the Heron binary libraries to deploy all daemons required
to run
> and
> > > >>> manage
> > > >>>> the topology. The topology therefore has no reliance on centrally
> > > >> managed
> > > >>>> Heron services, only on a generic job scheduling system, which
> lends
> > > >>> itself
> > > >>>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN
> > > >> (among
> > > >>>> others).
> > > >>>>
> > > >>>> The scheduler runs each topology as a job consisting of multiple
> > > >>>> containers. One of the containers runs the topology master,
> > > responsible
> > > >>> for
> > > >>>> managing the topology. The remaining containers each runs
a stream
> > > >>> manager
> > > >>>> responsible for data routing, a metrics manager that collects
and
> > > >> reports
> > > >>>> various metrics and a number of processes called Heron instances
> > which
> > > >>> run
> > > >>>> the user-defined logic on the stream of tuples. Parallelism
is
> > > achieved
> > > >>> via
> > > >>>> process-based isolation of Heron instances, which provides
> > predictable
> > > >>>> performance while simplifying debugging. The containers are
> > allocated
> > > >> and
> > > >>>> managed by the scheduler framework based on resource availability
> of
> > > >>> nodes
> > > >>>> in the cluster. The metadata for the topology, such as the
> physical
> > > >> plan
> > > >>>> and execution details, are stored in the pluggable Heron State
> > Manager
> > > >>>> (e.g. Apache ZooKeeper).
> > > >>>>
> > > >>>> ## Rationale
> > > >>>>
> > > >>>> Heron is a general-purpose, modular and extensible platform
that
> can
> > > be
> > > >>>> leveraged to support common, real-time analytics use cases.
There
> is
> > > an
> > > >>>> increasing demand for open-source, scalable real-time analytics
> > > >> systems.
> > > >>> We
> > > >>>> believe that Heron can be leveraged by other organizations
to
> build
> > > >>>> streaming applications that can benefit from its robustness,
high
> > > >>>> performance, adaptability to cloud environments and ease of
use.
> > > >>> Moreover,
> > > >>>> we hope that open-sourcing Heron will help to further evolve
the
> > > >>> technology
> > > >>>> as the project attracts contributors with diverse backgrounds
and
> > > areas
> > > >>> of
> > > >>>> expertise.
> > > >>>>
> > > >>>> We believe the Apache foundation is a great fit as the long-term
> > home
> > > >> for
> > > >>>> Heron, as it provides an established process for community-driven
> > > >>>> development and decision making by consensus. This is exactly
the
> > > model
> > > >>> we
> > > >>>> want for future Heron development.
> > > >>>>
> > > >>>> ## Initial Goals
> > > >>>>
> > > >>>> * Move the existing codebase, website, documentation, and
mailing
> > > lists
> > > >>> to
> > > >>>> Apache-hosted infrastructure.
> > > >>>> * Integrate with the Apache development process.
> > > >>>> * Ensure all dependencies are compliant with Apache License
> version
> > > >> 2.0.
> > > >>>> * Incrementally develop and release per Apache guidelines.
> > > >>>>
> > > >>>> ## Current Status
> > > >>>>
> > > >>>> Heron is a stable project used in production at Twitter since
2014
> > and
> > > >>> open
> > > >>>> sourced under the ASL v2 license in 2016. The Heron source
code is
> > > >>>> currently hosted at github.com (https://github.com/twitter/heron
> ),
> > > >> which
> > > >>>> will seed the Apache git repository.
> > > >>>>
> > > >>>> ### Meritocracy
> > > >>>>
> > > >>>> By submitting this incubator proposal, we’re expressing
our intent
> > to
> > > >>> build
> > > >>>> a diverse developer community around Heron that will conduct
> itself
> > > >>>> according to The Apache Way and use a meritocratic means of
> building
> > > >> it's
> > > >>>> committer base. Several companies and universities have already
> > > >> expressed
> > > >>>> interest in and contributed to Heron. Our goal is to grow
the
> Heron
> > > >>>> community by encouraging open communication, contribution
and
> > > >>> participation
> > > >>>> of all types, and ensuring that contributors are recognized
> > > >>> appropriately.
> > > >>>>
> > > >>>> ### Community
> > > >>>>
> > > >>>> Heron is currently being used by Twitter, Google, Machine
Zone and
> > > >>>> ndustrial.io and has received significant contributions by
> > Microsoft
> > > >> and
> > > >>>> Streamlio. By bringing Heron into the Apache ecosystem, we
believe
> > we
> > > >> can
> > > >>>> attract even more developers who are interested in creating
> > real-time
> > > >>>> systems to build the project's contributor base.
> > > >>>>
> > > >>>> ### Core Developers
> > > >>>>
> > > >>>> Current core developers are engineers from Twitter, Google,
> > Microsoft
> > > >> and
> > > >>>> Streamlio.
> > > >>>>
> > > >>>> ### Alignment
> > > >>>>
> > > >>>> Heron utilizes a number of Apache technologies. Heron leverages
> > Apache
> > > >>>> ZooKeeper for coordination and has scheduler implementations
to
> > > >> integrate
> > > >>>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN
(via
> > Apache
> > > >>> REEF)
> > > >>>> as well as spout implementations to integrate with Apache
Kafka
> and
> > > >>> metrics
> > > >>>> implementations to integrate with Scribe. Heron also implements
> the
> > > >>> Apache
> > > >>>> Storm user-level API, which allows topologies written against
> Storm
> > to
> > > >>> run
> > > >>>> in Heron. We believe that having Heron at Apache will help
further
> > the
> > > >>>> growth of the streaming compute community, as well as encourage
> > > >>> cooperation
> > > >>>> and developer cross pollination with other Apache projects.
> > > >>>>
> > > >>>> ## Known Risks
> > > >>>>
> > > >>>> ### Orphaned Products
> > > >>>>
> > > >>>> The risk of the Heron project being abandoned is minimal.
It is
> used
> > > in
> > > >>>> production at Twitter and Google and other companies are
> evaluating
> > or
> > > >>>> adopting it for production use.
> > > >>>>
> > > >>>> ### Inexperience with Open Source
> > > >>>>
> > > >>>> All of the core contributors to the project have considerable
> > > >> experience
> > > >>>> with open source software development. Bill Graham[2], Ashvin
> > > >> Agrawal[3]
> > > >>>> and Supun Kamburugamuve[4], committers on the project, are
PMCs on
> > > >> other
> > > >>>> Apache projects and Bill and Ashvin have gone through the
Apache
> > > >>> incubator
> > > >>>> process. Twitter has already donated numerous projects to
the ASF
> > > >> (e.g.,
> > > >>>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan
to be
> > > >> mentored
> > > >>>> by experienced ASF members that can help with any roadblocks.
> > > >>>>
> > > >>>> ### Homogenous Developers
> > > >>>>
> > > >>>> Initial committers come from 5 separate organizations. Our
> intention
> > > is
> > > >>>> increase the diversity of contributing developers and their
> > > >> affiliations.
> > > >>>> To date github contributions have come from approximately
50
> > > >> contributors
> > > >>>> from outside the Twitter team.
> > > >>>>
> > > >>>> ### Reliance on Salaried Developers
> > > >>>>
> > > >>>> It is expected that Heron development will occur on both salaried
> > time
> > > >>> and
> > > >>>> on volunteer time. The majority of initial committers are
paid by
> > > their
> > > >>>> employers to contribute to this project. We are committed
to
> > > recruiting
> > > >>>> additional committers from other organizations as well as
> > non-salaried
> > > >>>> committers to join project.
> > > >>>>
> > > >>>> ### Relationships with Other Apache Products
> > > >>>>
> > > >>>> As mentioned in the Alignment section, Heron implements the
Apache
> > > >> Storm
> > > >>>> API and integrates with multiple Apache schedulers (Apache
Mesos,
> > > >> Apache
> > > >>>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper
and
> > > Apache
> > > >>>> Thrift.
> > > >>>>
> > > >>>> ### An Excessive Fascination with the Apache Brand
> > > >>>>
> > > >>>> Heron's popularity is growing in the streaming compute space
and
> we
> > > are
> > > >>>> long time supporters of the Apache brand. This proposal is
not for
> > the
> > > >>>> purpose of generating publicity through. Rather, the primary
> > benefits
> > > >> to
> > > >>>> joining Apache are those of community building and open decision
> > > making
> > > >>>> outlined in the Rationale section.
> > > >>>>
> > > >>>> ## Documentation
> > > >>>>
> > > >>>> This proposal exists online as http://wiki.apache.org/
> > > >>>> incubator/HeronProposal. Extensive documentation can be found
on
> > > github
> > > >>> at
> > > >>>> https://twitter.github.io/heron and the source code is well
> > > >> documented.
> > > >>>>
> > > >>>> ## Source and Intellectual Property Submission Plan
> > > >>>>
> > > >>>> The Heron codebase is currently hosted on Github:
> > > >>>> https://github.com/twitter/heron. During incubation, the codebase
> > > will
> > > >>> be
> > > >>>> migrated to Apache infrastructure. The source code is already
ASF
> > 2.0
> > > >>>> licensed.
> > > >>>>
> > > >>>> ## External Dependencies
> > > >>>>
> > > >>>> All external libraries have ASF 2.0 compatible licenses except
for
> > > >>> pylint.
> > > >>>> The pylint library is GPL licensed, but is only used for pre-build
> > > >> Python
> > > >>>> style checks and is neither bundled with, nor relied upon
by, the
> > > Heron
> > > >>>> source or binary release artifacts.
> > > >>>>
> > > >>>> ## Cryptography
> > > >>>>
> > > >>>> Heron does not use any cryptography libraries.
> > > >>>>
> > > >>>> ## Required Resources
> > > >>>>
> > > >>>> ### Mailing lists
> > > >>>>
> > > >>>> private@heron.incubator.apache.org (with moderated subscriptions)
> > > >>>> dev@heron.incubator.apache.org
> > > >>>> commits@heron.incubator.apache.org
> > > >>>> user@heron.incubator.apache.org
> > > >>>>
> > > >>>> ## Subversion Directory
> > > >>>>
> > > >>>> Git is the preferred source control system: git://
> > > git.apache.org/heron
> > > >>>>
> > > >>>> ## Issue Tracking
> > > >>>>
> > > >>>> JIRA: Heron (HERON)
> > > >>>>
> > > >>>> ## Initial Committers
> > > >>>>
> > > >>>> * Andrew Jorgensen (andrew at andrewjorgensen dot com)
> > > >>>> * Ashvin Agrawal (ashvin at apache dot org)*
> > > >>>> * Avrilia Floratou (avrilia dot floratou at gmail dot com)
> > > >>>> * Bill Graham (billgraham at apache dot org)*
> > > >>>> * Brian Hatfield (bmhatfield at gmail dot com)
> > > >>>> * Chris Kellogg (cckellogg at gmail dot com)
> > > >>>> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com)
> > > >>>> * Karthik Ramasamy (karthik at gmail dot com)
> > > >>>> * Maosong Fu (maosongfu at gmail dot com)
> > > >>>> * Neng Lu(freeneng at gmail dot com)
> > > >>>> * Runhang Li (obj dot runhang at gmail dot com)
> > > >>>> * Sanjeev Kulkarni (sanjeevrk at gmail dot com)
> > > >>>> * Supun Kamburugamuve (supun at apache dot org)*
> > > >>>> * Thomas Sun (tom dot ssf at gmail dot com)
> > > >>>> * Yaliang Wang (yaliang dot w dot wang at ieee dot org)
> > > >>>>
> > > >>>> ## Affiliations
> > > >>>>
> > > >>>> * Andrew Jorgensen (Google)
> > > >>>> * Ashvin Agrawal (Microsoft)
> > > >>>> * Avrilia Floratou (Microsoft)
> > > >>>> * Bill Graham (Twitter)
> > > >>>> * Brian Hatfield (Google)
> > > >>>> * Chris Kellogg (Twitter)
> > > >>>> * Huijun Wu (Twitter)
> > > >>>> * Karthik Ramasamy (Streamlio)
> > > >>>> * Maosong Fu (Twitter)
> > > >>>> * Neng Lu (Twitter)
> > > >>>> * Runhang Li (Twitter)
> > > >>>> * Sanjeev Kulkarni (Streamlio)
> > > >>>> * Supun Kamburugamuve (Indiana University)
> > > >>>> * Thomas Sun (Twitter)
> > > >>>> * Yaliang Wang (Twitter)
> > > >>>>
> > > >>>> ## Sponsors
> > > >>>>
> > > >>>> ### Champion
> > > >>>>
> > > >>>> * Julien Le Dem (julien at apache dot org)
> > > >>>>
> > > >>>> ### Nominated Mentors
> > > >>>>
> > > >>>> * Jake Farrell (jfarrell at apache dot org)
> > > >>>> * Jacques Nadeau (jacques at apache dot org)
> > > >>>> * Julien Le Dem (julien at apache dot org)
> > > >>>>
> > > >>>> ### Sponsoring Entity
> > > >>>>
> > > >>>> The Apache Incubator
> > > >>>>
> > > >>>> ### Footnotes
> > > >>>>
> > > >>>> 1 - Papers detailing Heron are available at
> > > http://dl.acm.org/citation
> > > >> .
> > > >>>> cfm?id=2742788 and http://sites.computer.org/
> debull/A15dec/p15.pdf.
> > > >>>> 2 - http://home.apache.org/phonebook.html?uid=billgraham
> > > >>>> 3 - http://home.apache.org/phonebook.html?uid=ashvin
> > > >>>> 4 - http://home.apache.org/phonebook.html?uid=supun
> > > >>>
> > > >>> ------------------------------------------------------------
> > ---------
> > > >>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > >>> For additional commands, e-mail: general-help@incubator.apache.org
> > > >>>
> > > >>>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Supun Kamburugamuve
> > > > Member, Apache Software Foundation; http://www.apache.org
> > > > E-mail: supun@apache.o <supun06@gmail.com>rg;  Mobile: +1 812 219
> 2563
> > > > <(812)%20219-2563>
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
> >
> > --
> > ~/William
> >
>
>
>
> --
> Supun Kamburugamuve
> Member, Apache Software Foundation; http://www.apache.org
> E-mail: supun@apache.o <supun06@gmail.com>rg;  Mobile: +1 812 219 2563
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message