incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kenneth Knowles <k...@apache.org>
Subject Re: [VOTE] Accept StreamPipes into the Apache Incubator
Date Fri, 08 Nov 2019 18:55:43 GMT
+1 (binding)

Kenn

On Fri, Nov 8, 2019 at 9:14 AM 俊平堵 <junping_du@apache.org> wrote:

> +1. Good luck!
>
> Thanks,
>
> Junping
>
> Dominik Riemer <riemer@fzi.de>于2019年11月8日 周五上午3:00写道:
>
> > Hi all,
> >
> > following up the [DISCUSS] thread on StreamPipes (
> >
> https://lists.apache.org/thread.html/1cf79ef65888f695b4b925fd67ef8a2b845f6b0931c251a0ff1115e1@%3Cgeneral.incubator.apache.org%3E
> ),
> > I would like to call a VOTE to accept StreamPipes into the Apache
> Incubator.
> >
> > Please cast your vote:
> >
> >   [ ] +1, bring StreamPipes into the Incubator
> >   [ ] +0, I don't care either way
> >   [ ] -1, do not bring StreamPipes into the Incubator, because...
> >
> > The vote will open at least for 72 hours and only votes from the
> Incubator
> > PMC are binding, but votes from everyone are welcome.
> >
> > Dominik
> >
> > ----
> > StreamPipes Proposal (
> >
> https://cwiki.apache.org/confluence/display/INCUBATOR/StreamPipesProposal)
> >
> > == Abstract ==
> > StreamPipes is a self-service (Industrial) IoT toolbox to enable
> > non-technical users to connect, analyze and explore (Industrial) IoT data
> > streams.
> >
> > = Proposal =
> >
> > The goal of StreamPipes (www.streampipes.org) is to provide an
> > easy-to-use toolbox for non-technical users, e.g., domain experts, to
> > exploit data streams coming from (Industrial) IoT devices. Such users are
> > provided with an intuitive graphical user interface with the Pipeline
> > Editor at its core. Users are able to graphically model processing
> > pipelines based on data sources (streams), data processors and data
> sinks.
> > Data processors and sinks are self-contained microservices, which
> implement
> > either stateful or stateless processing logic (e.g., a trend detection or
> > image classifier). Their processing logic is implemented using one of
> > several provided wrappers (we currently have wrappers for
> > standalone/Edge-based processing, Apache Flink, Siddhi and working
> wrapper
> > prototypes for Apache Kafka Streams and Spark, in the future we also plan
> > to integrate with Apache Beam). An SDK allows to easily create new
> pipeline
> > elements. Pipeline elements can be installed at runtime. To support users
> > in creating pipelines, an underlying semantics-based data model enables
> > pipeline elements to express requirements on incoming data streams that
> > need to be fulfilled, thus reducing modeling errors.
> > Data streams are integrated by using StreamPipes Connect, which allows to
> > connect data sources (based on standard protocols, such as MQTT, Kafka,
> > Pulsar, OPC-UA and further PLC4X-supported protocols) without further
> > programming using a graphical wizard. Additional user-faced modules of
> > StreamPipes are a Live dashboard to quickly explore IoT data streams and
> a
> > wizard that generates code templates for new pipeline elements, a
> Pipeline
> > Element Installer used to extend the algorithm feature set at runtime.
> >
> > === Background ===
> > StreamPipes was started in 2014 by researchers from FZI Research Center
> > for Information Technology in Karlsruhe, Germany. The original prototype
> > was funded by an EU project centered around predictive analytics for the
> > manufacturing domain. Since then, StreamPipes was constantly improved and
> > extended by public funding mainly from federal German ministries. In
> early
> > 2018, the source code was officially released under the Apache License
> 2.0.
> > At the same time, while we focused on bringing the research prototype to
> a
> > production-grade tool, the first companies started to use StreamPipes.
> > Currently, the primary goal is to widen the user and developer base. At
> > ApacheCon NA 2019, after having talked to many people from the Apache
> > Community, we finally decided that we would like to bring StreamPipes to
> > the Apache Incubator.
> >
> > === Rationale ===
> > The (Industrial) IoT domain is a highly relevant and emerging sector.
> > Currently, IoT platforms are offered by many vendors ranging from SMEs up
> > to large enterprises. We believe that open source alternatives are an
> > important cornerstone for manufacturing companies to easily adopt
> > data-driven decision making. From our point of view, StreamPipes fits
> very
> > well into the existing (I)IoT ecosystem within the ASF, with projects
> such
> > as Apache PLC4X focusing on connecting machine data from PLCs, or other
> > tools we are also using either in the core of StreamPipes or with
> > integrations (Apache Kafka, Apache IoTDB, Apache Pulsar). StreamPipes
> > itself focuses on enabling self-service IoT data analytics for
> > non-technical users.
> > The whole StreamPipes code is currently on Github. To get a rough
> estimate
> > of the project size:
> > * streampipes: Backend and core modules, ~3300 commits
> > * streampipes-ui: User Interface, ~1300 commits
> > * streampipes-pipeline-elements: ~100 Pipeline Elements (data
> > processors/algorithms and sinks), ~500 Commits
> > * streampipes-connect-adapters: ~20 Adapters to connect data, ~100
> commits
> > To achieve our goal to further extend the code base with new features,
> new
> > connectors and new algorithms and to grow both the user and developer
> > community, we believe that a community-driven development process is the
> > best way to further develop StreamPipes. Finally, after having talked to
> > committers from various Apache IoT-related projects and participation in
> > spontaneous hacking sessions and being impressed by the collaboration
> among
> > individual projects, we decided that (from our point of view) the ASF is
> > the ideal place to be the future home of StreamPipes.
> >
> > === Initial Goals ===
> > * Move the existing codebase to Apache
> > * Fully align with Apache development- and release processes
> > * Perform name search and do a thorough review of existing licenses
> > * First Apache release
> >
> > === Current Status ===
> > ** Meritocracy **
> > We are absolutely committed to strengthen StreamPipes as a real
> > community-driven open source project. The existing committer base is
> highly
> > motivated to foster the open source way in the industrial IoT sector and,
> > together with existing Apache communities focused on this domain, provide
> > open source tooling for Industrial IoT projects in the same way Apache
> > offers in the Big Data space, for instance.
> > The development philosophy behind StreamPipes has always followed the
> > principles of meritocracy - although most committers are still active in
> > the project, we managed to onboard new, committed developers regularly. 2
> > people, who are today core of the developer team, have joined during the
> > past year. Therefore, we would aim to continuously expand the PMC and
> > committer base based on merit.
> >
> > ** Community **
> > Since being open-sourced in 2018, the public interest in StreamPipes has
> > steadily grown. Several companies, mainly from the manufacturing domain,
> > have tested StreamPipes in form of proof-of-concept projects. First
> > companies have started to use StreamPipes in production. This was due to
> a
> > high number of events from meetups, research conferences, demo sessions
> up
> > to hackathons we participated or organized during the past two years.
> After
> > having generated a general interest in StreamPipes, our next focus will
> be
> > to find more committers to diversify the contributor base.
> >
> > ** Core Developers **
> > The core developers of the system are Dominik Riemer, Philipp Zehnder,
> > Patrick Wiener and Johannes Tex. All core developers are initial
> committers
> > in the current proposal. Some former students who recently started to
> work
> > at companies and who have also worked on the project with great
> commitment,
> > will be asked to further contribute to the project.
> >
> > ** Alignment **
> > StreamPipes has dependencies to a lot of existing Apache projects - this
> > is one reason why we think that the ASF is the best future home for
> > StreamPipes. The messaging layer is based on Apache Kafka (and also
> Apache
> > Pulsar as a future option), and runtime wrappers exist for Apache Flink,
> > Apache Spark and Apache Kafka Streams. StreamPipes Connect already
> includes
> > adapters for several Apache projects. Most importantly, we integrate (and
> > plan to deepen the integration) with IIoT-focused projects such as Apache
> > PLC4X. Also, several data sinks exist to send messages to tools from
> other
> > Apache projects (e.g., Apache Kafka, Apache Pulsar, and Apache IoTDB).
> > Together with these tools (and also after having talked to the core
> > developers after this year's ApacheCon) we are absolutely convinced that
> a
> > tight integration between these tools will strengthen the open source IoT
> > ecosystem.
> >
> > === Known Risks ===
> > ** Orphaned Products **
> > We don't expect the risk of an orphaned product. The initial committers
> > have worked on the project for years and are absolutely committed to
> making
> > this open source tool a great success. All initial committers are
> committed
> > to work on StreamPipes in their free time.
> >
> > ** Inexperience with Open Source **
> > All initial committers have years of expertise related to open source
> > development and understand what open source means. However, none of the
> > initial committers are currently committers to Apache projects, although
> > some have already contributed to some projects. From a variety of events
> > and from intensively studying Apache mailing lists, we are sure that the
> > Apache Way is the way we'd like the project to move into the future. We
> > expect to benefit from the experiences from the ASF in building
> successful
> > open source projects.
> >
> > ** Length of Incubation **
> > We are aware that incubation is a process that is focused on building the
> > community, learning the Apache Way and other important things such as
> > learning the release process and handling licensing and trademark issues.
> > We are also aware that, although there is a steadily increasing interest
> in
> > StreamPipes, a major challenge we would need (and are willing) to work on
> > during the incubation phase is widening the committer base. We look
> forward
> > to that as a large developer base is exactly what we are striving for.
> >
> > ** Homogeneous Developers **
> > Most current developers work for the same institution (FZI). The
> > motivation of all developers goes beyond their commitment to work and all
> > current committers work on StreamPipes in their free time. Recently, we
> > have received first pull requests from external contributors and a
> growing
> > interest from users and companies outside of FZI. First manufacturing
> > companies have already evaluated and adopted StreamPipes. To attract
> > external developers, we've created an extensive documentation, have a
> Slack
> > channel to quickly answer questions, and provide help via mail.
> Therefore,
> > we believe that making the developer community more heterogeneous is not
> > only mandatory, but something that can be achieved during the next
> months.
> >
> > ** Reliance on salaried developers **
> > Currently, StreamPipes receives support from salaried developers, mainly
> > research scientists from FZI. However, all core developers substantially
> > work on StreamPipes in their spare time. As this has been the case from
> the
> > beginning in early 2014, it can be expected that a substantial portion of
> > volunteers will continue to be working on the project and we aim at
> > strengthening the base of non-paid committers and paid committers of
> other
> > companies. At the same time, funding of the initial StreamPipes team is
> > secured by public funding for the next few years, making sure that there
> > will be also enough commitment from developers during their work time.
> >
> > ** Relationships with other Apache products ** StreamPipes is often
> > compared to tools such as Node-Red and Apache Nifi. This is mainly based
> on
> > a similar UI concept (dataflow approach). Despite some technological
> > differences (e.g., the microservice analytics approach vs. single-host
> > runtime of Node-Red, the wrapper architecture and the underlying
> > semantics-based model), we believe the target audience differs. We aim to
> > collaborate with the Apache Nifi community in terms of exchanging best
> > practices and also integrating both projects (e.g., by building
> connectors).
> > As mentioned above, quite a few adapters and data sinks are already
> > available that link to existing Apache projects.
> >
> > ** An excessive fascination with the Apache Brand ** Although we
> recognize
> > the Apache brand as the most visible brand in the open source domain, the
> > primary goal of this proposal is not to create publicity, but to widen
> the
> > developer base. We believe that successful projects have broad and
> diverse
> > communities. We expect that an Apache project, with a clear and proven
> way
> > to develop open source software, helps in finding new committers. As the
> > core development team has already worked on StreamPipes for the past few
> > years and is fully committed to the software and its benefit for the
> > industrial IoT domain, we would also continue development without being
> an
> > Apache project.
> >
> > === Documentation ===
> > Currently, we host a website at https://www.streampipes.org More
> > technical info (user + developer guide) can be found in the
> documentation:
> > https://docs.streampipes.org, where users can find tutorials and manuals
> > on how to extend StreamPipes using the SDK.
> >
> > === Initial Source ===
> > Currently, the following Github repositories exist, all licensed under
> the
> > Apache Software License 2.0:
> > * streampipes (https://www.github.com/streampipes/streampipes, the
> > backend & pipeline management module)
> > * streampipes-ui (https://www.github.com/streampipes/streampipes-ui, the
> > UI module)
> > * streampipes-pipeline-elements (
> > https://www.github.com/streampipes/streampipes-pipeline-elements,
> library
> > of data processors and sinks)
> > * streampipes-connect-adapters (
> > https://www.github.com/streampipes/streampipes-connect-adapters,
> > StreamPipes connect adapters)
> > * streampipes-docs (https://www.github.com/streampipes/streampipes-docs,
> > the abovementioned documentation)
> >
> > === Source and intellectual property submission plan === All initial
> > committers will sign a ICLA with the ASF. FZI, as the organizational body
> > that has employed the main contributors of StreamPipes, will sign a CCLA
> > and donate the codebase to the ASF (both subject to formal approval). All
> > major contributors are still active in the project.
> >
> > === External Dependencies ===
> > We did an initial review of all dependencies used in the various
> projects.
> > No critical libraries that depend on category X licenses were found, some
> > minor issues have already been resolved (e.g., removing dependencies to
> > org.json libraries). Most external dependencies used by the Java-based
> > (backend, pipeline-elements and connect) modules are licensed under the
> > Apache License 2.0, whereas some licenses are Cat B (e.g., CDDL). Most
> > external dependencies the UI requires on are licensed under the MIT
> > license.
> > Once we are moving to the Incubator, we would do a complete check of all
> > transitive dependencies. We don't expect any surprises here.
> >
> > === Cryptography ===
> > (not applicable)
> >
> > === Required Resources ===
> >
> > ** Mailing Lists **
> > We plan to use the following mailing lists:
> > * users@streampipes.incubator.apache.org
> > * dev@streampipes.incubator.apache.org
> > * private@streampipes.incubator.apache.org
> > * commits@streampipes.incubator.apache.org
> > As StreamPipes is targeted to a non-technical audience, we see a
> dedicated
> > user mailing list as an important requirement to help users.
> >
> > ** Subversion directory **
> > (not applicable)
> >
> > ** Git repositories **
> > We would like to use Git for source code management and enable Github
> > mirroring functionality.
> >
> > As we plan to merge some of the repos described above to simplify the
> > release process we ask to create the following source repositories:
> > * streampipes (containing backend + UI)
> > * streampipes-extensions (containing modules that can be dynamically
> > installed at runtime: pipeline elements and connect adapters)
> > * streampipes-website (containing docs + website)
> >
> > ** Issue tracking **
> > JIRA ID: StreamPipes
> >
> > === Initial Committers ===
> > List of initial committers in alphabetical order:
> > * Christofer Dutz (christofer.dutz at c-ware dot de)
> > * Dominik Riemer (dominik dot riemer at gmail dot com)
> > * Johannes Tex (tex at fzi dot de)
> > * Patrick Wiener (wiener at fzi dot de)
> > * Philipp Zehnder (zehnder at fzi dot de)
> >
> > === Sponsors ===
> >
> > ** Champion **
> > * Christofer Dutz (christofer.dutz at c-ware dot de)
> >
> > ** Mentors **
> > * Christofer Dutz (christofer.dutz at c-ware dot de)
> > * Julian Feinauer (Jfeinauer at apache dot org)
> > * Kenneth Knowles (kenn at apache dot org)
> > * Justin Mclean (justin at classsoftware dot com)
> > * Jean-Baptiste Onofré (jb at nanthrax dot net)
> >
> > ** Sponsoring Entity **
> > The Apache Incubator
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message