incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ming Wen <>
Subject Re: [VOTE] Accept StreamPipes into the Apache Incubator
Date Mon, 11 Nov 2019 01:54:45 GMT
+1 (non-binding)

Ming Wen, Apache APISIX
Twitter: _WenMing

Dominik Riemer <> 于2019年11月8日周五 上午3:00写道:

> Hi all,
> following up the [DISCUSS] thread on StreamPipes (
> I would like to call a VOTE to accept StreamPipes into the Apache Incubator.
> Please cast your vote:
>   [ ] +1, bring StreamPipes into the Incubator
>   [ ] +0, I don't care either way
>   [ ] -1, do not bring StreamPipes into the Incubator, because...
> The vote will open at least for 72 hours and only votes from the Incubator
> PMC are binding, but votes from everyone are welcome.
> Dominik
> ----
> StreamPipes Proposal (
> == Abstract ==
> StreamPipes is a self-service (Industrial) IoT toolbox to enable
> non-technical users to connect, analyze and explore (Industrial) IoT data
> streams.
> = Proposal =
> The goal of StreamPipes ( is to provide an
> easy-to-use toolbox for non-technical users, e.g., domain experts, to
> exploit data streams coming from (Industrial) IoT devices. Such users are
> provided with an intuitive graphical user interface with the Pipeline
> Editor at its core. Users are able to graphically model processing
> pipelines based on data sources (streams), data processors and data sinks.
> Data processors and sinks are self-contained microservices, which implement
> either stateful or stateless processing logic (e.g., a trend detection or
> image classifier). Their processing logic is implemented using one of
> several provided wrappers (we currently have wrappers for
> standalone/Edge-based processing, Apache Flink, Siddhi and working wrapper
> prototypes for Apache Kafka Streams and Spark, in the future we also plan
> to integrate with Apache Beam). An SDK allows to easily create new pipeline
> elements. Pipeline elements can be installed at runtime. To support users
> in creating pipelines, an underlying semantics-based data model enables
> pipeline elements to express requirements on incoming data streams that
> need to be fulfilled, thus reducing modeling errors.
> Data streams are integrated by using StreamPipes Connect, which allows to
> connect data sources (based on standard protocols, such as MQTT, Kafka,
> Pulsar, OPC-UA and further PLC4X-supported protocols) without further
> programming using a graphical wizard. Additional user-faced modules of
> StreamPipes are a Live dashboard to quickly explore IoT data streams and a
> wizard that generates code templates for new pipeline elements, a Pipeline
> Element Installer used to extend the algorithm feature set at runtime.
> === Background ===
> StreamPipes was started in 2014 by researchers from FZI Research Center
> for Information Technology in Karlsruhe, Germany. The original prototype
> was funded by an EU project centered around predictive analytics for the
> manufacturing domain. Since then, StreamPipes was constantly improved and
> extended by public funding mainly from federal German ministries. In early
> 2018, the source code was officially released under the Apache License 2.0.
> At the same time, while we focused on bringing the research prototype to a
> production-grade tool, the first companies started to use StreamPipes.
> Currently, the primary goal is to widen the user and developer base. At
> ApacheCon NA 2019, after having talked to many people from the Apache
> Community, we finally decided that we would like to bring StreamPipes to
> the Apache Incubator.
> === Rationale ===
> The (Industrial) IoT domain is a highly relevant and emerging sector.
> Currently, IoT platforms are offered by many vendors ranging from SMEs up
> to large enterprises. We believe that open source alternatives are an
> important cornerstone for manufacturing companies to easily adopt
> data-driven decision making. From our point of view, StreamPipes fits very
> well into the existing (I)IoT ecosystem within the ASF, with projects such
> as Apache PLC4X focusing on connecting machine data from PLCs, or other
> tools we are also using either in the core of StreamPipes or with
> integrations (Apache Kafka, Apache IoTDB, Apache Pulsar). StreamPipes
> itself focuses on enabling self-service IoT data analytics for
> non-technical users.
> The whole StreamPipes code is currently on Github. To get a rough estimate
> of the project size:
> * streampipes: Backend and core modules, ~3300 commits
> * streampipes-ui: User Interface, ~1300 commits
> * streampipes-pipeline-elements: ~100 Pipeline Elements (data
> processors/algorithms and sinks), ~500 Commits
> * streampipes-connect-adapters: ~20 Adapters to connect data, ~100 commits
> To achieve our goal to further extend the code base with new features, new
> connectors and new algorithms and to grow both the user and developer
> community, we believe that a community-driven development process is the
> best way to further develop StreamPipes. Finally, after having talked to
> committers from various Apache IoT-related projects and participation in
> spontaneous hacking sessions and being impressed by the collaboration among
> individual projects, we decided that (from our point of view) the ASF is
> the ideal place to be the future home of StreamPipes.
> === Initial Goals ===
> * Move the existing codebase to Apache
> * Fully align with Apache development- and release processes
> * Perform name search and do a thorough review of existing licenses
> * First Apache release
> === Current Status ===
> ** Meritocracy **
> We are absolutely committed to strengthen StreamPipes as a real
> community-driven open source project. The existing committer base is highly
> motivated to foster the open source way in the industrial IoT sector and,
> together with existing Apache communities focused on this domain, provide
> open source tooling for Industrial IoT projects in the same way Apache
> offers in the Big Data space, for instance.
> The development philosophy behind StreamPipes has always followed the
> principles of meritocracy - although most committers are still active in
> the project, we managed to onboard new, committed developers regularly. 2
> people, who are today core of the developer team, have joined during the
> past year. Therefore, we would aim to continuously expand the PMC and
> committer base based on merit.
> ** Community **
> Since being open-sourced in 2018, the public interest in StreamPipes has
> steadily grown. Several companies, mainly from the manufacturing domain,
> have tested StreamPipes in form of proof-of-concept projects. First
> companies have started to use StreamPipes in production. This was due to a
> high number of events from meetups, research conferences, demo sessions up
> to hackathons we participated or organized during the past two years. After
> having generated a general interest in StreamPipes, our next focus will be
> to find more committers to diversify the contributor base.
> ** Core Developers **
> The core developers of the system are Dominik Riemer, Philipp Zehnder,
> Patrick Wiener and Johannes Tex. All core developers are initial committers
> in the current proposal. Some former students who recently started to work
> at companies and who have also worked on the project with great commitment,
> will be asked to further contribute to the project.
> ** Alignment **
> StreamPipes has dependencies to a lot of existing Apache projects - this
> is one reason why we think that the ASF is the best future home for
> StreamPipes. The messaging layer is based on Apache Kafka (and also Apache
> Pulsar as a future option), and runtime wrappers exist for Apache Flink,
> Apache Spark and Apache Kafka Streams. StreamPipes Connect already includes
> adapters for several Apache projects. Most importantly, we integrate (and
> plan to deepen the integration) with IIoT-focused projects such as Apache
> PLC4X. Also, several data sinks exist to send messages to tools from other
> Apache projects (e.g., Apache Kafka, Apache Pulsar, and Apache IoTDB).
> Together with these tools (and also after having talked to the core
> developers after this year's ApacheCon) we are absolutely convinced that a
> tight integration between these tools will strengthen the open source IoT
> ecosystem.
> === Known Risks ===
> ** Orphaned Products **
> We don't expect the risk of an orphaned product. The initial committers
> have worked on the project for years and are absolutely committed to making
> this open source tool a great success. All initial committers are committed
> to work on StreamPipes in their free time.
> ** Inexperience with Open Source **
> All initial committers have years of expertise related to open source
> development and understand what open source means. However, none of the
> initial committers are currently committers to Apache projects, although
> some have already contributed to some projects. From a variety of events
> and from intensively studying Apache mailing lists, we are sure that the
> Apache Way is the way we'd like the project to move into the future. We
> expect to benefit from the experiences from the ASF in building successful
> open source projects.
> ** Length of Incubation **
> We are aware that incubation is a process that is focused on building the
> community, learning the Apache Way and other important things such as
> learning the release process and handling licensing and trademark issues.
> We are also aware that, although there is a steadily increasing interest in
> StreamPipes, a major challenge we would need (and are willing) to work on
> during the incubation phase is widening the committer base. We look forward
> to that as a large developer base is exactly what we are striving for.
> ** Homogeneous Developers **
> Most current developers work for the same institution (FZI). The
> motivation of all developers goes beyond their commitment to work and all
> current committers work on StreamPipes in their free time. Recently, we
> have received first pull requests from external contributors and a growing
> interest from users and companies outside of FZI. First manufacturing
> companies have already evaluated and adopted StreamPipes. To attract
> external developers, we've created an extensive documentation, have a Slack
> channel to quickly answer questions, and provide help via mail. Therefore,
> we believe that making the developer community more heterogeneous is not
> only mandatory, but something that can be achieved during the next months.
> ** Reliance on salaried developers **
> Currently, StreamPipes receives support from salaried developers, mainly
> research scientists from FZI. However, all core developers substantially
> work on StreamPipes in their spare time. As this has been the case from the
> beginning in early 2014, it can be expected that a substantial portion of
> volunteers will continue to be working on the project and we aim at
> strengthening the base of non-paid committers and paid committers of other
> companies. At the same time, funding of the initial StreamPipes team is
> secured by public funding for the next few years, making sure that there
> will be also enough commitment from developers during their work time.
> ** Relationships with other Apache products ** StreamPipes is often
> compared to tools such as Node-Red and Apache Nifi. This is mainly based on
> a similar UI concept (dataflow approach). Despite some technological
> differences (e.g., the microservice analytics approach vs. single-host
> runtime of Node-Red, the wrapper architecture and the underlying
> semantics-based model), we believe the target audience differs. We aim to
> collaborate with the Apache Nifi community in terms of exchanging best
> practices and also integrating both projects (e.g., by building connectors).
> As mentioned above, quite a few adapters and data sinks are already
> available that link to existing Apache projects.
> ** An excessive fascination with the Apache Brand ** Although we recognize
> the Apache brand as the most visible brand in the open source domain, the
> primary goal of this proposal is not to create publicity, but to widen the
> developer base. We believe that successful projects have broad and diverse
> communities. We expect that an Apache project, with a clear and proven way
> to develop open source software, helps in finding new committers. As the
> core development team has already worked on StreamPipes for the past few
> years and is fully committed to the software and its benefit for the
> industrial IoT domain, we would also continue development without being an
> Apache project.
> === Documentation ===
> Currently, we host a website at More
> technical info (user + developer guide) can be found in the documentation:
>, where users can find tutorials and manuals
> on how to extend StreamPipes using the SDK.
> === Initial Source ===
> Currently, the following Github repositories exist, all licensed under the
> Apache Software License 2.0:
> * streampipes (, the
> backend & pipeline management module)
> * streampipes-ui (, the
> UI module)
> * streampipes-pipeline-elements (
>, library
> of data processors and sinks)
> * streampipes-connect-adapters (
> StreamPipes connect adapters)
> * streampipes-docs (,
> the abovementioned documentation)
> === Source and intellectual property submission plan === All initial
> committers will sign a ICLA with the ASF. FZI, as the organizational body
> that has employed the main contributors of StreamPipes, will sign a CCLA
> and donate the codebase to the ASF (both subject to formal approval). All
> major contributors are still active in the project.
> === External Dependencies ===
> We did an initial review of all dependencies used in the various projects.
> No critical libraries that depend on category X licenses were found, some
> minor issues have already been resolved (e.g., removing dependencies to
> org.json libraries). Most external dependencies used by the Java-based
> (backend, pipeline-elements and connect) modules are licensed under the
> Apache License 2.0, whereas some licenses are Cat B (e.g., CDDL). Most
> external dependencies the UI requires on are licensed under the MIT
> license.
> Once we are moving to the Incubator, we would do a complete check of all
> transitive dependencies. We don't expect any surprises here.
> === Cryptography ===
> (not applicable)
> === Required Resources ===
> ** Mailing Lists **
> We plan to use the following mailing lists:
> *
> *
> *
> *
> As StreamPipes is targeted to a non-technical audience, we see a dedicated
> user mailing list as an important requirement to help users.
> ** Subversion directory **
> (not applicable)
> ** Git repositories **
> We would like to use Git for source code management and enable Github
> mirroring functionality.
> As we plan to merge some of the repos described above to simplify the
> release process we ask to create the following source repositories:
> * streampipes (containing backend + UI)
> * streampipes-extensions (containing modules that can be dynamically
> installed at runtime: pipeline elements and connect adapters)
> * streampipes-website (containing docs + website)
> ** Issue tracking **
> JIRA ID: StreamPipes
> === Initial Committers ===
> List of initial committers in alphabetical order:
> * Christofer Dutz (christofer.dutz at c-ware dot de)
> * Dominik Riemer (dominik dot riemer at gmail dot com)
> * Johannes Tex (tex at fzi dot de)
> * Patrick Wiener (wiener at fzi dot de)
> * Philipp Zehnder (zehnder at fzi dot de)
> === Sponsors ===
> ** Champion **
> * Christofer Dutz (christofer.dutz at c-ware dot de)
> ** Mentors **
> * Christofer Dutz (christofer.dutz at c-ware dot de)
> * Julian Feinauer (Jfeinauer at apache dot org)
> * Kenneth Knowles (kenn at apache dot org)
> * Justin Mclean (justin at classsoftware dot com)
> * Jean-Baptiste Onofré (jb at nanthrax dot net)
> ** Sponsoring Entity **
> The Apache Incubator
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message