incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Furkan KAMACI <>
Subject Re: [VOTE] Accept StreamPipes into the Apache Incubator
Date Thu, 07 Nov 2019 20:10:04 GMT

+1, good lucks!

Kind Regards,

7 Kas 2019 Per, saat 22:40 tarihinde Christofer Dutz <> şunu yazdı:

> A big +1 (binding) from me
> Chris
> ________________________________
> Von: Julian Feinauer <>
> Gesendet: Donnerstag, 7. November 2019 20:19:43
> An: <>
> Betreff: Re: [VOTE] Accept StreamPipes into the Apache Incubator
> Hi Dominik,
> I like StreamPipes, as you know, thus my vote is
> +1 (binding)
> Julian
> Am 07.11.19, 20:00 schrieb "Dominik Riemer" <>:
>     Hi all,
>     following up the [DISCUSS] thread on StreamPipes (
> I would like to call a VOTE to accept StreamPipes into the Apache Incubator.
>     Please cast your vote:
>       [ ] +1, bring StreamPipes into the Incubator
>       [ ] +0, I don't care either way
>       [ ] -1, do not bring StreamPipes into the Incubator, because...
>     The vote will open at least for 72 hours and only votes from the
> Incubator PMC are binding, but votes from everyone are welcome.
>     Dominik
>     ----
>     StreamPipes Proposal (
>     == Abstract ==
>     StreamPipes is a self-service (Industrial) IoT toolbox to enable
> non-technical users to connect, analyze and explore (Industrial) IoT data
> streams.
>     = Proposal =
>     The goal of StreamPipes (<
>>) is to provide an easy-to-use toolbox for
> non-technical users, e.g., domain experts, to exploit data streams coming
> from (Industrial) IoT devices. Such users are provided with an intuitive
> graphical user interface with the Pipeline Editor at its core. Users are
> able to graphically model processing pipelines based on data sources
> (streams), data processors and data sinks. Data processors and sinks are
> self-contained microservices, which implement either stateful or stateless
> processing logic (e.g., a trend detection or image classifier). Their
> processing logic is implemented using one of several provided wrappers (we
> currently have wrappers for standalone/Edge-based processing, Apache Flink,
> Siddhi and working wrapper prototypes for Apache Kafka Streams and Spark,
> in the future we also plan to integrate with Apache Beam). An SDK allows to
> easily create new pipeline elements. Pipeline elements can be installed at
> runtime. To support users in creating pipelines, an underlying
> semantics-based data model enables pipeline elements to express
> requirements on incoming data streams that need to be fulfilled, thus
> reducing modeling errors.
>     Data streams are integrated by using StreamPipes Connect, which allows
> to connect data sources (based on standard protocols, such as MQTT, Kafka,
> Pulsar, OPC-UA and further PLC4X-supported protocols) without further
> programming using a graphical wizard. Additional user-faced modules of
> StreamPipes are a Live dashboard to quickly explore IoT data streams and a
> wizard that generates code templates for new pipeline elements, a Pipeline
> Element Installer used to extend the algorithm feature set at runtime.
>     === Background ===
>     StreamPipes was started in 2014 by researchers from FZI Research
> Center for Information Technology in Karlsruhe, Germany. The original
> prototype was funded by an EU project centered around predictive analytics
> for the manufacturing domain. Since then, StreamPipes was constantly
> improved and extended by public funding mainly from federal German
> ministries. In early 2018, the source code was officially released under
> the Apache License 2.0. At the same time, while we focused on bringing the
> research prototype to a production-grade tool, the first companies started
> to use StreamPipes. Currently, the primary goal is to widen the user and
> developer base. At ApacheCon NA 2019, after having talked to many people
> from the Apache Community, we finally decided that we would like to bring
> StreamPipes to the Apache Incubator.
>     === Rationale ===
>     The (Industrial) IoT domain is a highly relevant and emerging sector.
> Currently, IoT platforms are offered by many vendors ranging from SMEs up
> to large enterprises. We believe that open source alternatives are an
> important cornerstone for manufacturing companies to easily adopt
> data-driven decision making. From our point of view, StreamPipes fits very
> well into the existing (I)IoT ecosystem within the ASF, with projects such
> as Apache PLC4X focusing on connecting machine data from PLCs, or other
> tools we are also using either in the core of StreamPipes or with
> integrations (Apache Kafka, Apache IoTDB, Apache Pulsar). StreamPipes
> itself focuses on enabling self-service IoT data analytics for
> non-technical users.
>     The whole StreamPipes code is currently on Github. To get a rough
> estimate of the project size:
>     * streampipes: Backend and core modules, ~3300 commits
>     * streampipes-ui: User Interface, ~1300 commits
>     * streampipes-pipeline-elements: ~100 Pipeline Elements (data
> processors/algorithms and sinks), ~500 Commits
>     * streampipes-connect-adapters: ~20 Adapters to connect data, ~100
> commits To achieve our goal to further extend the code base with new
> features, new connectors and new algorithms and to grow both the user and
> developer community, we believe that a community-driven development process
> is the best way to further develop StreamPipes. Finally, after having
> talked to committers from various Apache IoT-related projects and
> participation in spontaneous hacking sessions and being impressed by the
> collaboration among individual projects, we decided that (from our point of
> view) the ASF is the ideal place to be the future home of StreamPipes.
>     === Initial Goals ===
>     * Move the existing codebase to Apache
>     * Fully align with Apache development- and release processes
>     * Perform name search and do a thorough review of existing licenses
>     * First Apache release
>     === Current Status ===
>     ** Meritocracy **
>     We are absolutely committed to strengthen StreamPipes as a real
> community-driven open source project. The existing committer base is highly
> motivated to foster the open source way in the industrial IoT sector and,
> together with existing Apache communities focused on this domain, provide
> open source tooling for Industrial IoT projects in the same way Apache
> offers in the Big Data space, for instance.
>     The development philosophy behind StreamPipes has always followed the
> principles of meritocracy - although most committers are still active in
> the project, we managed to onboard new, committed developers regularly. 2
> people, who are today core of the developer team, have joined during the
> past year. Therefore, we would aim to continuously expand the PMC and
> committer base based on merit.
>     ** Community **
>     Since being open-sourced in 2018, the public interest in StreamPipes
> has steadily grown. Several companies, mainly from the manufacturing
> domain, have tested StreamPipes in form of proof-of-concept projects. First
> companies have started to use StreamPipes in production. This was due to a
> high number of events from meetups, research conferences, demo sessions up
> to hackathons we participated or organized during the past two years. After
> having generated a general interest in StreamPipes, our next focus will be
> to find more committers to diversify the contributor base.
>     ** Core Developers **
>     The core developers of the system are Dominik Riemer, Philipp Zehnder,
> Patrick Wiener and Johannes Tex. All core developers are initial committers
> in the current proposal. Some former students who recently started to work
> at companies and who have also worked on the project with great commitment,
> will be asked to further contribute to the project.
>     ** Alignment **
>     StreamPipes has dependencies to a lot of existing Apache projects -
> this is one reason why we think that the ASF is the best future home for
> StreamPipes. The messaging layer is based on Apache Kafka (and also Apache
> Pulsar as a future option), and runtime wrappers exist for Apache Flink,
> Apache Spark and Apache Kafka Streams. StreamPipes Connect already includes
> adapters for several Apache projects. Most importantly, we integrate (and
> plan to deepen the integration) with IIoT-focused projects such as Apache
> PLC4X. Also, several data sinks exist to send messages to tools from other
> Apache projects (e.g., Apache Kafka, Apache Pulsar, and Apache IoTDB).
> Together with these tools (and also after having talked to the core
> developers after this year's ApacheCon) we are absolutely convinced that a
> tight integration between these tools will strengthen the open source IoT
> ecosystem.
>     === Known Risks ===
>     ** Orphaned Products **
>     We don't expect the risk of an orphaned product. The initial
> committers have worked on the project for years and are absolutely
> committed to making this open source tool a great success. All initial
> committers are committed to work on StreamPipes in their free time.
>     ** Inexperience with Open Source **
>     All initial committers have years of expertise related to open source
> development and understand what open source means. However, none of the
> initial committers are currently committers to Apache projects, although
> some have already contributed to some projects. From a variety of events
> and from intensively studying Apache mailing lists, we are sure that the
> Apache Way is the way we'd like the project to move into the future. We
> expect to benefit from the experiences from the ASF in building successful
> open source projects.
>     ** Length of Incubation **
>     We are aware that incubation is a process that is focused on building
> the community, learning the Apache Way and other important things such as
> learning the release process and handling licensing and trademark issues.
> We are also aware that, although there is a steadily increasing interest in
> StreamPipes, a major challenge we would need (and are willing) to work on
> during the incubation phase is widening the committer base. We look forward
> to that as a large developer base is exactly what we are striving for.
>     ** Homogeneous Developers **
>     Most current developers work for the same institution (FZI). The
> motivation of all developers goes beyond their commitment to work and all
> current committers work on StreamPipes in their free time. Recently, we
> have received first pull requests from external contributors and a growing
> interest from users and companies outside of FZI. First manufacturing
> companies have already evaluated and adopted StreamPipes. To attract
> external developers, we've created an extensive documentation, have a Slack
> channel to quickly answer questions, and provide help via mail. Therefore,
> we believe that making the developer community more heterogeneous is not
> only mandatory, but something that can be achieved during the next months.
>     ** Reliance on salaried developers **
>     Currently, StreamPipes receives support from salaried developers,
> mainly research scientists from FZI. However, all core developers
> substantially work on StreamPipes in their spare time. As this has been the
> case from the beginning in early 2014, it can be expected that a
> substantial portion of volunteers will continue to be working on the
> project and we aim at strengthening the base of non-paid committers and
> paid committers of other companies. At the same time, funding of the
> initial StreamPipes team is secured by public funding for the next few
> years, making sure that there will be also enough commitment from
> developers during their work time.
>     ** Relationships with other Apache products ** StreamPipes is often
> compared to tools such as Node-Red and Apache Nifi. This is mainly based on
> a similar UI concept (dataflow approach). Despite some technological
> differences (e.g., the microservice analytics approach vs. single-host
> runtime of Node-Red, the wrapper architecture and the underlying
> semantics-based model), we believe the target audience differs. We aim to
> collaborate with the Apache Nifi community in terms of exchanging best
> practices and also integrating both projects (e.g., by building connectors).
>     As mentioned above, quite a few adapters and data sinks are already
> available that link to existing Apache projects.
>     ** An excessive fascination with the Apache Brand ** Although we
> recognize the Apache brand as the most visible brand in the open source
> domain, the primary goal of this proposal is not to create publicity, but
> to widen the developer base. We believe that successful projects have broad
> and diverse communities. We expect that an Apache project, with a clear and
> proven way to develop open source software, helps in finding new
> committers. As the core development team has already worked on StreamPipes
> for the past few years and is fully committed to the software and its
> benefit for the industrial IoT domain, we would also continue development
> without being an Apache project.
>     === Documentation ===
>     Currently, we host a website at More
> technical info (user + developer guide) can be found in the documentation:
>, where users can find tutorials and manuals
> on how to extend StreamPipes using the SDK.
>     === Initial Source ===
>     Currently, the following Github repositories exist, all licensed under
> the Apache Software License 2.0:
>     * streampipes (, the
> backend & pipeline management module)
>     * streampipes-ui (,
> the UI module)
>     * streampipes-pipeline-elements (
>, library
> of data processors and sinks)
>     * streampipes-connect-adapters (
> StreamPipes connect adapters)
>     * streampipes-docs (
>, the abovementioned
> documentation)
>     === Source and intellectual property submission plan === All initial
> committers will sign a ICLA with the ASF. FZI, as the organizational body
> that has employed the main contributors of StreamPipes, will sign a CCLA
> and donate the codebase to the ASF (both subject to formal approval). All
> major contributors are still active in the project.
>     === External Dependencies ===
>     We did an initial review of all dependencies used in the various
> projects. No critical libraries that depend on category X licenses were
> found, some minor issues have already been resolved (e.g., removing
> dependencies to org.json libraries). Most external dependencies used by the
> Java-based (backend, pipeline-elements and connect) modules are licensed
> under the Apache License 2.0, whereas some licenses are Cat B (e.g., CDDL).
> Most external dependencies the UI requires on are licensed under the MIT
> license.
>     Once we are moving to the Incubator, we would do a complete check of
> all transitive dependencies. We don't expect any surprises here.
>     === Cryptography ===
>     (not applicable)
>     === Required Resources ===
>     ** Mailing Lists **
>     We plan to use the following mailing lists:
>     *
>     *
>     *
>     *
>     As StreamPipes is targeted to a non-technical audience, we see a
> dedicated user mailing list as an important requirement to help users.
>     ** Subversion directory **
>     (not applicable)
>     ** Git repositories **
>     We would like to use Git for source code management and enable Github
> mirroring functionality.
>     As we plan to merge some of the repos described above to simplify the
> release process we ask to create the following source repositories:
>     * streampipes (containing backend + UI)
>     * streampipes-extensions (containing modules that can be dynamically
> installed at runtime: pipeline elements and connect adapters)
>     * streampipes-website (containing docs + website)
>     ** Issue tracking **
>     JIRA ID: StreamPipes
>     === Initial Committers ===
>     List of initial committers in alphabetical order:
>     * Christofer Dutz (christofer.dutz at c-ware dot de)
>     * Dominik Riemer (dominik dot riemer at gmail dot com)
>     * Johannes Tex (tex at fzi dot de)
>     * Patrick Wiener (wiener at fzi dot de)
>     * Philipp Zehnder (zehnder at fzi dot de)
>     === Sponsors ===
>     ** Champion **
>     * Christofer Dutz (christofer.dutz at c-ware dot de)
>     ** Mentors **
>     * Christofer Dutz (christofer.dutz at c-ware dot de)
>     * Julian Feinauer (Jfeinauer at apache dot org)
>     * Kenneth Knowles (kenn at apache dot org)
>     * Justin Mclean (justin at classsoftware dot com)
>     * Jean-Baptiste Onofré (jb at nanthrax dot net)
>     ** Sponsoring Entity **
>     The Apache Incubator
>     ---------------------------------------------------------------------
>     To unsubscribe, e-mail:
>     For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message