incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Katherine Marsden <>
Subject [DISCUSS] Quarks proposal
Date Tue, 16 Feb 2016 00:59:53 GMT
Find below a draft proposal for a new incubator project, Quarks for 
discussion.  Quarks is seeking experienced mentors  as well as 
contributors to the project.  Please discuss and provide feedback. The 
proposal is also available on the Wiki at 


Kathey Marsden

= Quarks Proposal =
=== Abstract ===
Quarks is a a stream processing programming model and lightweight 
runtime to execute analytics at devices on the edge or at the gateway.

=== Proposal ===
  Quarks  is a programming model and runtime for streaming analytics at 
the edge.   Applications are developed using a functional flow api to  
define operations on data streams that is executed as a graph of 
"oplets" in a lightweight embeddable runtime.   The SDK provides 
capabilities like windowing, aggregation  and connectors with an 
extensible model for the community to expand its capabilities.

=== Background ===
  Stream processing systems are commonly used to process  data from edge 
devices and there is a need to push some of the  streaming analytics to 
the edge to reduce communication costs, react  locally and offload 
processing from the central systems.  Quarks was developed by IBM as an 
entirely new project to provide an SDK  and lightweight embeddable 
runtime for streaming analytics at the edge.   Quarks was created to be 
an open source project that could provide edge  analytics to a broad 
community and foster collaboration on common  analytics and connectors 
across a broad ecosystem of devices.

=== Rationale ===
  With the growth in number of connected devices (Internet of Things)  
there is a need to execute analytics at the edge in order to take local  
actions based upon sensor information and/or reduce the volume of data  
sent to back-end analytic systems to reduce communication cost.
  Quarks rationale is to provide  consistent and easy to use programming 
models to allow application developers to focus on their  application 
rather than issues like device connectivity, threading etc.   Quarks' 
functional data flow programming model is similar to systems  like 
Apache Flink, Google DataFlow, Java 8 Streams & Apache Spark.   The API 
currently has language bindings for Java8, Java7 and Android. Quarks was 
developed to address requirements for analytics at the  edge for IoT use 
cases that were not addressed by central analytic  solutions.  We 
believe that these capabilities will be useful to many  organizations 
and that the diverse nature of edge devices and use cases  is best 
addressed by an open community. Therefore, we would like to contribute 
Quarks to the ASF as an open source project and begin developing a 
community of developers and users within Apache.

=== Initial Goals ===
  Quarks initial code contribution provides:

  * APIs for developing applications that execute  analytics using a 
per-event (data item) streaming paradigm including  support for windows  
against a stream for aggregation
  * A micro-kernel style runtime for execution.
  * Connectors for MQTT, HTTP, JDBC, File, Apache Kafka &  IBM Watson 
IoT Platform
  * Simple analytics aimed at device sensors (using Apache Common Math)
  * Development mode including a web-console to view the graph of 
running applications
  * Testing mechanism for Quarks applications that integrates with 
assertion based testing systems like JUnit
  * Android specific functionality such as producing a stream that 
contains a phone's sensor events (e.g. ambient temperature, pressure)
  * JUnit tests

  All of the initial code is implemented using Java 8 and when built 
produces jars that can execute on Java 8, Java 7 and Android. The goal 
is to encourage community contributions in any area of Quarks, to  
expand the community (including new committers) and use of Quarks. We  
expect contributions will be driven by real-world use of Quarks by  
anyone active in the IoT space such as auto manufactures, insurance  
companies, etc. as well as individuals experimenting with devices such  
as Raspberry Pis, Arduinos and/or smart phone apps etc. Contributions 
would be welcomed in any aspect of Quarks including:

  * Support for additional programming languages used in devices such as 
C, OpenSwift, Python etc.
  * Specific device feature (e.g. Raspberry Pi, Android) or protocol 
(e.g. OBD-2) support
  * Connectors for device to device (e.g. AllJoyn), device local data 
sources,  or to back-end systems (e.g. a IoT cloud service)
  * Additional analytics, either exposing more functionality from Apache 
Common Math, other libraries or hand-coded analytics.
  * Improvements to the development console, e.g. additional 
visualizations of running applications
  * Documentation, improving existing documentation or adding new guides 
  * Sample applications
  * Testing

  The  code base has been designed to be modular so that additional 
functionality can be added without having to learn it completely, thus  
new contributors can get involved quickly by initially working on a  
focused item such as an additional analytic or connector.

The  only constraints on contributions will be to keep Quarks on its 
focus  of IoT and edge computing, with attributes such as small 
footprint and  modularity to allow deployments to only include what is 
needed for that  specific device and/or application.

=== Current Status ===
   Quarks is a recently released project on Github    The current code is alpha level code 
but is functional and has some  basic tests.  The team is looking 
forward to working in the Apache  community to enhance the functionality 
to allow robust streaming of  devices on the edge.

==== Meritocracy ====
   Quarks was originally created by Dan Debrunner, William Marshall, 
Victor Dogaru, Dale LaBossiere and Susan Cline. We plan to embrace 
meritocracy and encourage developers to participate and reach committer  
status.  Dan Debrunner was the initial creator of the Apache Derby code  
and a committer when Derby was accepted into incubation.  He is an  
Apache member and has experience with the Apache Way.  Derby is a   
successful project that embraces the Apache meritocracy and graduated  
from incubation  with a diverse group of committers.
   With an abundance of devices that potentially can take advantage of  
Quarks, there is a large pool of potential contributors and 
committers.   The initial  team is enthusiastic about assisting and 
encouraging  involvement.

==== Community ====
   Quarks currently has a very small community as it is new, but our 
goal is to build a diverse community at Apache.  The team strongly 
believes that a diverse and vibrant community is critical as devices on  
the edge vary quite a bit.  The community will benefit from developers  
who have expertise in various devices.   We will seek to build a strong  
developer and user community around Quarks.

==== Core Developers ====
  The initial developers have many years of development experience in 
stream processing.  The initial development  team includes developers 
who have experience with Apache, including one  Apache member, and with 
other open source projects on Github.

=== Alignment ===
   Quarks interacts with other Apache solutions such as Apache Kafka 
and  Apache Spark.  Quarks is API driven, modular and written in 100% 
java,  making it easy for developers to pick up and get involved.

=== Known Risks ===
==== Orphaned products ====
   The contributors are from a leading vendor in this space, who has 
shown a commitment to Apache projects in the past.  They are committed  
to working on the project at least for the next several years, as the  
community grows and becomes more diverse.

==== Inexperience with Open Source ====
   Several of the core developers have experience with Apache, 
including  a developer who is a committer on Derby and an Apache 
member.  All of  the core developers have some level of experience with 
the use of open  source packages and with contributions on projects on 
sites such as  GitHub.

==== Homogenous Developers ====
   The initial set of developers come from one company, but we are 
committed to finding a diverse set of committers and contributors. The  
current developers are already very familiar with working with many  
geographies, including developers in most geographies around the 
world.   They  are also very comfortable working in a distributed 

==== Reliance on Salaried Developers ====
   Quarks currently relies on salaried developers at this time, but we  
expect that Quarks will attract a diverse mix of contributors going  
forward.   For Quarks to fully transition to an "Apache Way" governance 
model, we will embrace the meritocracy-centric way of growing the 
community of contributors.

==== Relationships with Other Apache Products ====
   These Apache projects are used by the current codebase:

  * Apache Ant - Build
  * Apache Common Math - Initial analytics
  * Apache HTTP Components HttpClient - HTTP connectivity
  * Apache Kafka - Kafka is supported as a message hub between edge 
Quarks applications and back-end analytics systems
  Events from Quarks applications sent through message hubs (such as 
Apache Kafka) may be consumed by back-end systems such as Apache Flink,  
Apache Spark, Apache Samza, Apache Storm, Google DataFlow (in  
incubation) or others.

==== A Excessive Fascination with the Apache Brand ====
   Quarks will benefit greatly from wide collaboration with developers  
working in the device space.  We feel the Apache brand will help 
attract  those developers who really want to contribute to this space.  
Several  developers involved with this project have a very positive 
history with  Derby and feel that Apache is the right place to grow the 
Quarks  community.  We will respect Apache brand policies and follow the 
Apache  way.

=== Documentation ===

=== Initial Source ===
   Quarks code has been recently released on Github under the Apache 2.0 
license at . It was created by a 
small team of developers, and is written in Java.

=== Source and Intellectual Property Submission Plan ===
   After acceptance into the incubator, IBM will execute a Software 
Grant Agreement and the source code will be transitioned to the Apache  
infrastructure. The code is already licensed under the Apache Software  
License, version 2.0. We do not know of any legal issues that would  
inhibit the transfer to the ASF.

=== External Dependencies ===
  The dependencies all have Apache compatible license. These include 
Apache, MIT and EPL. The current dependencies are:
  * D3
  * Jetty
  * Apache Kafka
  * Metrics
  * MQTTV3
  * SLF4J
  * GSON
  * Apache commons Math3

   Development tools are

  * Java SDK 8
  * Eclipse 4.5
  * Ant 1.9
  * Junit 4.10

=== Cryptography ===
   No cryptographic code is involved with Quarks.

=== Required Resources ===
==== Mailing lists ==== (with moderated subscriptions)

==== Git Repository ====

==== Issue Tracking ====
   Jira Project Quarks (QUARKS)

==== Other Resources ====
   Means of setting up regular builds and test cycle.

=== Initial Committers ===

  * Daniel Debrunner:  djd at apache dot com -  CLA on file
  * Susan Cline:  home4slc at pacbell dot net  - CLA on file
  * William Marshall:  wcmarsha at gmail dot com  - CLA on file
  * Victor Dogaru:  vdogaru at gmail dot com - CLA on file
  * Dale LaBossiere:  dml.apache at gmail dot com  - CLA on file

=== Affiliations ===
  * Daniel Debrunner IBM
  * Susan Cline IBM
  * William Marshall IBM
  * Victor Dogaru  IBM
  * Dale Labossiere IBM

=== Sponsors ===
==== Champion ====
   Katherine Marsden (kmarsden at apache dot org)

==== Nominated Mentors ====
  * Katherine Marsden (kmarsden at apache dot org)
  * Daniel Debrunner (djd at apache dot org)
  * Luciano Resende (lresende at apache dot org)

==== Sponsoring Entity ====
   The Incubator

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message