incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <>
Subject Re: [VOTE] Accept Metron into Apache Incubator
Date Thu, 03 Dec 2015 17:43:34 GMT
+1 (binding)

--Chris Nauroth

On 12/3/15, 9:33 AM, "Owen O'Malley" <> wrote:

>The [DISCUSS] thread has would down, so I'd like to start a VOTE on
>Apache Incubator should accept Metron as a podling. The proposal is pasted
>below and is available on the wiki as well.
>We've added a paragraph in the background section discussing how Apache
>avoids hostile forks of projects, because we don't want to fork
>communities. We've also added Larry McCay, P. Taylor Goetz, and Phillip
>Rhodes to the proposal.
>The vote will run until 12pm PST on Sunday.
>   Owen
>= Apache Metron Proposal =
>/!\ '''FINAL''' /!\
>This proposal is now complete and has been submitted for a VOTE.
>== Abstract ==
>The Metron project is an open source project dedicated to providing an
>extensible and scalable advanced security analytics tool. It has strong
>foundations in the Apache Hadoop ecosystem.
>== Proposal ==
>Metron integrates a variety of open source big data technologies in order
>to offer a centralized tool for security monitoring and analysis. Metron
>provides capabilities for log aggregation, full packet capture indexing,
>storage, advanced behavioral analytics and data enrichment, while applying
>the most current threat-intelligence information to security telemetry
>within a single platform.
>Metron can be divided into 4 areas:
>  1. '''A mechanism to capture, store, and normalize any type of security
>telemetry at extremely high rates.''' Because security telemetry is
>constantly being generated, it requires a method for ingesting the data at
>high speeds and pushing it to various processing units for advanced
>computation and analytics.
>  1. '''Real time processing and application of enrichments''' such as
>threat intelligence, geolocation, and DNS information to telemetry being
>collected. The immediate application of this information to incoming
>telemetry provides the context and situational awareness, as well as the
>³who² and ³where² information that is critical for investigation.
>  1. '''Efficient information storage''' based on how the information will
>be used:
>    a. Logs and telemetry are stored such that they can be efficiently
>mined and analyzed for concise security visibility
>    a. The ability to extract and reconstruct full packets helps an
>answer questions such as who the true attacker was, what data was leaked,
>and where that data was sent
>    a. Long-term storage not only increases visibility over time, but also
>enables advanced analytics such as machine learning techniques to be used
>to create models on the information. Incoming data can then be scored
>against these stored models for advanced anomaly detection.
>  1. '''An interface that gives a security investigator a centralized view
>of data and alerts passed through the system.''' Metron¹s interface
>presents alert summaries with threat intelligence and enrichment data
>specific to that alert on one single page. Furthermore, advanced search
>capabilities and full packet extraction tools are presented to the analyst
>for investigation without the need to pivot into additional tools.
>Big data is a natural fit for powerful security analytics. The Metron
>framework integrates a number of elements from the Hadoop ecosystem to
>provide a scalable platform for security analytics, incorporating such
>functionality as full-packet capture, stream processing, batch processing,
>real-time search, and telemetry aggregation. With Metron, our goal is to
>tie big data into security analytics and drive towards an extensible
>centralized platform to effectively enable rapid detection and rapid
>response for advanced security threats.
>== Background ==
>OpenSOC was developed by Cisco over the last two years and pushed out to
>Github ( under the ALv2. However, the
>development was mostly closed and has largely stopped. As evidence of the
>inactivity, users have complained that pull requests are not answered for
>Finally, no public releases of OpenSOC have been made. From an Apache
>of view, the current community is not viable.
>However, some of the developers of the project have left Cisco and have
>found interest from several others that would like to work together to
>an active and open community at Apache starting from the current OpenSOC
>code base. A message to the current support group proposing moving to
>Apache got a single positive response.
>In general Apache accepts only voluntary contributions and avoids
>hostile forks. In this case, given that the community is demonstrably
>dead, it seems fair to fork the existing code at Apache to allow a new
>community to work on it. Once incubation starts, we will send a
>message pointing to the new home to the OpenSOC support group.
>Because Cisco is not currently interested in being involved, the project
>expects to change their name. The project would like to use Metron,
>although we will perform a podling name search to check for conflicts.
>Metron, meaning measure, is half of the greek root for the word
>'telemetry.'  Metron is also a DC Comics character who ³... wanders in
>search of greater knowledge beyond his own².
>== Rationale ==
>Metron strives to move the state of the art in security analytics forward.
>We want to move away from the proprietary nature of legacy security point
>tools and develop an open platform where people can contribute and share
>datasets, machine learning models, telemetry parsers, sources of telemetry
>enrichment, and threat intelligence feeds.  Cyber security is too large of
>a problem for a single corporation to tackle on its own and the current
>tooling is too fragmented and proprietary for us to be able to rally
>a single tool or vendor.
>In addition to being open and facilitating advancement in security
>analytics, Metron has several advantages over a conventional Security
>Information Management System (SIEM).
>  * Metron uses all open source stack under the hood and runs on commodity
>hardware.  This means Metron is much cheaper to run then the competition.
>In security cost plays a major factor because the cost of your
>countermeasure for monitoring and reacting to a threat should not exceed
>the cost of what is being protected.  By driving down the cost of security
>the economics works for more assets to be monitored, which means more
>secure data centers.
>  * Metron, being in the open, allows additional vetting and scrutiny by
>the open source community for all of its components.  This is a better
>model for a security-oriented tool than doing it closed source.  All the
>problems should be flushed out and fixed in the open. The closed source
>competition does not have this kind of rigor, is motivated by marketing
>sales, and thus, does not inspire confidence when it comes to security.
>  * Being Hadoop-based, Metron can process unprecedented volumes of
>streaming data via Apache Storm.  When an organization is hit with malware
>or malicious behavior most commonly this happens as a part of a global
>malware campaign, signatures for which are known and are available from
>third party threat intelligence feeds.  Having the ability to take in all
>the feeds and reference them against every telemetry message processed by
>Metron in real time does not only facilitate detection of such campaigns,
>it changes the economics for the ³bad guys².  If you have to customize
>malware for each of your targets these global attacks become a lot more
>expensive and non viable for them.
>  * Metron strives to shift conventional SOC workflows away from being
>rules-driven to a more data-driven approach that incorporates machine
>learning and a higher degree of automation and autonomous detection.  The
>modern threat landscape is too dynamic to be manageable via static rules
>alone, which is what conventional SIEMs rely on.  Rule bases tend to
>and if improperly maintained turn themselves into sources of false
>The ability to analyze and model large volumes of data at rest and then
>being able to push up the output of that into a stream processor is
>essential in disrupting the
>== Current Status ==
>As stated in the background section, the current community isn¹t healthy,
>which is why we are proposing moving to Apache Incubator. In this section,
>we will describe the current state of the OpenSOC project.
>=== Meritocracy ===
>The OpenSOC development is controlled by Cisco and pull requests are being
>ignored. The development list is private and requests to join are rejected
>because there is no activity on it. The goal of moving to Apache is to
>a meritocracy where a variety of individuals, regardless of their current
>employer, come together and work together. We understand that diversity,
>open development, and open governance are critical to being a successful
>Apache project.
>=== Community ===
>The OpenSOC project is not responding to pull requests or making releases.
>The easiest solution would be to create a variety of forks of the project
>on github, but that would further fracture the community and prevent it
>from reaching critical mass. Our prefered solution is to build a single
>large diverse and open community at Apache.
>=== Core Developers ===
>The core developers of Metron are James Sirota, Charles Porter, and Mark
>Bittmann. None of them have experience running an open source project, but
>they are eager to learn.
>=== Alignment ===
>The ASF is a natural host for Metron given that it is already the home of
>Hadoop, HBase, Hive, Storm, Kafka, Spark and other emerging big data
>projects. Metron leverages many of Apache open-source products. We are
>interested in a place to develop our community and integrations with the
>other Apache big data projects.
>== Known Risks ==
>=== Orphaned Products ===
>The current product developers are all salaried developers at a small
>number of companies and thus there is a risk of becoming an orphaned
>product. However, the companies view Metron as very important to their
>product offering and plan to ramp up their work in the space. The project
>is unique in the product space and thus has strong potential to become a
>sustainable community.
>=== Inexperience with Open Source ===
>The vast majority of the developers are inexperienced with open source
>development and the Apache Way. One of the major hurdles to graduation
>the Apache Incubator will be demonstrating that they have learned the
>Apache Way and are applying it to how the project is managed. Vinod Kumar
>Vavilapalli is an Apache Member and plans on actively working as a
>committer in the project. They also have the other mentors to help them
>learn as they progress.
>=== Homogenous Developers ===
>The developers are employed by four diverse companies (B23, Hortonworks,
>Mantech, and Rackspace), They are distributed across the United States. We
>hope to attract additional diversity as an Apache project.
>=== Reliance on Salaried Developers ===
>Metron is currently being developed exclusively by salaried developers,
>the goal of coming to Apache is to form a community of users and
>that is much more diverse including non-salaried developers.
>=== Relationships with Other Apache Products ===
>Metron has a strong relationship and dependency with Apache Flume, Hadoop,
>HBase, Hive, Kafka, Spark, and Storm. Being part of Apache¹s Incubation
>community could help with a closer collaboration among these projects and
>as well as others.
>We note that although there is a superficial resemblance to Apache Eagle,
>which does security analysis of Hadoop audit events, the projects are
>significantly different. In particular, Metron is focused on analyzing
>network packet traffic and thus has a very different scope and scale of
>events than Eagle.
>=== An Excessive Fascination with the Apache Brand ===
>While the Apache brand is important, we are much more interested in
>a home for the project that encourages open development and open
>governance. We want to form the new community using the Apache Way with
>strong focus on meritocracy, organizational independence, and open
>== Documentation ==
>The current information on the OpenSOC project is here:
>A slide deck presenting background material is here:
>== Initial Source ==
>The initial code is on github:
>== External Dependencies ==
>Metron has the following external dependencies:
>  * Apache Flume
>  * Apache Hadoop
>  * Apache HBase
>  * Apache Hive
>  * Apache Kafka
>  * Apache Spark
>  * Apache Storm
>  * ElasticSearch
>  * MySQL
>The project understands that it will need to support alternatives for
>that are licensed under a ALv2 compatible license.
>== Cryptography ==
>Metron will eventually support encryption on the wire, but this is not one
>of the initial goals, and we do not expect Metron to be a controlled
>item due to the use of encryption. Metron supports but does not require
>Kerberos authentication mechanism to access secured Hadoop services.
>== Required Resources ==
>=== Mailing List ===
>  * metron-private for private PMC discussions
>  * metron-dev for developers
>  * metron-commits for all commits
>  * metron-users for all users
>=== Version Control ===
>Git is the preferred source control system.
>=== Issue Tracking ===
>=== Other Resources ===
>The existing code already has unit tests so we will make use of existing
>Apache continuous testing infrastructure. The resulting load should not be
>very large.
>== Initial Committers ==
>  * Jim Baker < jim.baker at rackspace dot com >
>  * Mark Bittmann < mark at b23 dot io >
>  * Sheetal Dolas < sheetal at hortonworks dot com >
>  * Discovery Gerdes < discovery.gerdes at rackspace dot com >
>  * P. Taylor Goetz < ptgoetz at apache dot org >
>  * Andrew Hartnett < andrew.hartnett at rackspace dot com >
>  * Dave Hirko < dave at b23 dot io >
>  * Paul Kehrer < paul.kehrer at rackspace dot com >
>  * Brad Kolarov < brad at b23 dot io >
>  * Kiran Komaravolu <kkomaravolu at hortonworks dot com >
>  * Larry McCay < lmccay at >
>  * Ryan Merriman < rmerriman at hortonworks dot com >
>  * Michael Perez < michael.perez at hortonworks dot com>
>  * Charles Porter < Charles.Porter at mcs dot mantech dot com >
>  * Phillip Rhodes < at gmail dot com >
>  * Sean Schulte < sean.schulte at rackspace dot com >
>  * James Sirota < jsirota at hortonworks dot com >
>  * Casey Stella < cstella at hortonworks dot com >
>  * Bryan Taylor < bryan.taylor at rackspace dot com >
>  * Ray Urciuoli < Ray.Urciuoli at mcs dot mantech dot com >
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org >
>  * George Vetticaden < gvetticaden at hortonworks dot com >
>  * Oskar Zabik < oskar.zabik at rackspace dot com >
>== Affiliations ==
>The initial committers are employees of:
>  * Jim Baker - Rackspace
>  * Mark Bittmann - B23
>  * Sheetal Dolas - Hortonworks
>  * Discovery Gerdes - Rackspace
>  * P. Taylor Goetz - Hortonworks
>  * Andrew Hartnett - Rackspace
>  * Dave Hirko - B23
>  * Paul Kehrer - Rackspace
>  * Brad Kolarov - B23
>  * Kiran Komaravolu - Hortonworks
>  * Larry McCay - Hortonworks
>  * Ryan Merriman - Hortonworks
>  * Michael Perez - Hortonworks
>  * Charles Porter - Mantech
>  * Phillip Rhodes - Fogbeam Labs
>  * Sean Schulte - Rackspace
>  * James Sirota - Hortonworks
>  * Casey Stella - Hortonworks
>  * Bryan Taylor - Rackspace
>  * Ray Urciuoli - Mantech
>  * Vinod Kumar Vavilapalli - Hortonworks
>  * George Vetticaden - Hortonworks
>  * Oskar Zabik - Rackspace
>== Sponsors ==
>=== Champion ===
>  * Owen O¹Malley - Apache IPMC member
>=== Nominated Mentors ===
>  * P. Taylor Goetz < ptgoetz at apache dot org > - Apache IPMC member,
>  * Chris Mattmann < mattmann at apache dot org > - Apache IPMC member,
>  * Owen O¹Malley < omalley at apache dot org > - Apache IPMC member,
>  * Billie Rinaldi < billie at apache dot org > - Apache IPMC member,
>  * Vinod Kumar Vavilapalli < vinodkv at apache dot org > - Apache IPMC
>member, Hortonworks
>=== Sponsoring Entity ===
>We are requesting the Incubator to sponsor this project.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message