incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Jagielski <>
Subject Re: [discuss] Apache Gobblin Incubator Proposal
Date Wed, 15 Feb 2017 15:35:27 GMT
If you need/want another mentor, I volunteer

> On Feb 14, 2017, at 3:53 PM, Olivier Lamy <> wrote:
> Hi
> Well I don't see issues as no one discuss the proposal.
> So I will start the official vote tomorrow.
> Cheers
> Olivier
> On 6 February 2017 at 14:08, Olivier Lamy <> wrote:
>> Hello everyone,
>> I would like to submit to you a proposal to bring Gooblin to the Apache
>> Software Foundation.
>> The text of the proposal is included below and available as a draft here
>> in the Wiki:
>> We will appreciate any feedback and input.
>> Olivier on behalf of the Gobblin community
>> = Apache Gobblin Proposal =
>> == Abstract ==
>> Gobblin is a distributed data integration framework that simplifies common
>> aspects of big data integration such as data ingestion, replication,
>> organization and lifecycle management for both streaming and batch data
>> ecosystems.
>> == Proposal ==
>> Gobblin is a universal data integration framework. The framework has been
>> used to build a variety of big data applications such as ingestion,
>> replication, and data retention. The fundamental constructs provided by the
>> Gobblin framework are:
>> 1. An expandable set of connectors that allow data to be integrated from
>> a variety of sources and sinks. The range of connectors already available
>> in Gobblin are quite diverse and are an ever expanding set. To highlight
>> just a few examples, connectors exist for databases (e.g., MySQL, Oracle
>> Teradata, Couchbase etc.), web based technologies (REST APIs, FTP/SFTP
>> servers, Filers), scalable storage (HDFS, S3, Ambry etc,), streaming data
>> (Kafka, EventHubs etc.), and a variety of proprietary data sources and
>> sinks (e.g.Salesforce, Google Analytics, Google Webmaster etc.). Similarly,
>> Gobblin has a rich library of converters that allow for conversion of data
>> from one format to another as data moves across system boundaries (e.g.
>> AVRO in HDFS to JSON in another system).
>> 2. Gobblin has a well defined and customizable state management layer
>> that allows writing stateful applications. These are particularly useful
>> when solving problems like bulk incremental ingest and keeping several
>> clusters replicated in sync. The ability to record work that has been
>> completed and what remains in a scalable manner is critical to writing such
>> diverse applications successfully.
>> 3. Gobblin is agnostic to the underlying execution engine. It can be
>> tailored to run ontop of a variety of execution frameworks ranging from
>> multiple processes on a single node, to open source execution engines like
>> MapReduce, Spark or Samza, natively on top of raw containers like Yarn or
>> Mesos, and the public cloud like Amazon AWS or Microsoft Azure. We are
>> extending Gobblin to run on top of a self managed cluster when security is
>> vital.  This allows different applications that require different degrees
>> of scalability, latency or security to be customized to for their specific
>> needs. For example, highly latency sensitive applications can be executed
>> in a streaming environment while batch based execution might benefit
>> applications where the priority might be geared towards optimal container
>> utilization.
>> 4.Gobblin comes out of the box with several diagnosability features like
>> Gobblin metrics and error handling. Collectively, these features allow
>> Gobblin to operate at the scale of petabytes of data. To give just one
>> example, the ability to quarantine a few bad records from an isolated Kafka
>> topic without stopping the entire flow from continued execution is vital
>> when the number of Kafka topics range in the thousands and the collective
>> data handled is in the petabytes.
>> Gobblin thus provides crisply defined software constructs that can be used
>> to build a vast array of data integration applications customizable for
>> varied user needs. It has become a preferred technology for data
>> integration use-cases by many organizations worldwide (see a partial list
>> here).
>> == Background ==
>> Over the last decade, data integration has evolved use case by use case in
>> most companies. For example, at LinkedIn, when Kafka became a significant
>> part of the data ecosystem, a system called Camus was built to ingest this
>> data for analytics processing on Hadoop. Similarly, we had custom pipelines
>> to ingest data from Salesforce, Oracle and myriad other sources. This
>> pattern became the norm rather than the exception and one point, LinkedIn
>> was running at least fifteen different types of ingestion pipelines. This
>> fragmentation has several unfortunate implications. Operational costs scale
>> with the number of pipelines even if the myriad pipelines share a vasty
>> array of common features. Bug fixes and performance optimizations cannot be
>> shared across the pipelines. A common set of practices around debugging and
>> deployment does not emerge. Each pipeline operator will continue to invest
>> in his little silo of the data integration world completely oblivious to
>> the challenges of his fellow operator sitting five tables down.
>> These experiences were the genesis behind the design and implementation of
>> Gobblin. Gobblin thus started out as a universal data ingestion framework
>> focussed on extracting, transforming, and synchronizing large volumes of
>> data between different data sources and sinks. Not surprisingly, given its
>> origins, the initial design of Gobblin placed great emphasis on
>> abstractions that can be leveraged repeatedly. These abstractions have
>> stood the test of time at LinkedIn and we have been able to leverage the
>> constructs well beyond ingest. Gobblin's architecture has allowed us at
>> LinkedIn to use it for a variety of applications ranging from from optimal
>> format conversion to adhering to compliance policies set by European
>> standards. Finally, as noted earlier, Gobblin can be deployed in a variety
>> of execution environments: it can be deployed as a library embedded in
>> another application or can be used to execute jobs on a public cloud. A
>> fluid architectural and execution design story has allowed Gobblin to
>> become a truly successful data integration platform.
>> Gobblin has continued to evolve with a variety of utility packages like
>> Gobblin metrics and Gobblin config management. Collectively, these allow
>> organizations utilizing Gobblin to use a system that has been battle tested
>> at LinkedIn scale. This is something that its consumers have to come to
>> appreciate greatly.
>> == Rationale ==
>> Gobblin's entry to the Apache foundation is beneficial to both the Gobblin
>> and the Apache communities. Gobblin has greatly benefited from its open
>> source roots. Its community and adoption has grown greatly as a result.
>> More importantly, the feedback from the community whether through
>> interactions at meetups or through the mailing list have allowed for a rich
>> exchange of ideas. In order to grow up the Gobblin community and improve
>> the project, we would like to propose Gobblin to the Apache incubator. The
>> Gobblin community will greatly benefit from the established development and
>> consensus processes that have worked well for other projects. The Apache
>> process has served many other open source projects well and we believe that
>> the Gobblin community will greatly benefit from these practices as well.
>> == Initial Goals ==
>> Migrate the existing codebase to Apache
>> Study and Integrate with the Apache development process
>> Ensure all dependencies are compliant with Apache License version 2.0
>> Incremental development and releases per Apache guidelines
>> Improve the relationship between Gobblin and other Apache projects
>> == Current Status ==
>> Gobblin has undergone five major releases (0.5, 0.6, 0.7, 0.8, 0.9) and
>> many minor ones. The latest version, Gobblin 0.9 has just been released in
>> December, 2016. Gobblin is being used in production by over 20
>> organizations. Gobblin codebase is currently hosted at, which
>> will seed the Apache git repository.
>> === Meritocracy ===
>> We plan to invest in supporting a meritocracy. We will discuss the
>> requirements in an open forum. Several companies have already expressed
>> interest in this project, and we intend to invite additional developers to
>> participate. We will encourage and monitor community participation so that
>> privileges can be extended to those that contribute.
>> === Community ===
>> The need for a extensible and flexible data integration platform in the
>> open source is tremendous. Gobblin is currently being used by at least 20
>> organizations worldwide (some examples are listed here). By bringing
>> Gobblin into Apache, we believe that the community will grow even bigger.
>> === Core Developers ===
>> Gobblin was started by engineers at LinkedIn, and now has developers from
>> Google, Facebook, LinkedIn, Cloudera, Nerdwallet, Swisscom, and many other
>> companies.
>> === Alignment ===
>> Gobblin aligns exceedingly well with the Apache ecosystem. Gobblin is
>> built leveraging several existing Apache projects (Apache Helix, Yarn,
>> Zookeeper etc.). As Gobblin matures, we expect to leverage several other
>> Apache projects further. This leverage invariably results in contributions
>> back to these projects (e.g., a contribution to Helix was made during the
>> Gobblin Yarn development). Finally, being an integration platform, it
>> serves as a bridge between several Apache projects like Apache Hadoop and
>> Apache Kafka. This integration is highly desired and their interaction
>> through Gobblin will lead to a virtuous cycle of greater adoption and newer
>> features in these projects. Thus, we believe that it will be a nice
>> addition to the current set of big data projects under the auspices of the
>> Apache foundation.
>> == Known Risks ==
>> === Orphaned Products ===
>> The risk of the Gobblin project being abandoned is minimal. As noted
>> earlier, there are many organizations that have already invested in Gobblin
>> significantly and are thus incentivized to continue development. Many of
>> these organizations operate critical data ingest, compliance and retention
>> pipelines built with Gobblin and are thus heavily invested in the continued
>> success of Gobblin.
>> === Inexperience with Open Source ===
>> Gobblin has existed as a healthy open source project for several years.
>> During that time, we have curated an open-source community successfully.
>> Any risks that we foresee are ones associated with scaling our open source
>> communication and operation process rather than with inherent inexperience
>> in operating an open source project.
>> === Homogenous Developers ===
>> Gobblin’s committers are employed by companies of varying sizes and
>> industry. Committers come from well heeled internet companies like Google,
>> LinkedIn and Facebook. We also have developers from traditional enterprise
>> companies like SwissCom. Well funded startups like Nerdwallet are active in
>> the community of developers. We  plan to double our efforts in cultivating
>> a diverse set of committers for Gobblin.
>> === Reliance on Salaried Developers ===
>> It is expected that Gobblin development will occur on both salaried time
>> and on volunteer time, after hours. The majority of initial committers are
>> paid by their employer to contribute to this project. However, they are all
>> passionate about the project, and we are confident that the project will
>> continue even if no salaried developers contribute to the project. We are
>> committed to recruiting additional committers including non-salaried
>> developers.
>> === Relationships with Other Apache Products ===
>> As noted earlier, Gobblin leverages several open source projects and
>> contributes back to them. There is also overlap with aspects of other
>> Apache projects that we will discuss briefly here. Apache Nifi, like
>> Gobblin aspires to reduce the operational overhead arising from data
>> heterogeneity. Apache Nifi is structured as a visual flow based approach
>> and provides built-in constructs for buffering data, prioritizing data, and
>> understanding data lineage as data flows across systems. Apache Nifi has
>> its own dataflow based execution engine with buffering, scheduling and
>> streaming capabilities. Apache Falcon is a Hadoop centric data governance
>> engine for defining, scheduling, and monitoring data management policies
>> through flow definition typically for data that has been ingested into
>> Hadoop already. Apache Falcon generally delegates data management jobs to
>> tools that already exist in the Hadoop ecosystem (e.g. Distcp, Sqoop, Hive
>> etc). Apache Sqoop is primarily geared for bulk ingest especially from
>> databases which is one part of Gobblin’s feature set. Apache Flume focuses
>> primarily on streaming data movement. Finally, general purpose data
>> processing engines like Apache Flink, Apache Samza, and Apache Spark focus
>> on generic computation.
>> Gobblin design choices intersect with specific features in all of these
>> systems, however in aggregate, it is a different point in the design space.
>> It is designed to handle both streaming and batch data. It supports
>> execution through a standalone cluster mode as well as through existing
>> frameworks such as MR, Yarn, Hive, Samza etc allowing users to choose the
>> deployment model that is optimal for the specific data integration
>> challenge. It provides native optimized implementations for critical
>> integrations such as Kafka, Hadoop - Hadoop copies etc. Gobblin also
>> supports both Hadoop and non-Hadoop data, being able to ingest data into
>> Kafka as well as other key-value stores like Couchbase. Gobblin is also not
>> just a generic computation framework, it has specific constructs for data
>> integration patterns such as data quality metrics and policies. Gobblin’s
>> configuration management system allows it to be fully multi-tenant and take
>> advantage of grouped policies when required. For batch workloads, Gobblin
>> has a planning phase that provides for better resource utilization.
>> In summary, there is healthy diversity in the number of systems
>> approaching the interesting and pressing problem of big data integration.
>> We believe that Gobblin will provide another compelling choice in that
>> design space.
>> === An Excessive Fascination with the Apache Brand ===
>> Gobblin is already a healthy and well known open source project. This
>> proposal is not for the purpose of generating publicity. Rather, the
>> primary benefits to joining Apache are already outlined in the Rationale
>> section.
>> == Documentation ==
>> The reader will find these websites highly relevant:
>> * Website:
>> * Documentation:
>> * Codebase:
>> * User group:!forum/gobblin-users
>> == Source and Intellectual Property Submission Plan ==
>> The Gobblin codebase is currently hosted on Github. This is the exact
>> codebase that we would migrate to the Apache foundation.The Gobblin source
>> code is already licensed under Apache License Version 2.0. Going forward,
>> we will continue to have all the contributions licensed directly to the
>> Apache foundation through our signed Individual Contributor License
>> Agreements for all the committers on the project.
>> == External Dependencies ==
>> To the best of our knowledge, all of Gobblin dependencies are distributed
>> under Apache compatible licenses. Upon acceptance to the incubator, we
>> would begin a thorough analysis of all transitive dependencies to verify
>> this fact and introduce license checking into the build and release process
>> (for instance integrating Apache Rat).
>> == Cryptography ==
>> We do not expect Gobblin to be a controlled export item due to the use of
>> encryption.
>> == Required Resources ==
>> === Mailing lists ===
>> * gobblin-user
>> * gobblin-dev
>> * gobblin-commits
>> * gobblin-private for private PMC discussions (with moderated
>> subscriptions)
>> === Subversion Directory ===
>> Git is the preferred source control system: git://
>> === Issue Tracking ===
>> JIRA Gobblin (GOBBLIN)
>> === Other Resources ===
>> The existing code already has unit and integration tests, so we would
>> like a Jenkins instance to run them whenever a new patch is submitted. This
>> can be added after project creation.
>> == Initial Committers ==
>> * Abhishek Tiwari <abhishektiwari dot btech at gmail dot com>
>> * Shirshanka Das <shirshanka at apache dot org>
>> * Chavdar Botev <cbotev at gmail dot com>
>> * Sahil Takiar <takiar.sahil at gmail dot com>
>> * Yinan Li <liyinan926 at gmail dot com>
>> * Ziyang Liu <>
>> * Lorand Bendig <lbendig at gmail dot com>
>> * Issac Buenrostro <ibuenros at linkedin dot com>
>> * Hung Tran <hutran at linkedin dot com>
>> * Olivier Lamy <olamy at apache dot org>
>> * Jean-Baptiste Onofré <>
>> == Affiliations ==
>> * Abhishek Tiwari - LinkedIn
>> * Shirshanka Das - LinkedIn
>> * Chavdar Botev - Stealth Startup
>> * Sahil Takiar - Cloudera
>> * Yinan Li - Google
>> * Ziyang Liu - Facebook
>> * Lorand Bendig - Swisscom
>> * Issac Buenrostro - LinkedIn
>> * Hung Tran - LinkedIn
>> * Olivier Lamy - Webtide
>> * Jean-Baptiste Onofre - Talend
>> == Sponsors ==
>> === Champion ===
>> Olivier Lamy < olamy at apache dot org>
>> === Nominated Mentors ===
>> * Olivier Lamy <olamy at apache dot org>
>> * Jean-Baptiste Onofre <jbonofre at apache dot org>
>> * ?
>> * ?
>> == Sponsoring Entity ==
>> The Apache Incubator
> -- 
> Olivier Lamy
> |

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message