incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nygard, Carl J" <carl.nyg...@cengage.com>
Subject [PROPOSAL] MetaObjects for Apache Incubator
Date Fri, 26 Jul 2019 15:54:47 GMT
I would like to propose MetaObjects as an Apache Incubator project.

The text of the proposal is included below.

--carl

= Abstract

MetaObjects is a data model and metadata description framework to
drive Model-Driven Development (MDD) across multiple environments
and language stacks providing capabilities such as generated
documentation, code generation, model-driven behaviors, and
model-driven interfaces.


== Proposal

In general, Model-Driven Development describes a process which uses
an abstract definition of a “model” of “something” to generate
concrete artifacts related to that “something” which are consumable by
end-users (client code, developers, etc).  Typical use-cases provide
for automatic generation of related artifacts/capabilities such as:

* Data
* Documentation
* QA/Verification
* Interface definitions
* Code implementations
* DTO/DAO/CRUD operations

Common examples of model-driven development include:

* Spring Annotations
* .NET Data Annotations
* Ruby on Rails
* Swagger/OpenAPI
* Faker.js
* Protobuf

MetaObjects aims to provide a generalized schema for describing data
and related artifacts, along with generative templates to produce
language- or environment-specific artifacts based upon the "canonical"
data description.

MetaObjects hoists the definition of object schemas up one level,
allowing us to treat these schemas as "data" and derive further
business value from the metadata in the model. The model metadata
(attributes) allow for driving behaviors around the models.  These
behaviors can be manifested through conversion to annotations in the
target language via code generation. In addition, systems can implement
behavioral code by using the model metadata attributes directly to
modify default behaviors.


== Background

Having a shared understanding of the core data models and the concepts
they describe is critical to delivering value across a variety of
user-facing systems.

MetaObjects was born to support faster development following the 80/20
rule of most data models having 80% of the same behaviors that could be
solved using MDD, while still supporting 20% with custom-written code.
Over the past 15 years, the primary use cases were building web
applications and APIs where the persistence, business layer, and user
interface were all built using MDD techniques.  As MetaObjects usage
grew, its use-cases evolved to meet the needs of the users.

At Liquent, it was extended to support "overlays", allowing different
deployments of the same codebase to have their behaviors and interfaces
modified by only modifying the MetaObjects data model and metadata
(attributes) to customize for difference clients, without the need for
heavy custom code modifications.

At Cengage, MetaObjects evolved to support interoperable
representations of the same data within both XML and JSON data formats.

Through these experiences, the project adopted the perspective that XML
and JSON are not themselves data models. Instead, they are just
representations (serializations) of a higher-level "object".  In a very
real sense, MetaObjects provides metadata which describes the
high-level "object" in ways that are maximally useful to a wide range
of consumers.

Along the way, it was recognized that more and more functionality could
be efficiently generated from a core "meta-model" of the objects.
Artifacts and capabilities such as DAO/DTO code, model-driven UI,
validations, database schemas, API models, etc were added to the
project.

It was also recognized that there is a strong tendency to "solve the
problem in front of you" and thereby suffer from a focus that is too
narrow. As the MetaObjects project matured, architecture and
capabilities evolved to solve for this myopia.

However, it was also recognized that an organization such as Apache
would attract a wide variety of users and use-cases and serve as the
catalyst to realize the full potential of MetaObjects.

== Rationale

There is a strong need for common models which exist independent of the
technology implementing the model. Identifying opportunities for
generalization and commoditization of these models has been key to the
success of many projects. The success of projects like Swagger/OpenAPI,
Spring, Protobuf clearly indicates the perceived value of MDD. They
also clearly point to the criticality of ensuring that these
capabilities are Open Standards.

MetaObjects embraces these Open Standards by providing capability and
interop across as many technology standards as possible.

MetaObjects also acknowledges the potential to suffer from an inverse
relationship -- as more technologies are integrated, the "lowest common
denominator" shrinks. As part of its design, MetaObjects has facilities
for incorporating specialized behaviors on top of core functionality.
This enables MetaObjects to remain generally useful across technologies
while remaining maximally useful for specific technologies.

For example, when implementing within a micro-services environment, the
contracts defined and provided by these services are the critical
points of integration between service providers and consumers. API
endpoints and API payloads (objects) define the coupling between
components and services within a system.

In an Enterprise environment, we suffer from typical pain points
including:

* Accurate implementation of Core data structures across consumers and
  service integrations
* Maintenance, change management, API versioning, API Payload updates
* Backward (and forward) compatibility within and among service
  implementations

Independent maintenance (across teams) of these common data structures
and the code for manipulating these structures can be a significant
part of the cost of implementation. Similar to the benefits derived
from the Micro-Service Chassis pattern, having a common definition of
these structures provides opportunities for automation and elimination
of redundant effort. The ability to code-generate artifacts across the
breadth of technology existing within an Enterprise organization can
potentially save costs and increase quality/velocity.

Quality, velocity, efficiency and adoption are goals every organization
seeks to achieve. We believe that MetaObjects can make significant
contributions to enable an organization to realize all of these goals.


== Initial Goals

Initial goals for the project is to complete the documentation and add
use-case examples for common modeling problems. In addition,
prioritization of the code generation artifacts across different
consumers and application environments is key to delivering value and
driving adoption within a diverse Enterprise environment.

Both of these goals require input from the community in order to
effectively prioritize the Roadmap.

== Current Status

The current codebase grew out of development on large-scale gaming and
content-delivery systems, initially developed by Doug Mealing. The
codebase has incorporated use-cases from a variety of industries and
applications, and has found success modeling content and core data
structures at the Enterprise level.

As technologies improve, they have been adopted as artifact targets for
the overlay and code-generation capabilities. For example, XSD Schema
can be generated from the model, and the JSON Schema output has been
updated to include v0.7 standards. Related to the JSON Schema, Faker
overlays have been developed, allowing annotations to be added to the
model enabling integration with json-schema-faker
(https://github.com/json-schema-faker/json-schema-faker) for
model-driven data generation.

Validation metadata has been incorporated into the model, which is used
to drive behaviors and generate annotated interfaces across languages
and systems within the enterprise. View metadata supporting interfaces
to render the data for view and edit exist, which drive an older
Struts/JSP UI implementation. There is an effort underway (within
Cengage) developing a similar model-driven UI implemented in Apache
Wicket.

Java POJOs are generated, and typical annotations (e.g. @NotNull) are
supported as appropriate, depending upon metadata in the object model
definition. ORM annotations on the POJOs allow usage with JPA-based
persistence layers.

Objects based on the Immutables library (https://immutables.github.io/)
can also be generated from the same model. Support for polymorphic
serialization to/from JSON is implemented natively within the MetaObjects
library, as well as via Gson TypeAdapters based upon the Immutables
library.

Perl artifacts are currently generated, based on `moose` conventions.

In the near future, targets like C#/.NET, Google Protobuf, Python are
under consideration depending upon need and bandwidth. Partial (or even
full) Swagger/OpenAPI generation is currently being investigated.

In terms of the development community we would like to build around the
MetaObjects project, we are closely aligned with Apache principles.

=== Meritocracy

We believe that collaboration is a necessary prerequisite to great
outcomes. We believe that healthy debate is an aspect of collaboration
necessary for high-quality decisions. We believe that a shared view of
all perspectives generally leads to alignment behind these decisions.
We believe that a good idea is independent of the source of the idea.
We strive to find as many good ideas as possible in part by promoting
and ensuring an inclusive development community. We believe that people
who are invested in the project and supported by their peers produce
great outcomes for the community.

=== Community

The MetaObjects project has grown beyond Doug Mealing’s initial
development. It has been used and extended by many developers in
different organizations, including:

.Community Usage
[%header, cols="25,10,10,55"]
|===
| Company
| Year
| IP Release
| Description


| Draagon Software
| 2002
| Yes
| Doug Mealing's company, built a credit card gateway licensed for use
by third party financial processing systems.

| Liquent
| 2004
| Yes
| Web-based pharmaceutical regulatory product (Insight Manager) sold to
Pharma companies. Each client company customized the solution with
their own overlays and distributed their own versions on CDs.

| Draagon Software
| 2005
| Yes
| Doug Mealing's company, built an online poker software system used in
an online-gaming startup. Gaming company went out of business, IP
transfer to Doug Mealing executed in 2010.

| Sendside Networks
  http://sendside.com
| 2006
| Yes
| Licensed to a startup building a secure communications solution on
the web targeted at the banking industry.
Now owned by www.titleio.com.

| Draagon Software
| 2014
| Yes
| MetaObjects software was updated to the new version and open-sourced
on GitHub (https://github.com/Draagon/draagon-metaobjects)

| Cengage Learning
  http://cengage.com/
| 2014
| No
| Open-source version brought in and modified/enhanced to meet
requirements for online learning applications.

|===

Note, MetaObjects has been open-sourced in 2014 and is available at
https://github.com/Draagon/draagon-metaobjects.

One of the reasons we are pursuing Apache Incubator is in part to find
and build a larger community around MetaObjects. We have benefited as
developers and organizations from the capabilities that are enabled by
MetaObjects, and believe that the larger community would share in those
benefits.

=== Core Developers

The core developers are a collection of individuals, both Contractors
and Employees who are using and improving the MetaObjects tools. There
are at least three Apache Members who work either directly on the
codebase, or are building additional capabilities on top of the
MetaObjects library. Other Apache Members have contributed to the
project in the past in various forms. These Apache Members are
committers on open-source projects such as Camel, ActiveMQ, ServiceMix,
Karaf and others.

=== Alignment

Apache is known not just for the products that are used to deliver
business value, but also the tools that enable developer efficiency.

We believe that MetaObjects is one such tool which also has significant
alignment with other Apache projects, such as:

* Apache Wicket
* Apache Maven
* Apache Daffodil

=== Known Risks

Some may perceive an overlap in functionality between MetaObjects and
other projects, including Swagger/OpenAPI, DFDL (Daffodil), UML, etc.
As these other projects progress, there may be a natural evolution that
erodes the value of MetaObjects.

Each of the projects mentioned above have a slightly differing focus,
and have a strong incentive to "solve the problem in front of you",
which increases variation and divergence between projects. As the
MetaObjects project matured, architecture and capabilities evolved to
solve for this variability. Concepts such as "inheritance",
"extensions" and "overlays" provides flexibility at the "edge" for
supporting different target artifacts while maintaining a consistent
fundamental core. These concepts mitigate the risk of misalignment as
target artifacts evolve.

Regardless of "competitors" in this space, our focus is on making the
right decisions for the project, based on the domain and the value that
we can provide to our end users.

Some may perceive a conflict of interest, or potentially too much
involvement from one organization which becomes overly influential on
the project direction. In reality, we perceive enormous value in the
ideas and contributions that a wider community can provide.

=== Orphaned Products

As mentioned above, multiple organizations have committed to
MetaObjects and have built significant functionality within these
organizations that depends upon MetaObjects. Consequently, we believe
there is little risk to the project in terms of stagnation or
orphanage.

=== Inexperience with Open Source

Many of the committers have experience working on significant open
source projects, many within the Apache umbrella.

In addition, within the current organization, committers within the
MetaObjects project have been champions and sponsors of a "Corporate
Source" project structure strongly based upon the Apache policies and
philosophies.

=== Length of Incubation

We have no illusions about how long the incubation period will be. We
believe that the time spent in Incubation is an investment in the
long-term health of the project, and is in large part dependent upon
the community that is built around the project. Therefore, expectations
about length of time in Incubation are premature without community
input to the process.

=== Homogeneous Developers

As mentioned above, the committers to the MetaObjects project largely
come from a collection of organizations that have incorporated
MetaObjects into their systems.

In addition, we are fortunate to have additional contributions from
current Apache Members who happen to be working within these
organizations.

=== Reliance on Salaried Developers

Currently most of the contributions to the codebase are from salaried
employees.

=== Relationships with Other Apache Products

One of the interesting aspects of this project is the potential for
integration with other open source projects, as well as open standards.

We see potential integrations with the following:

* Swagger/OpenAPI
* Hecate
* Spring Data
* .NET
* Perl
* Python
* Google Protobuf
* JSON Schema
* Faker.js
* ReactJsonSchema
* AngularJsonSchema
* Apache Wicket
* RDBMS
* Apache DFDL (Daffodil)

Depending on how projects progress in the future, we see possible
overlaps in functionality with:

* Swagger/OpenAPI
* Apache DFDL


=== An Excessive Fascination with the Apache Brand

The ASF has a strong brand, and that brand is in itself attractive
because of the attention it brings to their projects. We believe that
the MetaObjects project can benefit from that attention, and that will
help us achieve our long-term goals to bring more efficiency and
quality to the development process through automation and MDD
practices.

It is not surprising that we are interested in joining the ASF in
order to increase our contacts and visibility in the open source world.
Furthermore, we have been enthusiastic users of Apache for a long time,
 and have worked side-by-side with other Apache Members. This personal
 connection strengthens our resolve to be sensitive to the ways in
 which we impact the Apache brand and will work with the Incubator PMC
 and the PRC to ensure the brand policies are respected.


== Documentation


== Initial Source

The MetaObjects project has been under development by Doug Mealing
et al. at various organizations for over 15 years.

== Source and Intellectual Property Submission Plan

The licensing of the MetaObjects project is already under an
open-source umbrella.

== External Dependencies

Core external dependencies include:

* Apache Maven
* Apache Commons
* W3C
* JSON Schema

Additional external dependencies may exist based on plugins or
extensions.

== Cryptography

There is no cryptography involved in the core codebase.


== Required Resources

=== Mailing Lists

* private@metaobjects.incubator.apache.org (w/ moderated subscriptions)
* dev@metaobjects.incubator.apache.org
* users@metaobjects.incubator.apache.org
* commits@metaobjects.incubator.apache.org

=== Subversion Directory

https://svn.apache.org/repos/asf/incubator/meta-objects

This should be a read-only replica of the Git repository (if it's even
necessary).

=== Git Repositories

https://git-wip-us.apache.org/repos/asf/meta-objects.git

=== Issue Tracking

JIRA meta-objects

=== Other Resources

None


== Initial Committers

* Doug Mealing (doug at dougmealing dot com)
* Alejandro Mora (alejandro dot mora dot castro at gmail dot com)
* James Carman (jcarman at apache dot org)
* Johan Edstrom (joed at apache dot org)
* Jamie Goodyear (jgoodyear at apache dot org)
* Heath Kesler (hkesler at apache dot org)
* Carl Nygard (cjnygard at gmail dot com)


== Sponsors

* Champion:  Johan Edstrom - Apache, Savoirtech

* Nominated Mentors:
** James Carman - Apache, Cengage
** Heath Kesler - Apache, Savoirtech
** Jamie Goodyear - Apache, Savoirtech

* Sponsoring Entity: Incubator




---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message