incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <>
Subject Re: [DISCUSS] Crunch to join the Apache Incubator
Date Sat, 26 May 2012 14:12:51 GMT
On 25 May 2012 20:00, Josh Wills <> wrote:

> Hi Steve,
> Thank you for your thoughtful comments. Replies inlined below.
> > 1. He's using it at work, so represents the end users.
> A super-majority of the initial committers are also end users. I use
> Crunch on my own projects (e.g.,
> and
> ), Cloudera solutions architects
> use Crunch on client projects, Robert is building tools on top of
> Crunch at WibiData, and Gabriel and Chris use it for building
> pipelines at TomTom. I can't speak for Tom and Vinod, but of course,
> they have other positive qualities. :)

It's still a fairly limited set of organisations and lacks independence.
Jakob and colleagues have no strategic goals in ensuring the
success/failure of any specific OSS project, merely getting the right tools
for their job.

In a true OSS project -not one that is released under some OSS license but
is effectively a single-vendor-project (JBoss, MySQL, etc) - end users are
not merely consumers of the output, they are potential engineering
resources to be co-opted, be it in their suggestions for improvement,
documentation, bugreps, tests and code itself. That's the challenge -and
it's not easy, especially in a project where some of the developers work on
it full time, others are people that use it a bit and find bugs. Those
little contributors need to be nurtured until they become good ones.

> > 2. His code is always of high quality
> I in no way meant to disparage Jakob or his coding. The objective of
> my reply was say "no" in the most apologetic, obsequious way possible
> while not going so far over the top as to sound insincere. Having
> LinkedIn on board would be a tremendous PR boost for the project. It
> was painful to say no.
> I am in no way savvy in the ways of Apache or the politics of the ASF.

Not Apache politics, but a core belief: the notion that a community is
actually more important than the codebase itself. The goal of an incubating
project is not so much to get into the code into a shape where it is ready
to graduate -but build a community to a point where it is considered

If you don't want that, but instead want to have a project over which you
retain tight control, you are free to continue to host it on github.

> I understand that smart people who I respect a great deal think that
> this is the wrong decision. But I think that it takes something really
> great for someone to see a project like Crunch, play around with, and
> then take the time to make some contributions to it without any
> expectation of recognition, in the form of an Apache committership or
> anything else. That was what Gabriel and Chris and Robert did over the
> past few months. I really admire that, and I think that it deserves
> some special recognition, however small.

That is good, and their past and hopefully ongoing work will help the
project -I just think that it would have helped the project if Jakob's had
been embraced

> I'm willing to have some
> people not like me or think I'm dumb if that's the price of giving
> that to them.

I all I have are concerns that the proposal is at risk from the same
problems that others have had in incubation.

> > 3. Given the ongoing discussion on diversity w.r.t Flume, I think it
> would
> > be wise to not follow that projects example, and try to get broader
> > involvement from the outset.
> I agree that it is critical to have broad involvement at the outset.
> Both S4 and Flume started out with at least 50% of their initial
> committers from a single company, and no single company constitutes a
> majority of the initial committers to Crunch (Cloudera has three,
> TomTom has two, WibiData has one, and Hortonworks has one). That de
> jure diversity mirrors the de facto diversity in Crunch's commit logs
> over the past several months:
> There is nothing more important than increasing that de facto
> diversity over time. I fully expect that my role during the incubator
> process is to be the best documenter, repository maintainer, and
> recruiter of new contributors that I can be.

It's not clear that Flume has widened its developer base significantly
enough for it to graduate. I fear that Crunch is exposed to the same risks,
and the fact that you are opting to exclude Jakob from the initial dev team
concerns me.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message