incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joseph Schaefer <>
Subject Re: [VOTE] Release Apache Spark 0.8.0-incubating (rc4)
Date Sun, 15 Dec 2013 06:07:49 GMT
The core issue is that what you are currently doing was actually the recommended way of dealing
with external dependencies back when maven central didn't exist.

We changed course a few years ago based on Roy's complaints about source packages consisting
solely of source code.  I don't remember precisely when nor do I remember any formal process
vote indicating a change which in retrospect might have helped lessen the confusion but it
was a foundation wide problem not just an incubator issue.

In this situation I'd like to say go ahead and complete the release and just fix this prior
to graduation.

Sent from my iPhone

> On Dec 15, 2013, at 12:24 AM, Patrick Wendell <> wrote:
> Hey Marvin,
> Is this policy actually written up anywhere (along with best practices
> on how to deal with this issue if you indeed have third party
> dependencies)? I'm just asking because I don't see an obvious "fix"
> for this based on the way Spark is built.
> Second - this issue was not brought to our attention before - and in
> particular was not raised during our 0.8.0 major release through the
> incubator. Since this (0.8.1) release is a maintenance release, doing
> a large change to the build system is not possible. It seems to me a
> bit much to ask us to completely re-tool this entire project in order
> for a simple maintenance release to pass. Especially since other top
> level projects are clearly still employing this practice (not that
> they are in-the-right, but just that this is a policy which it seems
> is still being shaped).
> We are planning to do our next major release (Spark 0.9.0) while still
> under incubation in the next few weeks. Could I propose that we create
> a parallel discussion about how we might re-tool or build process with
> the aim towards satisfying whatever policy exists in that release?
> We'll probably need guidance on that from people at Apache since,
> again, there is no documented guidelines about what is allowed and
> what isn't.
> - Patrick
>> On Sat, Dec 14, 2013 at 8:20 PM, Marvin Humphrey <> wrote:
>> On Sat, Dec 14, 2013 at 3:43 PM, Patrick Wendell <> wrote:
>>>> However they both contain binaries, which is not good.
>>>> Third party jars should *not* be included in SCM nor in source releases.
>>> These are not binary artifacts containing our project's code. They are
>>> our build tool and immediate dependencies that are not published in
>>> maven. I've looked around to find TL projects that also use sbt and
>>> they also include the sbt jar in the source release. For instance
>>> Apache Kafka does the same thing:
>> I appreciate your doing the research, and I understand why you might think
>> following Kafka's example is a reasonable approach.  However, that binary is a
>> problem for Kafka.  If Kafka's releases were like that when they graduated,
>> it's a failure of the Incubator as well.
>> Please read these messages from ASF Board member Roy Fielding:
>> This has to be fixed.  If some TLP PMCs have not been made aware that binary
>> dependencies may not be bundled in source releases, the Incubator must not
>> compound the problem by failing to educate current podlings.
>> Marvin Humphrey
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> For additional commands, e-mail:
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message