incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Drob <>
Subject Re: [DISCUSS] Dr. Elephant Incubator Proposal
Date Wed, 07 Mar 2018 00:05:36 GMT
Thanks for explaining your thoughts, Carl! Those answers all sound great to

Best of luck!

On Tue, Mar 6, 2018, 5:58 PM Carl Steinbach <> wrote:

> Hi Mike,
> Why does Dr. Elephant make sense as a separate project instead of
> > contributing to Hadoop directly?
> >
> Here are a couple reasons why I think Dr. Elephant is more likely to
> succeed as a separate project:
> * Dr. Elephant supports Hadoop *and* Spark, and may support other
>   execution layers in the future. If we make Dr. Elephant a part of
>   Hadoop I expect that it will discourage contributions from people
>   who are interested mainly in Spark support, and vice versa.
> * If Dr. Elephant is added to Hadoop it will be necessary for the
>   Hadoop project to declare a dependency on Spark. I doubt this change
>   will get approved.
> * We don't want to tie Dr. Elephant to a specific version of Hadoop or
>   Spark, or tie the Dr. Elephant release cycle to the Hadoop or Spark
>   release cycles.
> * None of the current Dr. Elephant committers are Hadoop committers,
>   and I doubt that the Hadoop PMC is going to give them a commit bit
>   just to work on Dr. Elephant. As a result the existing committers
>   would be effectively forfeiting their right to continue maintaining
>   their own project. I think this is one of the reasons why many
>   Hadoop contrib projects are poorly maintained.
> > What is the relationship between Dr. Elephant and the (now seemingly
> > defunct) Hadoop Vaidya?
> >
> Vaidya was a command line tool for tuning Hadoop jobs. Dr. Elephant is
> an always-on service for tuning Hadoop and Spark jobs. We were unaware
> of Vaidya when we started working on Dr. Elephant.
> - Carl

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message