incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carl Steinbach <>
Subject Re: [DISCUSS] Dr. Elephant Incubator Proposal
Date Tue, 06 Mar 2018 23:58:16 GMT
Hi Mike,

Why does Dr. Elephant make sense as a separate project instead of
> contributing to Hadoop directly?

Here are a couple reasons why I think Dr. Elephant is more likely to
succeed as a separate project:

* Dr. Elephant supports Hadoop *and* Spark, and may support other
  execution layers in the future. If we make Dr. Elephant a part of
  Hadoop I expect that it will discourage contributions from people
  who are interested mainly in Spark support, and vice versa.

* If Dr. Elephant is added to Hadoop it will be necessary for the
  Hadoop project to declare a dependency on Spark. I doubt this change
  will get approved.

* We don't want to tie Dr. Elephant to a specific version of Hadoop or
  Spark, or tie the Dr. Elephant release cycle to the Hadoop or Spark
  release cycles.

* None of the current Dr. Elephant committers are Hadoop committers,
  and I doubt that the Hadoop PMC is going to give them a commit bit
  just to work on Dr. Elephant. As a result the existing committers
  would be effectively forfeiting their right to continue maintaining
  their own project. I think this is one of the reasons why many
  Hadoop contrib projects are poorly maintained.

> What is the relationship between Dr. Elephant and the (now seemingly
> defunct) Hadoop Vaidya?

Vaidya was a command line tool for tuning Hadoop jobs. Dr. Elephant is
an always-on service for tuning Hadoop and Spark jobs. We were unaware
of Vaidya when we started working on Dr. Elephant.

- Carl

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message