kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ewe...@apache.org
Subject kafka git commit: KAFKA-4465: Create docker image and scripts for running tests locally
Date Mon, 09 Jan 2017 00:12:50 GMT
Repository: kafka
Updated Branches:
  refs/heads/trunk 10b330cf0 -> fa80093c0


KAFKA-4465: Create docker image and scripts for running tests locally

Author: Raghav Kumar Gautam <raghav@apache.org>

Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <harsha@hortonworks.com>,
Ewen Cheslack-Postava <ewen@confluent.io>

Closes #2197 from raghavgautam/trunk


Project: http://git-wip-us.apache.org/repos/asf/kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/fa80093c
Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/fa80093c
Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/fa80093c

Branch: refs/heads/trunk
Commit: fa80093c067e6684e005bfd548770541b5b2778b
Parents: 10b330c
Author: Raghav Kumar Gautam <raghav@apache.org>
Authored: Sun Jan 8 16:14:16 2017 -0800
Committer: Ewen Cheslack-Postava <me@ewencp.org>
Committed: Sun Jan 8 16:15:05 2017 -0800

----------------------------------------------------------------------
 build.gradle                     |  4 +-
 tests/.gitignore                 |  1 +
 tests/README.md                  | 25 ++++++++++++
 tests/cluster_file_generator.sh  | 63 ++++++++++++++++++++++++++++++
 tests/docker/Dockerfile          | 36 ++++++++++++++++++
 tests/docker/run_tests.sh        | 72 +++++++++++++++++++++++++++++++++++
 tests/docker/ssh/authorized_keys | 15 ++++++++
 tests/docker/ssh/config          | 21 ++++++++++
 tests/docker/ssh/id_rsa          | 27 +++++++++++++
 tests/docker/ssh/id_rsa.pub      |  1 +
 10 files changed, 264 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/build.gradle
----------------------------------------------------------------------
diff --git a/build.gradle b/build.gradle
index 249cb2a..87ffb08 100644
--- a/build.gradle
+++ b/build.gradle
@@ -108,7 +108,9 @@ if (new File('.git').exists()) {
         'CONTRIBUTING.md',
         'gradlew',
         'gradlew.bat',
-        '**/README.md'
+        '**/README.md',
+        '**/id_rsa',
+        '**/id_rsa.pub'
     ])
   }
 }

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/.gitignore
----------------------------------------------------------------------
diff --git a/tests/.gitignore b/tests/.gitignore
index b218b83..402057a 100644
--- a/tests/.gitignore
+++ b/tests/.gitignore
@@ -9,3 +9,4 @@ Vagrantfile.local
 
 .ducktape
 results/
+*.json

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/README.md
----------------------------------------------------------------------
diff --git a/tests/README.md b/tests/README.md
index 098922f..87a6fed 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -6,6 +6,31 @@ This directory contains Kafka system integration and performance tests.
 (ducktape is a distributed testing framework which provides test runner,
 result reporter and utilities to pull up and tear down services.)
 
+Running tests using docker
+--------------------------
+Docker containers can be used for running kafka system tests locally.
+* Requirements
+  - Docker 1.12.3 is installed and running on the machine.
+  - Test require a single kafka_*SNAPSHOT.tgz to be present in core/build/distributions.
+   This can be done by running ./gradlew clean releaseTarGz  
+* Run all tests
+```
+bash tests/docker/run_tests.sh
+```
+* Run all tests with debug on (warning will produce log of logs)
+```
+_DUCKTAPE_OPTIONS="--debug" bash tests/docker/run_tests.sh | tee debug_logs.txt
+```
+* Run a subset of tests
+```
+TC_PATHS="tests/kafkatest/tests/streams tests/kafkatest/tests/tools" bash tests/docker/run_tests.sh
+```
+* Notes
+  - The scripts to run tests creates and destroys docker network named *knw*.
+   This network can't be used for any other purpose.
+  - The docker containers are named knode01, knode02 etc.
+   These nodes can't be used for any other purpose.
+
 Local Quickstart
 ----------------
 This quickstart will help you run the Kafka system tests on your local machine. Note this
requires bringing up a cluster of virtual machines on your local computer, which is memory
intensive; it currently requires around 10G RAM.

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/cluster_file_generator.sh
----------------------------------------------------------------------
diff --git a/tests/cluster_file_generator.sh b/tests/cluster_file_generator.sh
new file mode 100644
index 0000000..2530a28
--- /dev/null
+++ b/tests/cluster_file_generator.sh
@@ -0,0 +1,63 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cat<<EOF
+{
+  "_comment": [
+    "Licensed to the Apache Software Foundation (ASF) under one or more",
+    "contributor license agreements.  See the NOTICE file distributed with",
+    "this work for additional information regarding copyright ownership.",
+    "The ASF licenses this file to You under the Apache License, Version 2.0",
+    "(the \"License\"); you may not use this file except in compliance with",
+    "the License.  You may obtain a copy of the License at",
+    "",
+    "http://www.apache.org/licenses/LICENSE-2.0",
+    "",
+    "Unless required by applicable law or agreed to in writing, software",
+    "distributed under the License is distributed on an \"AS IS\" BASIS,",
+    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.",
+    "See the License for the specific language governing permissions and",
+    "limitations under the License."
+  ],
+  "nodes": [
+EOF
+
+for K in $(seq -w 02 ${KAFKA_NUM_CONTAINERS}); do
+  KNODE="knode${K}"
+  if [ $K -eq 12 ]; then
+    SUFFIX=""
+  else
+    SUFFIX=","
+  fi
+  cat<<EOF
+    {
+      "externally_routable_ip": "$KNODE",
+      "ssh_config": {
+        "host": "$KNODE",
+        "hostname": "$KNODE",
+        "identityfile": "/root/.ssh/id_rsa",
+        "password": "",
+        "port": 22,
+        "user": "root"
+      }
+    }$SUFFIX
+EOF
+done
+
+cat<<EOF
+  ]
+}
+EOF
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/docker/Dockerfile
----------------------------------------------------------------------
diff --git a/tests/docker/Dockerfile b/tests/docker/Dockerfile
new file mode 100644
index 0000000..bdd6af6
--- /dev/null
+++ b/tests/docker/Dockerfile
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM openjdk:8
+
+MAINTAINER Apache Kafka dev@kafka.apache.org
+RUN apt update
+RUN apt install -y unzip wget curl jq coreutils openssh-server net-tools vim openjdk-8-jdk
python-pip python-dev libffi-dev libssl-dev
+RUN pip install -U pip
+RUN pip install --upgrade cffi
+RUN pip install ducktape==0.6.0
+
+VOLUME ["/kafka_src"]
+
+ENV MIRROR="http://apache.cs.utah.edu/"
+RUN wget -q "${MIRROR}kafka/0.8.2.2/kafka_2.10-0.8.2.2.tgz" -O "/tmp/kafka_2.10-0.8.2.2.tgz"
&& tar xfz /tmp/kafka_2.10-0.8.2.2.tgz -C /opt && mv "/opt/kafka_2.10-0.8.2.2"
"/opt/kafka-0.8.2.2"
+RUN wget -q "${MIRROR}kafka/0.9.0.1/kafka_2.10-0.9.0.1.tgz" -O "/tmp/kafka_2.10-0.9.0.1.tgz"
&& tar xfz /tmp/kafka_2.10-0.9.0.1.tgz -C /opt && mv "/opt/kafka_2.10-0.9.0.1"
"/opt/kafka-0.9.0.1"
+RUN wget -q "${MIRROR}kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz" -O "/tmp/kafka_2.10-0.10.0.1.tgz"
&& tar xfz /tmp/kafka_2.10-0.10.0.1.tgz -C /opt && mv "/opt/kafka_2.10-0.10.0.1"
"/opt/kafka-0.10.0.1"
+
+RUN rm /tmp/kafka_*.tgz
+ADD ssh /root/.ssh
+RUN chmod 600 /root/.ssh/id_rsa
+
+CMD service ssh start && tail -f /dev/null

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/docker/run_tests.sh
----------------------------------------------------------------------
diff --git a/tests/docker/run_tests.sh b/tests/docker/run_tests.sh
new file mode 100755
index 0000000..dadb031
--- /dev/null
+++ b/tests/docker/run_tests.sh
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# To run tests use a command like:
+#   TC_PATHS="tests/kafkatest/tests/streams tests/kafkatest/tests/tools" bash tests/docker/run_tests.sh
+set -x
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+TESTS_DIR=`dirname ${SCRIPT_DIR}`
+KAFKA_SRC=`dirname ${TESTS_DIR}`
+KAFKA_VERSION=$(grep "version=.*" ${KAFKA_SRC}/gradle.properties | cut -f 2 -d =)
+JDK_INFO="openjdk8"
+KAFKA_IMAGE=${KAFKA_IMAGE:-kafkadev/kafka-image:${KAFKA_VERSION}}_${JDK_INFO}
+export KAFKA_NUM_CONTAINERS=12
+
+chmod 600 ${SCRIPT_DIR}/ssh/id_rsa
+cd ${KAFKA_SRC}
+(( $(ls -1 core/build/distributions/kafka_*SNAPSHOT.tgz | wc -l) != 1 )) && echo
'Expecting a single file like core/build/distributions/kafka_*SNAPSHOT.tgz, found:' $(ls -1
core/build/distributions/kafka_*SNAPSHOT.tgz) && echo "Did you run ./gradlew clean
releaseTarGz ?" && exit 1
+
+docker network rm knw
+docker network create knw
+
+docker kill $(docker ps -f=network=knw -q)
+docker rm $(docker ps -a -f=network=knw -q)
+
+docker run --rm -it ${KAFKA_IMAGE} "true"
+if [[ $? != 0 || ${KAFKA_IMAGE_REBUILD} != "" ]]; then
+    echo "kafka image ${KAFKA_IMAGE} does not exist. Building it from scratch."
+    COMMIT_INFO=$(git describe HEAD)
+    docker build -t ${KAFKA_IMAGE} --label=commit_info=${COMMIT_INFO} ${SCRIPT_DIR}
+fi
+
+echo "Using kafka image: ${KAFKA_IMAGE}"
+docker inspect ${KAFKA_IMAGE}
+for i in $(seq -w 1 ${KAFKA_NUM_CONTAINERS}); do
+  docker run -d -t --name knode${i} --network knw -v ${KAFKA_SRC}:/kafka_src ${KAFKA_IMAGE}
+done
+
+docker info
+docker ps
+docker network inspect knw
+
+for i in $(seq -w 1 ${KAFKA_NUM_CONTAINERS}); do
+  echo knode${i}
+  docker exec knode${i} bash -c "(tar xfz /kafka_src/core/build/distributions/kafka_*SNAPSHOT.tgz
-C /opt || echo missing kafka tgz did you build kafka tarball) && mv /opt/kafka*SNAPSHOT
/opt/kafka-trunk && ls -l /opt"
+  docker exec knode01 bash -c "ssh knode$i hostname"
+done
+
+# hack to copy test dependencies
+# this is required for running MiniKDC
+(cd ${KAFKA_SRC} && ./gradlew copyDependantTestLibs)
+for i in $(seq -w 1 ${KAFKA_NUM_CONTAINERS}); do
+  echo knode${i}
+  docker exec knode${i} bash -c "cp /kafka_src/core/build/dependant-testlibs/* /opt/kafka-trunk/libs/"
+  docker exec knode01 bash -c "ssh knode$i hostname"
+done
+
+bash tests/cluster_file_generator.sh > tests/cluster_file.json
+docker exec knode01 bash -c "cd /kafka_src; ducktape ${_DUCKTAPE_OPTIONS} --cluster-file
tests/cluster_file.json ${TC_PATHS:-tests/kafkatest/tests}"

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/docker/ssh/authorized_keys
----------------------------------------------------------------------
diff --git a/tests/docker/ssh/authorized_keys b/tests/docker/ssh/authorized_keys
new file mode 100644
index 0000000..9f9da1f
--- /dev/null
+++ b/tests/docker/ssh/authorized_keys
@@ -0,0 +1,15 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0qDT9kEPWc8JQ53b4KnT/ZJOLwb+3c//jpLW/2ofjDyIsPW4FohLpicfouch/zsRpN4G38lua+2BsGls9sMIZc6PXY2L+NIGCkqEMdCoU1Ym8SMtyJklfzp3m/0PeK9s2dLlR3PFRYvyFA4btQK5hkbYDNZPzf4airvzdRzLkrFf81+RemaMI2EtONwJRcbLViPaTXVKJdbFwJTJ1u7yu9wDYWHKBMA92mHTQeP6bhVYCqxJn3to/RfZYd+sHw6mfxVg5OrAlUOYpSV4pDNCAsIHdtZ56V8NQlJL6NJ2vzzSSYUwLMqe88fhrC8yYHoxC07QPy1EdkSTHdohAicyT
root@knode01.knw

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/docker/ssh/config
----------------------------------------------------------------------
diff --git a/tests/docker/ssh/config b/tests/docker/ssh/config
new file mode 100644
index 0000000..1f87417
--- /dev/null
+++ b/tests/docker/ssh/config
@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+Host *
+  ControlMaster auto
+  ControlPath ~/.ssh/master-%r@%h:%p
+  StrictHostKeyChecking no
+  ConnectTimeout=10
+  IdentityFile ~/.ssh/id_rsa

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/docker/ssh/id_rsa
----------------------------------------------------------------------
diff --git a/tests/docker/ssh/id_rsa b/tests/docker/ssh/id_rsa
new file mode 100644
index 0000000..276e07b
--- /dev/null
+++ b/tests/docker/ssh/id_rsa
@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEpQIBAAKCAQEAtKg0/ZBD1nPCUOd2+Cp0/2STi8G/t3P/46S1v9qH4w8iLD1u
+BaIS6YnH6LnIf87EaTeBt/JbmvtgbBpbPbDCGXOj12Ni/jSBgpKhDHQqFNWJvEjL
+ciZJX86d5v9D3ivbNnS5UdzxUWL8hQOG7UCuYZG2AzWT83+Goq783Ucy5KxX/Nfk
+XpmjCNhLTjcCUXGy1Yj2k11SiXWxcCUydbu8rvcA2FhygTAPdph00Hj+m4VWAqsS
+Z97aP0X2WHfrB8Opn8VYOTqwJVDmKUleKQzQgLCB3bWeelfDUJSS+jSdr880kmFM
+CzKnvPH4awvMmB6MQtO0D8tRHZEkx3aIQInMkwIDAQABAoIBAQCz6EMFNNLp0NP1
+X9yRXS6wW4e4CRWUazesiw3YZpcmnp6IchCMGZA99FEZyVILPW1J3tYWyotBdw7Z
++RFeCRXy5L+IMtiVkNJcpwss7M4ve0w0LkY0gj5V49xJ+3Gp4gDnZSxcguvrAem5
+yP5obR572fDpl0SknB4HCr6U2l+rauzrLyevy5eeDT/vmXbuM1cdHpNIXmmElz4L
+t31n+exQRn6tP1h516iXbcYbopxDgdv2qKGAqzWKE6TyWpzF5x7kjOEYt0bZ5QO3
+Lwh7AAqE/3mwxlYwng1L4WAT7RtcP19W+9JDIc7ENInMGxq6q46p1S3IPZsf1cj/
+aAJ9q3LBAoGBAOVJr0+WkR786n3BuswpGQWBgVxfai4y9Lf90vuGKawdQUzXv0/c
+EB/CFqP/dIsquukA8PfzjNMyTNmEHXi4Sf16H8Rg4EGhIYMEqIQojx1t/yLLm0aU
+YPEvW/02Umtlg3pJw9fQAAzFVqCasw2E2lUdAUkydGRwDUJZmv2/b3NzAoGBAMm0
+Jo7Et7ochH8Vku6uA+hG+RdwlKFm5JA7/Ci3DOdQ1zmJNrvBBFQLo7AjA4iSCoBd
+s9+y0nrSPcF4pM3l6ghLheaqbnIi2HqIMH9mjDbrOZiWvbnjvjpOketgNX8vV3Ye
+GUkSjoNcmvRmdsICmUjeML8bGOmq4zF9W/GIfTphAoGBAKGRo8R8f/SLGh3VtvCI
+gUY89NAHuEWnyIQii1qMNq8+yjYAzaHTm1UVqmiT6SbrzFvGOwcuCu0Dw91+2Fmp
+2xGPzfTOoxf8GCY/0ROXlQmS6jc1rEw24Hzz92ldrwRYuyYf9q4Ltw1IvXtcp5F+
+LW/OiYpv0E66Gs3HYI0wKbP7AoGBAJMZWeFW37LQJ2TTJAQDToAwemq4xPxsoJX7
+2SsMTFHKKBwi0JLe8jwk/OxwrJwF/bieHZcvv8ao2zbkuDQcz6/a/D074C5G8V9z
+QQM4k1td8vQwQw91Yv782/gvgvRNX1iaHNCowtxURgGlVEirQoTc3eoRZfrLkMM/
+7DTa2JEhAoGACEu3zHJ1sgyeOEgLArUJXlQM30A/ulMrnCd4MEyIE+ReyWAUevUQ
+0lYdVNva0/W4C5e2lUOJL41jjIPLqI7tcFR2PZE6n0xTTkxNH5W2u1WpFeKjx+O3
+czv7Bt6wYyLHIMy1JEqAQ7pw1mtJ5s76UDvXUhciF+DU2pWYc6APKR0=
+-----END RSA PRIVATE KEY-----

http://git-wip-us.apache.org/repos/asf/kafka/blob/fa80093c/tests/docker/ssh/id_rsa.pub
----------------------------------------------------------------------
diff --git a/tests/docker/ssh/id_rsa.pub b/tests/docker/ssh/id_rsa.pub
new file mode 100644
index 0000000..76e8f5f
--- /dev/null
+++ b/tests/docker/ssh/id_rsa.pub
@@ -0,0 +1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0qDT9kEPWc8JQ53b4KnT/ZJOLwb+3c//jpLW/2ofjDyIsPW4FohLpicfouch/zsRpN4G38lua+2BsGls9sMIZc6PXY2L+NIGCkqEMdCoU1Ym8SMtyJklfzp3m/0PeK9s2dLlR3PFRYvyFA4btQK5hkbYDNZPzf4airvzdRzLkrFf81+RemaMI2EtONwJRcbLViPaTXVKJdbFwJTJ1u7yu9wDYWHKBMA92mHTQeP6bhVYCqxJn3to/RfZYd+sHw6mfxVg5OrAlUOYpSV4pDNCAsIHdtZ56V8NQlJL6NJ2vzzSSYUwLMqe88fhrC8yYHoxC07QPy1EdkSTHdohAicyT
root@knode01.knw


Mime
View raw message