kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cmcc...@apache.org
Subject [kafka] branch trunk updated: MINOR. implement --expose-ports option in ducker-ak (#7269)
Date Mon, 09 Sep 2019 14:57:51 GMT
This is an automated email from the ASF dual-hosted git repository.

cmccabe pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git


The following commit(s) were added to refs/heads/trunk by this push:
     new 312e4db  MINOR. implement --expose-ports option in ducker-ak (#7269)
312e4db is described below

commit 312e4db5902a2d18d7e1df4932608fe789cec6d0
Author: Vikas Singh <50422828+soondenana@users.noreply.github.com>
AuthorDate: Mon Sep 9 07:57:29 2019 -0700

    MINOR. implement --expose-ports option in ducker-ak (#7269)
    
    This change adds a command line option to the `ducker-ak up' command to enable exposing
ports from docker containers. The exposed ports will be mapped to the ephemeral ports on the
host. The option is called `expose-ports' and can take either a single value (like 5005) or
a range (like 5005-5009). This port will then exposed from each docker container that ducker-ak
sets up.
    
    Reviewers: Colin P. McCabe <cmccabe@apache.org>, José Armando García Sancio <jsancio@users.noreply.github.com>
---
 tests/README.md        | 35 +++++++++++++++++++++++++++++++++++
 tests/docker/ducker-ak | 21 ++++++++++++++++++---
 2 files changed, 53 insertions(+), 3 deletions(-)

diff --git a/tests/README.md b/tests/README.md
index cf355e5..2fce5c3 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -47,6 +47,41 @@ bash tests/docker/ducker-ak up -j 'openjdk:11'; tests/docker/run_tests.sh
   - The docker containers are named knode01, knode02 etc.
    These nodes can't be used for any other purpose.
 
+* Exposing ports using --expose-ports option of `ducker-ak up` command
+
+    If `--expose-ports` is specified then we will expose those ports to random ephemeral
ports
+    on the host. The argument can be a single port (like 5005), a port range like (5005-5009)
+    or a combination of port/port-range separated by comma (like 2181,9092 or 2181,5005-5008).
+    By default no port is exposed.
+    
+    The exposed port mapping can be seen by executing `docker ps` command. The PORT column
+    of the output shows the mapping like this (maps port 33891 on host to port 2182 in container):
+
+    0.0.0.0:33891->2182/tcp
+
+    Behind the scene Docker is setting up a DNAT rule for the mapping and it is visible in
+    the DOCKER section of iptables command (`sudo iptables -t nat -L -n`), something like:
+
+    <pre>DNAT       tcp  --  0.0.0.0/0      0.0.0.0/0      tcp       dpt:33882    
  to:172.22.0.2:9092</pre>
+
+    The exposed port(s) are useful to attach a remote debugger to the process running
+    in the docker image. For example if port 5005 was exposed and is mapped to an ephemeral
+    port (say 33891), then a debugger attaching to port 33891 on host will be connecting
to
+    a debug session started at port 5005 in the docker image. As an example, for above port
+    numbers, run following commands in the docker image (say by ssh using `./docker/ducker-ak
ssh ducker02`):
+
+    > $ export KAFKA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005"
+    
+    > $ /opt/kafka-dev/bin/kafka-topics.sh --bootstrap-server ducker03:9095 --topic __consumer_offsets
--describe
+
+    This will run the TopicCommand to describe the __consumer-offset topic. The java process
+    will stop and wait for debugger to attach as `suspend=y` option was specified. Now starting
+    a debugger on host with host `localhost` and following parameter as JVM setting:
+
+    `-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=33891`
+
+    will attach it to the TopicCommand process running in the docker image.
+
 Examining CI run
 ----------------
 * Set BUILD_ID is travis ci's build id. E.g. build id is 169519874 for the following build
diff --git a/tests/docker/ducker-ak b/tests/docker/ducker-ak
index 2a4e5f6..a29395e 100755
--- a/tests/docker/ducker-ak
+++ b/tests/docker/ducker-ak
@@ -61,7 +61,7 @@ help|-h|--help
     Display this help message
 
 up [-n|--num-nodes NUM_NODES] [-f|--force] [docker-image]
-        [-C|--custom-ducktape DIR]
+        [-C|--custom-ducktape DIR] [-e|--expose-ports ports]
     Bring up a cluster with the specified amount of nodes (defaults to ${default_num_nodes}).
     The docker image name defaults to ${default_image_name}.  If --force is specified, we
will
     attempt to bring up an image even some parameters are not valid.
@@ -70,6 +70,11 @@ up [-n|--num-nodes NUM_NODES] [-f|--force] [docker-image]
     ducktape source code directory before bringing up the nodes.  The provided
     directory should be the ducktape git repo, not the ducktape installed module directory.
 
+    if --expose-ports is specified then we will expose those ports to random ephemeral ports
+    on the host. The argument can be a single port (like 5005), a port range like (5005-5009)
+    or a combination of port/port-range separated by comma (like 2181,9092 or 2181,5005-5008).
+    By default no port is exposed. See README.md for more detail on this option.
+
 test [test-name(s)]
     Run a test or set of tests inside the currently active Ducker nodes.
     For example, to run the system test produce_bench_test, you would run:
@@ -235,12 +240,21 @@ $((${duration} % 60))s.  See ${build_log} for details."
 docker_run() {
     local node=${1}
     local image_name=${2}
+    local ports_option=${3}
+
+    local expose_ports=""
+    if [[ -n ${ports_option} ]]; then
+        expose_ports="-P"
+        for expose_port in ${ports_option//,/ }; do
+            expose_ports="${expose_ports} --expose ${expose_port}"
+        done
+    fi
 
     # Invoke docker-run. We need privileged mode to be able to run iptables
     # and mount FUSE filesystems inside the container.  We also need it to
     # run iptables inside the container.
     must_do -v docker run --privileged \
-        -d -t -h "${node}" --network ducknet \
+        -d -t -h "${node}" --network ducknet "${expose_ports}" \
         --memory=${docker_run_memory_limit} --memory-swappiness=1 \
         -v "${kafka_dir}:/opt/kafka-dev" --name "${node}" -- "${image_name}"
 }
@@ -269,6 +283,7 @@ ducker_up() {
             -f|--force) force=1; shift;;
             -n|--num-nodes) set_once num_nodes "${2}" "number of nodes"; shift 2;;
             -j|--jdk) set_once jdk_version "${2}" "the OpenJDK base image"; shift 2;;
+            -e|--expose-ports) set_once expose_ports "${2}" "the ports to expose"; shift
2;;
             *) set_once image_name "${1}" "docker image name"; shift;;
         esac
     done
@@ -322,7 +337,7 @@ attempting to start new ones."
     fi
     for n in $(seq -f %02g 1 ${num_nodes}); do
         local node="ducker${n}"
-        docker_run "${node}" "${image_name}"
+        docker_run "${node}" "${image_name}" "${expose_ports}"
     done
     mkdir -p "${ducker_dir}/build"
     exec 3<> "${ducker_dir}/build/node_hosts"


Mime
View raw message