kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jun...@apache.org
Subject svn commit: r1401875 [1/4] - in /incubator/kafka/branches/0.8/system_test: ./ migration_tool_testsuite/ migration_tool_testsuite/0.7/ migration_tool_testsuite/0.7/bin/ migration_tool_testsuite/0.7/lib/ migration_tool_testsuite/config/ migration_tool_te...
Date Wed, 24 Oct 2012 20:56:21 GMT
Author: junrao
Date: Wed Oct 24 20:56:16 2012
New Revision: 1401875

URL: http://svn.apache.org/viewvc?rev=1401875&view=rev
Log:
Add more test cases to System Test ; patched by John Fung; reviewed by Jun Rao; KAFKA-571

Added:
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh   (with props)
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/zookeeper-server-start.sh   (with props)
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-0.7.0.jar   (with props)
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar   (with props)
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/zkclient-0.1.jar   (with props)
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/__init__.py
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/cluster_config.json
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_consumer.properties
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_producer.properties
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/server.properties
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/zookeeper.properties
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/migration_tool_test.py
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/testcase_9001/
    incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/testcase_9001/testcase_9001_properties.json
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5002/
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5002/testcase_5002_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0151/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0151/testcase_0151_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0152/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0152/testcase_0152_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0153/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0153/testcase_0153_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0154/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0154/testcase_0154_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0155/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0155/testcase_0155_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0156/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0156/testcase_0156_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0157/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0157/testcase_0157_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0158/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0158/testcase_0158_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0201/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0201/testcase_0201_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0202/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0202/testcase_0202_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0203/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0203/testcase_0203_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0204/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0204/testcase_0204_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0205/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0205/testcase_0205_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0206/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0206/testcase_0206_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0207/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0207/testcase_0207_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0208/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0208/testcase_0208_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0251/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0251/testcase_0251_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0252/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0252/testcase_0252_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0253/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0253/testcase_0253_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0254/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0254/testcase_0254_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0255/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0255/testcase_0255_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0256/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0256/testcase_0256_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0257/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0257/testcase_0257_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0258/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0258/testcase_0258_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0301/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0301/testcase_0301_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0302/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0302/testcase_0302_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0303/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0303/testcase_0303_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0304/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0304/testcase_0304_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0305/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0305/testcase_0305_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0306/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0306/testcase_0306_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0307/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0307/testcase_0307_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0308/
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0308/testcase_0308_properties.json
    incubator/kafka/branches/0.8/system_test/testcase_to_run_all.json
Modified:
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/console_consumer.properties
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/consumer.properties
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/log4j.properties
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer.properties
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer_performance.properties
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/mirror_maker_test.py
    incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5001/testcase_5001_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/config/console_consumer.properties
    incubator/kafka/branches/0.8/system_test/replication_testsuite/config/consumer.properties
    incubator/kafka/branches/0.8/system_test/replication_testsuite/config/log4j.properties
    incubator/kafka/branches/0.8/system_test/replication_testsuite/config/producer.properties
    incubator/kafka/branches/0.8/system_test/replication_testsuite/config/producer_performance.properties
    incubator/kafka/branches/0.8/system_test/replication_testsuite/replica_basic_test.py
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0001/testcase_0001_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0002/testcase_0002_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0003/testcase_0003_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0004/testcase_0004_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0005/testcase_0005_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0006/testcase_0006_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0007/testcase_0007_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0008/testcase_0008_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0009/testcase_0009_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0010/testcase_0010_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0021/testcase_0021_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0022/testcase_0022_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0023/testcase_0023_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0101/testcase_0101_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0102/testcase_0102_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0103/testcase_0103_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0104/testcase_0104_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0105/testcase_0105_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0106/testcase_0106_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0107/testcase_0107_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0108/testcase_0108_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0109/testcase_0109_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0110/testcase_0110_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0111/testcase_0111_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0112/testcase_0112_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0113/testcase_0113_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0114/testcase_0114_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0115/testcase_0115_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0116/testcase_0116_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0117/testcase_0117_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0118/testcase_0118_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0121/testcase_0121_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0122/testcase_0122_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_0123/testcase_0123_properties.json
    incubator/kafka/branches/0.8/system_test/replication_testsuite/testcase_1/testcase_1_properties.json
    incubator/kafka/branches/0.8/system_test/system_test_runner.py
    incubator/kafka/branches/0.8/system_test/utils/kafka_system_test_utils.py
    incubator/kafka/branches/0.8/system_test/utils/replication_utils.py
    incubator/kafka/branches/0.8/system_test/utils/system_test_utils.py

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh Wed Oct 24 20:56:16 2012
@@ -0,0 +1,66 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+# 
+#    http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+if [ $# -lt 1 ];
+then
+  echo "USAGE: $0 classname [opts]"
+  exit 1
+fi
+
+base_dir=$(dirname $0)/../../../..
+kafka_07_lib_dir=$(dirname $0)/../lib
+
+# 0.8 - scala jars
+for file in $base_dir/project/boot/scala-2.8.0/lib/*.jar;
+do
+  CLASSPATH=$CLASSPATH:$file
+done
+
+# 0.7 - kafka-0.7.jar, zkclient-0.1.jar, kafka-perf-0.7.0.jar
+for file in ${kafka_07_lib_dir}/*.jar;
+do
+  CLASSPATH=$CLASSPATH:$file
+done
+
+# 0.8 - metrics jar
+for file in $base_dir/core/lib/metrics*.jar;
+do
+  CLASSPATH=$CLASSPATH:$file
+done
+
+# 0.8 - misc jars
+for file in $base_dir/core/lib_managed/scala_2.8.0/compile/*.jar;
+do
+  if [ ${file##*/} != "sbt-launch.jar" ]; then
+    CLASSPATH=$CLASSPATH:$file
+  fi
+done
+if [ -z "$KAFKA_JMX_OPTS" ]; then
+  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false "
+fi
+if [ -z "$KAFKA_OPTS" ]; then
+  KAFKA_OPTS="-Xmx512M -server  -Dlog4j.configuration=file:$base_dir/config/log4j.properties"
+fi
+if [  $JMX_PORT ]; then
+  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
+fi
+if [ -z "$JAVA_HOME" ]; then
+  JAVA="java"
+else
+  JAVA="$JAVA_HOME/bin/java"
+fi
+
+$JAVA $KAFKA_OPTS $KAFKA_JMX_OPTS -cp $CLASSPATH $@

Propchange: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/kafka-run-class.sh
------------------------------------------------------------------------------
    svn:executable = *

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/zookeeper-server-start.sh
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/zookeeper-server-start.sh?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/zookeeper-server-start.sh (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/zookeeper-server-start.sh Wed Oct 24 20:56:16 2012
@@ -0,0 +1,23 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+# 
+#    http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+if [ $# -ne 1 ];
+then
+	echo "USAGE: $0 zookeeper.properties"
+	exit 1
+fi
+
+$(dirname $0)/kafka-run-class.sh org.apache.zookeeper.server.quorum.QuorumPeerMain $@

Propchange: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/bin/zookeeper-server-start.sh
------------------------------------------------------------------------------
    svn:executable = *

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-0.7.0.jar
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-0.7.0.jar?rev=1401875&view=auto
==============================================================================
Binary file - no diff available.

Propchange: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-0.7.0.jar
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar?rev=1401875&view=auto
==============================================================================
Binary file - no diff available.

Propchange: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/kafka-perf-0.7.0.jar
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/zkclient-0.1.jar
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/zkclient-0.1.jar?rev=1401875&view=auto
==============================================================================
Binary file - no diff available.

Propchange: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/0.7/lib/zkclient-0.1.jar
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/__init__.py
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/__init__.py?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/__init__.py (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/__init__.py Wed Oct 24 20:56:16 2012
@@ -0,0 +1 @@
+ 

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/cluster_config.json
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/cluster_config.json?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/cluster_config.json (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/cluster_config.json Wed Oct 24 20:56:16 2012
@@ -0,0 +1,103 @@
+{
+    "cluster_config": [
+        {
+            "entity_id": "0",
+            "hostname": "localhost",
+            "role": "zookeeper",
+            "cluster_name":"source",
+            "kafka_home": "system_test/migration_tool_testsuite/0.7",
+            "java_home": "default",
+            "jmx_port": "9990"
+        },
+        {
+            "entity_id": "1",
+            "hostname": "localhost",
+            "role": "broker",
+            "cluster_name":"source",
+            "kafka_home": "system_test/migration_tool_testsuite/0.7",
+            "java_home": "default",
+            "jmx_port": "9991"
+        },
+        {
+            "entity_id": "2",
+            "hostname": "localhost",
+            "role": "broker",
+            "cluster_name":"source",
+            "kafka_home": "system_test/migration_tool_testsuite/0.7",
+            "java_home": "default",
+            "jmx_port": "9992"
+        },
+        {
+            "entity_id": "3",
+            "hostname": "localhost",
+            "role": "broker",
+            "cluster_name":"source",
+            "kafka_home": "system_test/migration_tool_testsuite/0.7",
+            "java_home": "default",
+            "jmx_port": "9993"
+        },
+        {
+            "entity_id": "4",
+            "hostname": "localhost",
+            "role": "broker",
+            "cluster_name":"target",
+            "kafka_home": "default",
+            "java_home": "default",
+            "jmx_port": "9994"
+        },
+        {
+            "entity_id": "5",
+            "hostname": "localhost",
+            "role": "broker",
+            "cluster_name":"target",
+            "kafka_home": "default",
+            "java_home": "default",
+            "jmx_port": "9995"
+        },
+        {
+            "entity_id": "6",
+            "hostname": "localhost",
+            "role": "broker",
+            "cluster_name":"target",
+            "kafka_home": "default",
+            "java_home": "default",
+            "jmx_port": "9996"
+        },
+        {
+            "entity_id": "7",
+            "hostname": "localhost",
+            "role": "producer_performance",
+            "cluster_name":"source",
+            "kafka_home": "system_test/migration_tool_testsuite/0.7",
+            "java_home": "/export/apps/jdk/JDK-1_6_0_27",
+            "jmx_port": "9997"
+        },
+        {
+            "entity_id": "8",
+            "hostname": "localhost",
+            "role": "console_consumer",
+            "cluster_name":"target",
+            "kafka_home": "default",
+            "java_home": "default",
+            "jmx_port": "9998"
+        },
+        {
+            "entity_id": "9",
+            "hostname": "localhost",
+            "role": "zookeeper",
+            "cluster_name":"target",
+            "kafka_home": "default",
+            "java_home": "default",
+            "jmx_port": "9999"
+        },
+        {
+            "entity_id": "10",
+            "hostname": "localhost",
+            "role": "migration_tool",
+            "cluster_name":"target",
+            "kafka_home": "default",
+            "java_home": "default",
+            "jmx_port": "9890"
+        }
+    ]
+}

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_consumer.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_consumer.properties?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_consumer.properties (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_consumer.properties Wed Oct 24 20:56:16 2012
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+# 
+#    http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# see kafka.consumer.ConsumerConfig for more details
+
+# zk connection string
+# comma separated host:port pairs, each corresponding to a zk
+# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
+zk.connect=127.0.0.1:2188
+
+# timeout in ms for connecting to zookeeper
+zk.connectiontimeout.ms=1000000
+
+#consumer group id
+groupid=test-consumer-group
+
+#consumer timeout
+#consumer.timeout.ms=5000

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_producer.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_producer.properties?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_producer.properties (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/migration_producer.properties Wed Oct 24 20:56:16 2012
@@ -0,0 +1,80 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# see kafka.producer.ProducerConfig for more details
+
+############################# Producer Basics #############################
+
+# need to set either broker.list or zk.connect
+
+# configure brokers statically
+# format: host1:port1,host2:port2 ...
+broker.list=localhost:9094,localhost:9095,localhost:9096
+
+# discover brokers from ZK
+#zk.connect=
+
+# zookeeper session timeout; default is 6000
+#zk.sessiontimeout.ms=
+
+# the max time that the client waits to establish a connection to zookeeper; default is 6000
+#zk.connectiontimeout.ms
+
+# name of the partitioner class for partitioning events; default partition spreads data randomly
+#partitioner.class=
+
+# specifies whether the messages are sent asynchronously (async) or synchronously (sync)
+producer.type=sync
+
+# specify the compression codec for all data generated: 0: no compression, 1: gzip
+compression.codec=0
+
+# message encoder
+serializer.class=kafka.serializer.DefaultEncoder
+
+# allow topic level compression
+#compressed.topics=
+
+# max message size; messages larger than that size are discarded; default is 1000000
+#max.message.size=
+
+
+############################# Async Producer #############################
+# maximum time, in milliseconds, for buffering data on the producer queue 
+#queue.time=
+
+# the maximum size of the blocking queue for buffering on the producer 
+#queue.size=
+
+# Timeout for event enqueue:
+# 0: events will be enqueued immediately or dropped if the queue is full
+# -ve: enqueue will block indefinitely if the queue is full
+# +ve: enqueue will block up to this many milliseconds if the queue is full
+#queue.enqueueTimeout.ms=
+
+# the number of messages batched at the producer 
+#batch.size=
+
+# the callback handler for one or multiple events 
+#callback.handler=
+
+# properties required to initialize the callback handler 
+#callback.handler.props=
+
+# the handler for events 
+#event.handler=
+
+# properties required to initialize the event handler 
+#event.handler.props=
+

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/server.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/server.properties?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/server.properties (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/server.properties Wed Oct 24 20:56:16 2012
@@ -0,0 +1,122 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+# 
+#    http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# see kafka.server.KafkaConfig for additional details and defaults
+
+############################# Server Basics #############################
+
+# The id of the broker. This must be set to a unique integer for each broker.
+brokerid=0
+
+# Hostname the broker will advertise to consumers. If not set, kafka will use the value returned
+# from InetAddress.getLocalHost().  If there are multiple interfaces getLocalHost
+# may not be what you want.
+#hostname=
+
+
+############################# Socket Server Settings #############################
+
+# The port the socket server listens on
+port=9091
+
+# The number of threads handling network requests
+network.threads=2
+ 
+# The number of threads doing disk I/O
+io.threads=2
+
+# The send buffer (SO_SNDBUF) used by the socket server
+socket.send.buffer=1048576
+
+# The receive buffer (SO_RCVBUF) used by the socket server
+socket.receive.buffer=1048576
+
+# The maximum size of a request that the socket server will accept (protection against OOM)
+max.socket.request.bytes=104857600
+
+
+############################# Log Basics #############################
+
+# The directory under which to store log files
+log.dir=/tmp/kafka_server_logs
+
+# The number of logical partitions per topic per server. More partitions allow greater parallelism
+# for consumption, but also mean more files.
+num.partitions=5
+
+# Overrides for for the default given by num.partitions on a per-topic basis
+#topic.partition.count.map=topic1:3, topic2:4
+
+############################# Log Flush Policy #############################
+
+# The following configurations control the flush of data to disk. This is the most
+# important performance knob in kafka.
+# There are a few important trade-offs here:
+#    1. Durability: Unflushed data is at greater risk of loss in the event of a crash.
+#    2. Latency: Data is not made available to consumers until it is flushed (which adds latency).
+#    3. Throughput: The flush is generally the most expensive operation. 
+# The settings below allow one to configure the flush policy to flush data after a period of time or
+# every N messages (or both). This can be done globally and overridden on a per-topic basis.
+
+# The number of messages to accept before forcing a flush of data to disk
+log.flush.interval=10000
+
+# The maximum amount of time a message can sit in a log before we force a flush
+log.default.flush.interval.ms=1000
+
+# Per-topic overrides for log.default.flush.interval.ms
+#topic.flush.intervals.ms=topic1:1000, topic2:3000
+
+# The interval (in ms) at which logs are checked to see if they need to be flushed to disk.
+log.default.flush.scheduler.interval.ms=1000
+
+############################# Log Retention Policy #############################
+
+# The following configurations control the disposal of log segments. The policy can
+# be set to delete segments after a period of time, or after a given size has accumulated.
+# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
+# from the end of the log.
+
+# The minimum age of a log file to be eligible for deletion
+log.retention.hours=168
+
+# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
+# segments don't drop below log.retention.size.
+#log.retention.size=1073741824
+
+# The maximum size of a log segment file. When this size is reached a new log segment will be created.
+#log.file.size=536870912
+#log.file.size=102400
+log.file.size=128
+
+# The interval at which log segments are checked to see if they can be deleted according 
+# to the retention policies
+log.cleanup.interval.mins=1
+
+############################# Zookeeper #############################
+
+# Enable connecting to zookeeper
+enable.zookeeper=true
+
+# Zk connection string (see zk docs for details).
+# This is a comma separated host:port pairs, each corresponding to a zk
+# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
+# You can also append an optional chroot string to the urls to specify the
+# root directory for all kafka znodes.
+zk.connect=localhost:2181
+
+# Timeout in ms for connecting to zookeeper
+zk.connectiontimeout.ms=1000000
+
+monitoring.period.secs=1

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/zookeeper.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/zookeeper.properties?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/zookeeper.properties (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/config/zookeeper.properties Wed Oct 24 20:56:16 2012
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+# 
+#    http://www.apache.org/licenses/LICENSE-2.0
+# 
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# the directory where the snapshot is stored.
+dataDir=/tmp/zookeeper
+# the port at which the clients will connect
+clientPort=2181
+# disable the per-ip limit on the number of connections since this is a non-production config
+maxClientCnxns=0

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/migration_tool_test.py
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/migration_tool_test.py?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/migration_tool_test.py (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/migration_tool_test.py Wed Oct 24 20:56:16 2012
@@ -0,0 +1,322 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#!/usr/bin/env python
+
+# ===================================
+# migration_tool_test.py
+# ===================================
+
+import inspect
+import logging
+import os
+import signal
+import subprocess
+import sys
+import time
+import traceback
+
+from   system_test_env    import SystemTestEnv
+sys.path.append(SystemTestEnv.SYSTEM_TEST_UTIL_DIR)
+
+from   setup_utils        import SetupUtils
+from   replication_utils  import ReplicationUtils
+import system_test_utils
+from   testcase_env       import TestcaseEnv
+
+# product specific: Kafka
+import kafka_system_test_utils
+import metrics
+
+class MigrationToolTest(ReplicationUtils, SetupUtils):
+
+    testModuleAbsPathName = os.path.realpath(__file__)
+    testSuiteAbsPathName  = os.path.abspath(os.path.dirname(testModuleAbsPathName))
+
+    def __init__(self, systemTestEnv):
+
+        # SystemTestEnv - provides cluster level environment settings
+        #     such as entity_id, hostname, kafka_home, java_home which
+        #     are available in a list of dictionary named 
+        #     "clusterEntityConfigDictList"
+        self.systemTestEnv = systemTestEnv
+
+        super(MigrationToolTest, self).__init__(self)
+
+        # dict to pass user-defined attributes to logger argument: "extra"
+        d = {'name_of_class': self.__class__.__name__}
+
+    def signal_handler(self, signal, frame):
+        self.log_message("Interrupt detected - User pressed Ctrl+c")
+
+        # perform the necessary cleanup here when user presses Ctrl+c and it may be product specific
+        self.log_message("stopping all entities - please wait ...")
+        kafka_system_test_utils.stop_all_remote_running_processes(self.systemTestEnv, self.testcaseEnv)
+        sys.exit(1) 
+
+    def runTest(self):
+
+        # ======================================================================
+        # get all testcase directories under this testsuite
+        # ======================================================================
+        testCasePathNameList = system_test_utils.get_dir_paths_with_prefix(
+            self.testSuiteAbsPathName, SystemTestEnv.SYSTEM_TEST_CASE_PREFIX)
+        testCasePathNameList.sort()
+
+        # =============================================================
+        # launch each testcase one by one: testcase_1, testcase_2, ...
+        # =============================================================
+        for testCasePathName in testCasePathNameList:
+   
+            skipThisTestCase = False
+
+            try: 
+                # ======================================================================
+                # A new instance of TestcaseEnv to keep track of this testcase's env vars
+                # and initialize some env vars as testCasePathName is available now
+                # ======================================================================
+                self.testcaseEnv = TestcaseEnv(self.systemTestEnv, self)
+                self.testcaseEnv.testSuiteBaseDir = self.testSuiteAbsPathName
+                self.testcaseEnv.initWithKnownTestCasePathName(testCasePathName)
+                self.testcaseEnv.testcaseArgumentsDict = self.testcaseEnv.testcaseNonEntityDataDict["testcase_args"]
+
+                # ======================================================================
+                # SKIP if this case is IN testcase_to_skip.json or NOT IN testcase_to_run.json
+                # ======================================================================
+                testcaseDirName = self.testcaseEnv.testcaseResultsDict["_test_case_name"]
+
+                if self.systemTestEnv.printTestDescriptionsOnly:
+                    self.testcaseEnv.printTestCaseDescription(testcaseDirName)
+                    continue
+                elif self.systemTestEnv.isTestCaseToSkip(self.__class__.__name__, testcaseDirName):
+                    self.log_message("Skipping : " + testcaseDirName)
+                    skipThisTestCase = True
+                    continue
+                else:
+                    self.testcaseEnv.printTestCaseDescription(testcaseDirName)
+                    system_test_utils.setup_remote_hosts_with_testcase_level_cluster_config(self.systemTestEnv, testCasePathName)
+
+                # ============================================================================== #
+                # ============================================================================== #
+                #                   Product Specific Testing Code Starts Here:                   #
+                # ============================================================================== #
+                # ============================================================================== #
+    
+                # initialize self.testcaseEnv with user-defined environment variables (product specific)
+                self.testcaseEnv.userDefinedEnvVarDict["zkConnectStr"] = ""
+                self.testcaseEnv.userDefinedEnvVarDict["stopBackgroundProducer"]    = False
+                self.testcaseEnv.userDefinedEnvVarDict["backgroundProducerStopped"] = False
+
+                # initialize signal handler
+                signal.signal(signal.SIGINT, self.signal_handler)
+    
+                # create "LOCAL" log directories for metrics, dashboards for each entity under this testcase
+                # for collecting logs from remote machines
+                kafka_system_test_utils.generate_testcase_log_dirs(self.systemTestEnv, self.testcaseEnv)
+    
+                # TestcaseEnv.testcaseConfigsList initialized by reading testcase properties file:
+                #   system_test/<suite_name>_testsuite/testcase_<n>/testcase_<n>_properties.json
+                self.testcaseEnv.testcaseConfigsList = system_test_utils.get_json_list_data(
+                    self.testcaseEnv.testcasePropJsonPathName)
+    
+                # TestcaseEnv - initialize producer & consumer config / log file pathnames
+                kafka_system_test_utils.init_entity_props(self.systemTestEnv, self.testcaseEnv)
+
+                # clean up data directories specified in zookeeper.properties and kafka_server_<n>.properties
+                kafka_system_test_utils.cleanup_data_at_remote_hosts(self.systemTestEnv, self.testcaseEnv)
+
+                # generate remote hosts log/config dirs if not exist
+                kafka_system_test_utils.generate_testcase_log_dirs_in_remote_hosts(self.systemTestEnv, self.testcaseEnv)
+    
+                # generate properties files for zookeeper, kafka, producer, consumer and mirror-maker:
+                # 1. copy system_test/<suite_name>_testsuite/config/*.properties to 
+                #    system_test/<suite_name>_testsuite/testcase_<n>/config/
+                # 2. update all properties files in system_test/<suite_name>_testsuite/testcase_<n>/config
+                #    by overriding the settings specified in:
+                #    system_test/<suite_name>_testsuite/testcase_<n>/testcase_<n>_properties.json
+                kafka_system_test_utils.generate_overriden_props_files(self.testSuiteAbsPathName,
+                    self.testcaseEnv, self.systemTestEnv)
+
+                # =============================================
+                # preparing all entities to start the test
+                # =============================================
+                self.log_message("starting zookeepers")
+                kafka_system_test_utils.start_zookeepers(self.systemTestEnv, self.testcaseEnv)
+                self.anonLogger.info("sleeping for 2s")
+                time.sleep(2)
+        
+                self.log_message("starting brokers")
+                kafka_system_test_utils.start_brokers(self.systemTestEnv, self.testcaseEnv)
+                self.anonLogger.info("sleeping for 5s")
+                time.sleep(5)
+
+                self.log_message("starting migration tool")
+                kafka_system_test_utils.start_migration_tool(self.systemTestEnv, self.testcaseEnv)
+                self.anonLogger.info("sleeping for 5s")
+                time.sleep(5)
+
+                self.log_message("creating topics")
+                kafka_system_test_utils.create_topic(self.systemTestEnv, self.testcaseEnv)
+                self.anonLogger.info("sleeping for 5s")
+                time.sleep(5)
+
+                # =============================================
+                # starting producer 
+                # =============================================
+                self.log_message("starting producer in the background")
+                kafka_system_test_utils.start_producer_performance(self.systemTestEnv, self.testcaseEnv, True)
+                msgProducingFreeTimeSec = self.testcaseEnv.testcaseArgumentsDict["message_producing_free_time_sec"]
+                self.anonLogger.info("sleeping for " + msgProducingFreeTimeSec + " sec to produce some messages")
+                time.sleep(int(msgProducingFreeTimeSec))
+
+                # =============================================
+                # A while-loop to bounce leader as specified
+                # by "num_iterations" in testcase_n_properties.json
+                # =============================================
+                i = 1
+                numIterations = int(self.testcaseEnv.testcaseArgumentsDict["num_iteration"])
+                while i <= numIterations:
+
+                    self.log_message("Iteration " + str(i) + " of " + str(numIterations))
+
+                    self.log_message("looking up leader")
+                    leaderDict = kafka_system_test_utils.get_leader_elected_log_line(
+                        self.systemTestEnv, self.testcaseEnv, self.leaderAttributesDict)
+        
+                    # ==========================
+                    # leaderDict looks like this:
+                    # ==========================
+                    #{'entity_id': u'3',
+                    # 'partition': '0',
+                    # 'timestamp': 1345050255.8280001,
+                    # 'hostname': u'localhost',
+                    # 'topic': 'test_1',
+                    # 'brokerid': '3'}
+
+                    # =============================================
+                    # validate to see if leader election is successful
+                    # =============================================
+                    self.log_message("validating leader election")
+                    result = kafka_system_test_utils.validate_leader_election_successful( 
+                         self.testcaseEnv, leaderDict, self.testcaseEnv.validationStatusDict)
+        
+                    # =============================================
+                    # trigger leader re-election by stopping leader
+                    # to get re-election latency
+                    # =============================================
+                    bounceLeaderFlag = self.testcaseEnv.testcaseArgumentsDict["bounce_leader"]
+                    self.log_message("bounce_leader flag : " + bounceLeaderFlag)
+                    if (bounceLeaderFlag.lower() == "true"):
+                        reelectionLatency = kafka_system_test_utils.get_reelection_latency(
+                            self.systemTestEnv, self.testcaseEnv, leaderDict, self.leaderAttributesDict)
+                        latencyKeyName = "Leader Election Latency - iter " + str(i) + " brokerid " + leaderDict["brokerid"]
+                        self.testcaseEnv.validationStatusDict[latencyKeyName] = str("{0:.2f}".format(reelectionLatency * 1000)) + " ms"
+       
+                    # =============================================
+                    # starting previously terminated broker 
+                    # =============================================
+                    if bounceLeaderFlag.lower() == "true":
+                        self.log_message("starting the previously terminated broker")
+                        stoppedLeaderEntityId  = leaderDict["entity_id"]
+                        kafka_system_test_utils.start_entity_in_background(self.systemTestEnv, self.testcaseEnv, stoppedLeaderEntityId)
+
+                    self.anonLogger.info("sleeping for 15s")
+                    time.sleep(15)
+                    i += 1
+                # while loop
+
+                # =============================================
+                # tell producer to stop
+                # =============================================
+                self.testcaseEnv.lock.acquire()
+                self.testcaseEnv.userDefinedEnvVarDict["stopBackgroundProducer"] = True
+                time.sleep(1)
+                self.testcaseEnv.lock.release()
+                time.sleep(1)
+
+                # =============================================
+                # wait for producer thread's update of
+                # "backgroundProducerStopped" to be "True"
+                # =============================================
+                while 1:
+                    self.testcaseEnv.lock.acquire()
+                    self.logger.info("status of backgroundProducerStopped : [" + \
+                        str(self.testcaseEnv.userDefinedEnvVarDict["backgroundProducerStopped"]) + "]", extra=self.d)
+                    if self.testcaseEnv.userDefinedEnvVarDict["backgroundProducerStopped"]:
+                        time.sleep(1)
+                        self.logger.info("all producer threads completed", extra=self.d)
+                        break
+                    time.sleep(1)
+                    self.testcaseEnv.lock.release()
+                    time.sleep(2)
+
+                #print "\n\n#### sleeping for 30 min ...\n\n"
+                #time.sleep(1800)
+                
+                # =============================================
+                # starting consumer
+                # =============================================
+                self.log_message("starting consumer in the background")
+                kafka_system_test_utils.start_console_consumer(self.systemTestEnv, self.testcaseEnv)
+                self.anonLogger.info("sleeping for 20s")
+                time.sleep(20)
+                    
+                # =============================================
+                # this testcase is completed - stop all entities
+                # =============================================
+                self.log_message("stopping all entities")
+                for entityId, parentPid in self.testcaseEnv.entityBrokerParentPidDict.items():
+                    kafka_system_test_utils.stop_remote_entity(self.systemTestEnv, entityId, parentPid)
+
+                for entityId, parentPid in self.testcaseEnv.entityZkParentPidDict.items():
+                    kafka_system_test_utils.stop_remote_entity(self.systemTestEnv, entityId, parentPid)
+
+                # make sure all entities are stopped
+                kafka_system_test_utils.ps_grep_terminate_running_entity(self.systemTestEnv)
+
+                # =============================================
+                # collect logs from remote hosts
+                # =============================================
+                kafka_system_test_utils.collect_logs_from_remote_hosts(self.systemTestEnv, self.testcaseEnv)
+    
+                # =============================================
+                # validate the data matched and checksum
+                # =============================================
+                self.log_message("validating data matched")
+                kafka_system_test_utils.validate_07_08_migrated_data_matched(self.systemTestEnv, self.testcaseEnv)
+                kafka_system_test_utils.validate_broker_log_segment_checksum(self.systemTestEnv, self.testcaseEnv)
+
+                # =============================================
+                # draw graphs
+                # =============================================
+                metrics.draw_all_graphs(self.systemTestEnv.METRICS_PATHNAME, 
+                                        self.testcaseEnv, 
+                                        self.systemTestEnv.clusterEntityConfigDictList)
+                
+                # build dashboard, one for each role
+                metrics.build_all_dashboards(self.systemTestEnv.METRICS_PATHNAME,
+                                             self.testcaseEnv.testCaseDashboardsDir,
+                                             self.systemTestEnv.clusterEntityConfigDictList)
+                
+            except Exception as e:
+                self.log_message("Exception while running test {0}".format(e))
+                traceback.print_exc()
+
+            finally:
+                if not skipThisTestCase and not self.systemTestEnv.printTestDescriptionsOnly:
+                    self.log_message("stopping all entities - please wait ...")
+                    kafka_system_test_utils.stop_all_remote_running_processes(self.systemTestEnv, self.testcaseEnv)
+

Added: incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/testcase_9001/testcase_9001_properties.json
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/testcase_9001/testcase_9001_properties.json?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/testcase_9001/testcase_9001_properties.json (added)
+++ incubator/kafka/branches/0.8/system_test/migration_tool_testsuite/testcase_9001/testcase_9001_properties.json Wed Oct 24 20:56:16 2012
@@ -0,0 +1,123 @@
+{
+  "description": {"01":"To Test : 'Replication with Migration Tool'",
+                  "02":"Set up 2 clusters such as : SOURCE => Migration Tool => TARGET",
+                  "03":"Produce and consume messages to a single topic - single partition.",
+                  "04":"This test sends messages to 3 replicas",
+                  "05":"At the end it verifies the log size and contents",
+                  "06":"Use a consumer to verify no message loss in TARGET cluster.",
+                  "07":"Producer dimensions : mode:sync, acks:-1, comp:0",
+                  "08":"Log segment size    : 51200"
+  },
+  "testcase_args": {
+    "bounce_leader": "false",
+    "replica_factor": "3",
+    "num_partition": "1",
+    "num_iteration": "1",
+    "sleep_seconds_between_producer_calls": "1",
+    "message_producing_free_time_sec": "30",
+    "num_messages_to_produce_per_producer_call": "50"
+  },
+  "entities": [
+    {
+      "entity_id": "0",
+      "clientPort": "2188",
+      "dataDir": "/tmp/zookeeper_0",
+      "log_filename": "zookeeper_0.log",
+      "config_filename": "zookeeper_0.properties"
+    },
+    {
+      "entity_id": "1",
+      "port": "9091",
+      "brokerid": "1",
+      "log.file.size": "51200",
+      "log.dir": "/tmp/kafka_server_1_logs",
+      "log_filename": "kafka_server_1.log",
+      "config_filename": "kafka_server_1.properties"
+    },
+    {
+      "entity_id": "2",
+      "port": "9092",
+      "brokerid": "2",
+      "log.file.size": "51200",
+      "log.dir": "/tmp/kafka_server_2_logs",
+      "log_filename": "kafka_server_2.log",
+      "config_filename": "kafka_server_2.properties"
+    },
+    {
+      "entity_id": "3",
+      "port": "9093",
+      "brokerid": "3",
+      "log.file.size": "51200",
+      "log.dir": "/tmp/kafka_server_3_logs",
+      "log_filename": "kafka_server_3.log",
+      "config_filename": "kafka_server_3.properties"
+    },
+    {
+      "entity_id": "4",
+      "port": "9094",
+      "brokerid": "4",
+      "log.file.size": "51200",
+      "log.dir": "/tmp/kafka_server_4_logs",
+      "log_filename": "kafka_server_4.log",
+      "config_filename": "kafka_server_4.properties"
+    },
+    {
+      "entity_id": "5",
+      "port": "9095",
+      "brokerid": "5",
+      "log.file.size": "51200",
+      "log.dir": "/tmp/kafka_server_5_logs",
+      "log_filename": "kafka_server_5.log",
+      "config_filename": "kafka_server_5.properties"
+    },
+    {
+      "entity_id": "6",
+      "port": "9096",
+      "brokerid": "6",
+      "log.file.size": "51200",
+      "log.dir": "/tmp/kafka_server_6_logs",
+      "log_filename": "kafka_server_6.log",
+      "config_filename": "kafka_server_6.properties"
+    },
+    {
+      "entity_id": "7",
+      "topic": "test_1",
+      "threads": "5",
+      "compression-codec": "0",
+      "message-size": "500",
+      "message": "500",
+      "request-num-acks": "-1",
+      "async": "false",
+      "log_filename": "producer_performance_7.log",
+      "config_filename": "producer_performance_7.properties"
+    },
+    {
+      "entity_id": "8",
+      "topic": "test_1",
+      "groupid": "mytestgroup",
+      "consumer-timeout-ms": "10000",
+      "log_filename": "console_consumer_8.log",
+      "formatter": "kafka.consumer.ChecksumMessageFormatter",
+      "config_filename": "console_consumer_8.properties"
+    },
+    {
+      "entity_id": "9",
+      "clientPort": "2191",
+      "dataDir": "/tmp/zookeeper_9",
+      "log_filename": "zookeeper_9.log",
+      "config_filename": "zookeeper_9.properties"
+    },
+    {
+      "entity_id": "10",
+      "whitelist": ".*",
+      "num.producers": "2",
+      "num.streams": "2",
+      "producer.config": "migration_tool_testsuite/config/migration_producer.properties",
+      "consumer.config": "migration_tool_testsuite/config/migration_consumer.properties",
+      "zkclient.01.jar": "migration_tool_testsuite/0.7/lib/zkclient-0.1.jar",
+      "kafka.07.jar"   : "migration_tool_testsuite/0.7/lib/kafka-0.7.0.jar",
+      "log_filename": "migration_tool.log",
+      "config_filename": "migration_tool.properties"
+    }
+   ]
+}

Modified: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/console_consumer.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/console_consumer.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/console_consumer.properties (original)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/console_consumer.properties Wed Oct 24 20:56:16 2012
@@ -1,4 +0,0 @@
-zookeeper=local:2181
-topic=test_1
-from-beginning
-consumer-timeout-ms=10000

Modified: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/consumer.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/consumer.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/consumer.properties (original)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/consumer.properties Wed Oct 24 20:56:16 2012
@@ -1,29 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-# 
-#    http://www.apache.org/licenses/LICENSE-2.0
-# 
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# see kafka.consumer.ConsumerConfig for more details
-
-# zk connection string
-# comma separated host:port pairs, each corresponding to a zk
-# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
-zk.connect=127.0.0.1:2181
-
-# timeout in ms for connecting to zookeeper
-zk.connectiontimeout.ms=1000000
-
-#consumer group id
-groupid=test-consumer-group
-
-#consumer timeout
-#consumer.timeout.ms=5000

Modified: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/log4j.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/log4j.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/log4j.properties (original)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/log4j.properties Wed Oct 24 20:56:16 2012
@@ -1,38 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-# 
-#    http://www.apache.org/licenses/LICENSE-2.0
-# 
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-log4j.rootLogger=INFO, stdout
-
-log4j.appender.stdout=org.apache.log4j.ConsoleAppender
-log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
-log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
-
-#log4j.appender.fileAppender=org.apache.log4j.FileAppender
-#log4j.appender.fileAppender.File=kafka-request.log
-#log4j.appender.fileAppender.layout=org.apache.log4j.PatternLayout
-#log4j.appender.fileAppender.layout.ConversionPattern= %-4r [%t] %-5p %c %x - %m%n
-
-
-# Turn on all our debugging info
-#log4j.logger.kafka=INFO
-#log4j.logger.org.I0Itec.zkclient.ZkClient=DEBUG
-
-# to print message checksum from ProducerPerformance
-log4j.logger.kafka.perf=DEBUG
-log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG
-
-# to print message checksum from ProducerPerformance
-log4j.logger.kafka.perf=DEBUG
-log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG
-

Modified: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer.properties (original)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer.properties Wed Oct 24 20:56:16 2012
@@ -1,80 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# see kafka.producer.ProducerConfig for more details
-
-############################# Producer Basics #############################
-
-# need to set either broker.list or zk.connect
-
-# configure brokers statically
-# format: brokerid1:host1:port1,brokerid2:host2:port2 ...
-#broker.list=0:localhost:9092
-
-# discover brokers from ZK
-zk.connect=localhost:2181
-
-# zookeeper session timeout; default is 6000
-#zk.sessiontimeout.ms=
-
-# the max time that the client waits to establish a connection to zookeeper; default is 6000
-#zk.connectiontimeout.ms
-
-# name of the partitioner class for partitioning events; default partition spreads data randomly
-#partitioner.class=
-
-# specifies whether the messages are sent asynchronously (async) or synchronously (sync)
-producer.type=sync
-
-# specify the compression codec for all data generated: 0: no compression, 1: gzip
-compression.codec=0
-
-# message encoder
-serializer.class=kafka.serializer.StringEncoder
-
-# allow topic level compression
-#compressed.topics=
-
-# max message size; messages larger than that size are discarded; default is 1000000
-#max.message.size=
-
-
-############################# Async Producer #############################
-# maximum time, in milliseconds, for buffering data on the producer queue 
-#queue.time=
-
-# the maximum size of the blocking queue for buffering on the producer 
-#queue.size=
-
-# Timeout for event enqueue:
-# 0: events will be enqueued immediately or dropped if the queue is full
-# -ve: enqueue will block indefinitely if the queue is full
-# +ve: enqueue will block up to this many milliseconds if the queue is full
-#queue.enqueueTimeout.ms=
-
-# the number of messages batched at the producer 
-#batch.size=
-
-# the callback handler for one or multiple events 
-#callback.handler=
-
-# properties required to initialize the callback handler 
-#callback.handler.props=
-
-# the handler for events 
-#event.handler=
-
-# properties required to initialize the event handler 
-#event.handler.props=
-

Modified: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer_performance.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer_performance.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer_performance.properties (original)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/config/producer_performance.properties Wed Oct 24 20:56:16 2012
@@ -1,5 +0,0 @@
-topic=mytest
-message-size=100
-thread=5
-compression-codec=0
-request-num-acks=-1

Modified: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/mirror_maker_test.py
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/mirror_maker_test.py?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/mirror_maker_test.py (original)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/mirror_maker_test.py Wed Oct 24 20:56:16 2012
@@ -170,9 +170,6 @@ class MirrorMakerTest(ReplicationUtils, 
                 self.anonLogger.info("sleeping for 5s")
                 time.sleep(5)
 
-                #print "#### sleeping for 30 min"
-                #time.sleep(1800)
-
                 self.log_message("creating topics")
                 kafka_system_test_utils.create_topic(self.systemTestEnv, self.testcaseEnv)
                 self.anonLogger.info("sleeping for 5s")
@@ -189,7 +186,7 @@ class MirrorMakerTest(ReplicationUtils, 
                 time.sleep(int(msgProducingFreeTimeSec))
 
                 # =============================================
-                # A while-loop to bounce leader as specified
+                # A while-loop to bounce mirror maker as specified
                 # by "num_iterations" in testcase_n_properties.json
                 # =============================================
                 i = 1
@@ -198,46 +195,25 @@ class MirrorMakerTest(ReplicationUtils, 
 
                     self.log_message("Iteration " + str(i) + " of " + str(numIterations))
 
-                    self.log_message("looking up leader")
-                    leaderDict = kafka_system_test_utils.get_leader_elected_log_line(
-                        self.systemTestEnv, self.testcaseEnv, self.leaderAttributesDict)
-        
-                    # ==========================
-                    # leaderDict looks like this:
-                    # ==========================
-                    #{'entity_id': u'3',
-                    # 'partition': '0',
-                    # 'timestamp': 1345050255.8280001,
-                    # 'hostname': u'localhost',
-                    # 'topic': 'test_1',
-                    # 'brokerid': '3'}
-
-                    # =============================================
-                    # validate to see if leader election is successful
-                    # =============================================
-                    self.log_message("validating leader election")
-                    result = kafka_system_test_utils.validate_leader_election_successful( 
-                         self.testcaseEnv, leaderDict, self.testcaseEnv.validationStatusDict)
-        
-                    # =============================================
-                    # trigger leader re-election by stopping leader
-                    # to get re-election latency
-                    # =============================================
-                    bounceLeaderFlag = self.testcaseEnv.testcaseArgumentsDict["bounce_leader"]
-                    self.log_message("bounce_leader flag : " + bounceLeaderFlag)
-                    if (bounceLeaderFlag.lower() == "true"):
-                        reelectionLatency = kafka_system_test_utils.get_reelection_latency(
-                            self.systemTestEnv, self.testcaseEnv, leaderDict, self.leaderAttributesDict)
-                        latencyKeyName = "Leader Election Latency - iter " + str(i) + " brokerid " + leaderDict["brokerid"]
-                        self.testcaseEnv.validationStatusDict[latencyKeyName] = str("{0:.2f}".format(reelectionLatency * 1000)) + " ms"
-       
                     # =============================================
-                    # starting previously terminated broker 
+                    # Bounce Mirror Maker if specified in testcase config
                     # =============================================
-                    if bounceLeaderFlag.lower() == "true":
-                        self.log_message("starting the previously terminated broker")
-                        stoppedLeaderEntityId  = leaderDict["entity_id"]
-                        kafka_system_test_utils.start_entity_in_background(self.systemTestEnv, self.testcaseEnv, stoppedLeaderEntityId)
+                    bounceMirrorMaker = self.testcaseEnv.testcaseArgumentsDict["bounce_mirror_maker"]
+                    self.log_message("bounce_mirror_maker flag : " + bounceMirrorMaker)
+                    if (bounceMirrorMaker.lower() == "true"):
+
+                        clusterConfigList       = self.systemTestEnv.clusterEntityConfigDictList
+                        mirrorMakerEntityIdList = system_test_utils.get_data_from_list_of_dicts(
+                            clusterConfigList, "role", "mirror_maker", "entity_id")
+
+                        mirrorMakerPPid = self.testcaseEnv.entityMirrorMakerParentPidDict[mirrorMakerEntityIdList[0]]
+                        self.log_message("stopping mirror maker : " + mirrorMakerPPid)
+                        kafka_system_test_utils.stop_remote_entity(self.systemTestEnv, mirrorMakerEntityIdList[0], mirrorMakerPPid)
+                        time.sleep(1)
+
+                        # starting previously terminated broker 
+                        self.log_message("starting the previously terminated mirror maker")
+                        kafka_system_test_utils.start_mirror_makers(self.systemTestEnv, self.testcaseEnv)
 
                     self.anonLogger.info("sleeping for 15s")
                     time.sleep(15)
@@ -294,13 +270,15 @@ class MirrorMakerTest(ReplicationUtils, 
                 # collect logs from remote hosts
                 # =============================================
                 kafka_system_test_utils.collect_logs_from_remote_hosts(self.systemTestEnv, self.testcaseEnv)
-    
+
                 # =============================================
                 # validate the data matched and checksum
                 # =============================================
                 self.log_message("validating data matched")
-                kafka_system_test_utils.validate_data_matched(self.systemTestEnv, self.testcaseEnv)
+                #kafka_system_test_utils.validate_data_matched(self.systemTestEnv, self.testcaseEnv)
+                kafka_system_test_utils.validate_simple_consumer_data_matched(self.systemTestEnv, self.testcaseEnv)
                 kafka_system_test_utils.validate_broker_log_segment_checksum(self.systemTestEnv, self.testcaseEnv)
+                kafka_system_test_utils.validate_broker_log_segment_checksum(self.systemTestEnv, self.testcaseEnv, "target")
 
                 # =============================================
                 # draw graphs
@@ -313,7 +291,7 @@ class MirrorMakerTest(ReplicationUtils, 
                 metrics.build_all_dashboards(self.systemTestEnv.METRICS_PATHNAME,
                                              self.testcaseEnv.testCaseDashboardsDir,
                                              self.systemTestEnv.clusterEntityConfigDictList)
-                
+
             except Exception as e:
                 self.log_message("Exception while running test {0}".format(e))
                 traceback.print_exc()

Modified: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5001/testcase_5001_properties.json
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5001/testcase_5001_properties.json?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5001/testcase_5001_properties.json (original)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5001/testcase_5001_properties.json Wed Oct 24 20:56:16 2012
@@ -11,6 +11,7 @@
   },
   "testcase_args": {
     "bounce_leader": "false",
+    "bounce_mirror_maker": "false",
     "replica_factor": "3",
     "num_partition": "1",
     "num_iteration": "1",

Added: incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5002/testcase_5002_properties.json
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5002/testcase_5002_properties.json?rev=1401875&view=auto
==============================================================================
--- incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5002/testcase_5002_properties.json (added)
+++ incubator/kafka/branches/0.8/system_test/mirror_maker_testsuite/testcase_5002/testcase_5002_properties.json Wed Oct 24 20:56:16 2012
@@ -0,0 +1,136 @@
+{
+  "description": {"01":"Replication with Mirror Maker => Bounce Mirror Maker",
+                  "02":"Set up 2 clusters such as : SOURCE => MirrorMaker => TARGET",
+                  "03":"Set up 2-node Zk cluster for both SOURCE & TARGET",
+                  "04":"Produce and consume messages to a single topic - single partition.",
+                  "05":"This test sends messages to 3 replicas",
+                  "06":"At the end it verifies the log size and contents",
+                  "07":"Use a consumer to verify no message loss in TARGET cluster.",
+                  "08":"Producer dimensions : mode:sync, acks:-1, comp:0",
+                  "09":"Log segment size    : 10240"
+  },
+  "testcase_args": {
+    "bounce_leader": "false",
+    "bounce_mirror_maker": "true",
+    "replica_factor": "3",
+    "num_partition": "1",
+    "num_iteration": "1",
+    "sleep_seconds_between_producer_calls": "1",
+    "message_producing_free_time_sec": "15",
+    "num_messages_to_produce_per_producer_call": "50"
+  },
+  "entities": [
+    {
+      "entity_id": "0",
+      "clientPort": "2108",
+      "dataDir": "/tmp/zookeeper_0",
+      "log_filename": "zookeeper_0.log",
+      "config_filename": "zookeeper_0.properties"
+    },
+    {
+      "entity_id": "1",
+      "clientPort": "2118",
+      "dataDir": "/tmp/zookeeper_1",
+      "log_filename": "zookeeper_1.log",
+      "config_filename": "zookeeper_1.properties"
+    },
+
+    {
+      "entity_id": "2",
+      "clientPort": "2128",
+      "dataDir": "/tmp/zookeeper_2",
+      "log_filename": "zookeeper_2.log",
+      "config_filename": "zookeeper_2.properties"
+    },
+    {
+      "entity_id": "3",
+      "clientPort": "2138",
+      "dataDir": "/tmp/zookeeper_3",
+      "log_filename": "zookeeper_3.log",
+      "config_filename": "zookeeper_3.properties"
+    },
+
+    {
+      "entity_id": "4",
+      "port": "9091",
+      "brokerid": "1",
+      "log.file.size": "10240",
+      "log.dir": "/tmp/kafka_server_4_logs",
+      "log_filename": "kafka_server_4.log",
+      "config_filename": "kafka_server_4.properties"
+    },
+    {
+      "entity_id": "5",
+      "port": "9092",
+      "brokerid": "2",
+      "log.file.size": "10240",
+      "log.dir": "/tmp/kafka_server_5_logs",
+      "log_filename": "kafka_server_5.log",
+      "config_filename": "kafka_server_5.properties"
+    },
+    {
+      "entity_id": "6",
+      "port": "9093",
+      "brokerid": "3",
+      "log.file.size": "10240",
+      "log.dir": "/tmp/kafka_server_6_logs",
+      "log_filename": "kafka_server_6.log",
+      "config_filename": "kafka_server_6.properties"
+    },
+    {
+      "entity_id": "7",
+      "port": "9094",
+      "brokerid": "4",
+      "log.file.size": "10240",
+      "log.dir": "/tmp/kafka_server_7_logs",
+      "log_filename": "kafka_server_7.log",
+      "config_filename": "kafka_server_7.properties"
+    },
+    {
+      "entity_id": "8",
+      "port": "9095",
+      "brokerid": "5",
+      "log.file.size": "10240",
+      "log.dir": "/tmp/kafka_server_8_logs",
+      "log_filename": "kafka_server_8.log",
+      "config_filename": "kafka_server_8.properties"
+    },
+    {
+      "entity_id": "9",
+      "port": "9096",
+      "brokerid": "6",
+      "log.file.size": "10240",
+      "log.dir": "/tmp/kafka_server_9_logs",
+      "log_filename": "kafka_server_9.log",
+      "config_filename": "kafka_server_9.properties"
+    },
+
+    {
+      "entity_id": "10",
+      "topic": "test_1",
+      "threads": "5",
+      "compression-codec": "0",
+      "message-size": "500",
+      "message": "100",
+      "request-num-acks": "-1",
+      "async":"false",
+      "log_filename": "producer_performance_10.log",
+      "config_filename": "producer_performance_10.properties"
+    },
+    {
+      "entity_id": "11",
+      "topic": "test_1",
+      "groupid": "mytestgroup",
+      "consumer-timeout-ms": "10000",
+      "log_filename": "console_consumer_11.log",
+      "config_filename": "console_consumer_11.properties"
+    },
+
+    {
+      "entity_id": "12",
+      "log_filename": "mirror_maker_12.log",
+      "mirror_consumer_config_filename": "mirror_consumer_12.properties",
+      "mirror_producer_config_filename": "mirror_producer_12.properties"
+    }
+   ]
+}

Modified: incubator/kafka/branches/0.8/system_test/replication_testsuite/config/console_consumer.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/replication_testsuite/config/console_consumer.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/replication_testsuite/config/console_consumer.properties (original)
+++ incubator/kafka/branches/0.8/system_test/replication_testsuite/config/console_consumer.properties Wed Oct 24 20:56:16 2012
@@ -1,4 +0,0 @@
-zookeeper=local:2181
-topic=test_1
-from-beginning
-consumer-timeout-ms=10000

Modified: incubator/kafka/branches/0.8/system_test/replication_testsuite/config/consumer.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/replication_testsuite/config/consumer.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/replication_testsuite/config/consumer.properties (original)
+++ incubator/kafka/branches/0.8/system_test/replication_testsuite/config/consumer.properties Wed Oct 24 20:56:16 2012
@@ -1,29 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-# 
-#    http://www.apache.org/licenses/LICENSE-2.0
-# 
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# see kafka.consumer.ConsumerConfig for more details
-
-# zk connection string
-# comma separated host:port pairs, each corresponding to a zk
-# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002"
-zk.connect=127.0.0.1:2181
-
-# timeout in ms for connecting to zookeeper
-zk.connectiontimeout.ms=1000000
-
-#consumer group id
-groupid=test-consumer-group
-
-#consumer timeout
-#consumer.timeout.ms=5000

Modified: incubator/kafka/branches/0.8/system_test/replication_testsuite/config/log4j.properties
URL: http://svn.apache.org/viewvc/incubator/kafka/branches/0.8/system_test/replication_testsuite/config/log4j.properties?rev=1401875&r1=1401874&r2=1401875&view=diff
==============================================================================
--- incubator/kafka/branches/0.8/system_test/replication_testsuite/config/log4j.properties (original)
+++ incubator/kafka/branches/0.8/system_test/replication_testsuite/config/log4j.properties Wed Oct 24 20:56:16 2012
@@ -1,38 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-# 
-#    http://www.apache.org/licenses/LICENSE-2.0
-# 
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-log4j.rootLogger=INFO, stdout
-
-log4j.appender.stdout=org.apache.log4j.ConsoleAppender
-log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
-log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
-
-#log4j.appender.fileAppender=org.apache.log4j.FileAppender
-#log4j.appender.fileAppender.File=kafka-request.log
-#log4j.appender.fileAppender.layout=org.apache.log4j.PatternLayout
-#log4j.appender.fileAppender.layout.ConversionPattern= %-4r [%t] %-5p %c %x - %m%n
-
-
-# Turn on all our debugging info
-#log4j.logger.kafka=INFO
-#log4j.logger.org.I0Itec.zkclient.ZkClient=DEBUG
-
-# to print message checksum from ProducerPerformance
-log4j.logger.kafka.perf=DEBUG
-log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG
-
-# to print message checksum from ProducerPerformance
-log4j.logger.kafka.perf=DEBUG
-log4j.logger.kafka.perf.ProducerPerformance$ProducerThread=DEBUG
-



Mime
View raw message