kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jkr...@apache.org
Subject git commit: Misc. minor house-keeping fixes: add reasonable GC settings, fix up README, fix up example configs, fix the logging for tools, use a log directory for logs instead of the root directory.
Date Mon, 05 Aug 2013 17:13:37 GMT
Updated Branches:
  refs/heads/0.8 f3f8fa5c8 -> 14af71325


Misc. minor house-keeping fixes: add reasonable GC settings, fix up README, fix up example
configs, fix the logging for tools, use a log directory for logs instead of the root directory.


Project: http://git-wip-us.apache.org/repos/asf/kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/14af7132
Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/14af7132
Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/14af7132

Branch: refs/heads/0.8
Commit: 14af713252f016929f8150f2f6c2df9d0d447ddb
Parents: f3f8fa5
Author: Jay Kreps <jay.kreps@gmail.com>
Authored: Thu Jul 11 08:06:43 2013 -0700
Committer: Jay Kreps <jay.kreps@gmail.com>
Committed: Mon Aug 5 10:10:18 2013 -0700

----------------------------------------------------------------------
 README.md                                   | 55 +++++++++++-------------
 bin/kafka-add-partitions.sh                 |  1 -
 bin/kafka-check-reassignment-status.sh      |  3 +-
 bin/kafka-console-consumer-log4j.properties | 21 ---------
 bin/kafka-console-consumer.sh               |  5 +--
 bin/kafka-console-producer.sh               |  4 +-
 bin/kafka-consumer-perf-test.sh             |  1 +
 bin/kafka-create-topic.sh                   |  4 +-
 bin/kafka-delete-topic.sh                   | 19 --------
 bin/kafka-list-topic.sh                     |  4 +-
 bin/kafka-preferred-replica-election.sh     |  3 +-
 bin/kafka-producer-perf-test.sh             |  1 +
 bin/kafka-reassign-partitions.sh            |  3 +-
 bin/kafka-replay-log-producer.sh            |  4 +-
 bin/kafka-run-class.sh                      | 53 +++++++++++++++++++----
 bin/kafka-server-start.sh                   |  8 ++--
 bin/kafka-simple-consumer-perf-test.sh      |  1 +
 bin/zookeeper-server-start.sh               |  5 ++-
 config/log4j.properties                     |  8 ++--
 config/producer.properties                  |  2 +-
 config/server.properties                    | 18 +++-----
 config/tools-log4j.properties               | 22 ++++++++++
 22 files changed, 123 insertions(+), 122 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index cf0d59e..9879427 100644
--- a/README.md
+++ b/README.md
@@ -1,59 +1,52 @@
-# Kafka is a distributed publish/subscribe messaging system #
+# Apache Kafka #
 
-It is designed to support the following
+See our [web site](http://kafka.apache.org) details on the project.
 
-* Persistent messaging with O(1) disk structures that provide constant time performance even
with many TB of stored messages.
-* High-throughput: even with very modest hardware Kafka can support hundreds of thousands
of messages per second.
-* Explicit support for partitioning messages over Kafka servers and distributing consumption
over a cluster of consumer machines while maintaining per-partition ordering semantics.
-* Support for parallel data load into Hadoop.
-
-Kafka is aimed at providing a publish-subscribe solution that can handle all activity stream
data and processing on a consumer-scale web site. This kind of activity (page views, searches,
and other user actions) are a key ingredient in many of the social feature on the modern web.
This data is typically handled by "logging" and ad hoc log aggregation solutions due to the
throughput requirements. This kind of ad hoc solution is a viable solution to providing logging
data to an offline analysis system like Hadoop, but is very limiting for building real-time
processing. Kafka aims to unify offline and online processing by providing a mechanism for
parallel load into Hadoop as well as the ability to partition real-time consumption over a
cluster of machines.
-
-See our [web site](http://kafka.apache.org/) for more details on the project.
-
-## Contribution ##
-
-Kafka is a new project, and we are interested in building the community; we would welcome
any thoughts or [patches](https://issues.apache.org/jira/browse/KAFKA). You can reach us [on
the Apache mailing lists](http://kafka.apache.org/contact.html).
-
-The Kafka code is available from:
- * git clone http://git-wip-us.apache.org/repos/asf/kafka.git kafka
-
-To contribute you can follow:
- * https://cwiki.apache.org/confluence/display/KAFKA/Git+Workflow
+## Building It ##
 
 To build for all supported versions of Scala: 
-
 1. ./sbt +package
 
 To build for a particular version of Scala (either 2.8.0, 2.8.2, 2.9.1 or 2.9.2): 
 
 1. ./sbt "++2.8.0 package" *or* ./sbt "++2.8.2 package" *or* ./sbt "++2.9.1 package" *or*
./sbt "++2.9.2 package"
 
+## Running It ##
+
+To run follow the instructions here:
+1. http://kafka.apache.org/08/quickstart.html
+
+## Other Build Tips ##
+
 Here are some useful sbt commands, to be executed at the sbt command prompt (./sbt). Prefixing
with "++<version> " runs the
 command for a specific Scala version, prefixing with "+" will perform the action for all
versions of Scala, and no prefix
 runs the command for the default (2.8.0) version of Scala. -
 
 tasks : Lists all the sbt commands and their descriptions
-
 clean : Deletes all generated files (the target directory).
-
 compile : Compile all the sub projects, but not create the jars
-
 test : Run all unit tests in all sub projects
-
 release-zip : Create all the jars, run unit tests and create a deployable release zip
-
 package: Creates jars for src, test, docs etc
-
 projects : List all the sub projects 
-
 project sub_project_name : Switch to a particular sub-project. For example, to switch to
the core kafka code, use "project core-kafka"
 
-Following commands can be run only on a particular sub project -
-
+The following commands can be run only on a particular sub project -
 test-only package.test.TestName : Runs only the specified test in the current sub project
-
 run : Provides options to run any of the classes that have a main method. For example, you
can switch to project java-examples, and run the examples there by executing "project java-examples"
followed by "run" 
 
 For more details please see the [SBT documentation](https://github.com/harrah/xsbt/wiki)
 
+## Contribution ##
+
+Kafka is a new project, and we are interested in building the community; we would welcome
any thoughts or [patches](https://issues.apache.org/jira/browse/KAFKA). You can reach us [on
the Apache mailing lists](http://kafka.apache.org/contact.html).
+
+To contribute follow the instructions here:
+ * http://kafka.apache.org/contributing.html
+
+We also welcome patches for the website and documentation which can be found here:
+ * https://svn.apache.org/repos/asf/kafka/site
+
+
+
+

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-add-partitions.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-add-partitions.sh b/bin/kafka-add-partitions.sh
index 7d217e2..c08b837 100755
--- a/bin/kafka-add-partitions.sh
+++ b/bin/kafka-add-partitions.sh
@@ -15,5 +15,4 @@
 # limitations under the License.
 
 base_dir=$(dirname $0)
-export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
 $base_dir/kafka-run-class.sh kafka.admin.AddPartitionsCommand $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-check-reassignment-status.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-check-reassignment-status.sh b/bin/kafka-check-reassignment-status.sh
index 4eb9aca..1f21858 100755
--- a/bin/kafka-check-reassignment-status.sh
+++ b/bin/kafka-check-reassignment-status.sh
@@ -14,5 +14,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-$base_dir/kafka-run-class.sh kafka.admin.CheckReassignmentStatus $@
+$(dirname $0)/kafka-run-class.sh kafka.admin.CheckReassignmentStatus $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-console-consumer-log4j.properties
----------------------------------------------------------------------
diff --git a/bin/kafka-console-consumer-log4j.properties b/bin/kafka-console-consumer-log4j.properties
deleted file mode 100644
index 6b76444..0000000
--- a/bin/kafka-console-consumer-log4j.properties
+++ /dev/null
@@ -1,21 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-# 
-#    http://www.apache.org/licenses/LICENSE-2.0
-# 
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-log4j.rootLogger=INFO, stderr
-
-log4j.appender.stderr=org.apache.log4j.ConsoleAppender
-log4j.appender.stderr.target=System.err
-log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
-log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n
-

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-console-consumer.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-console-consumer.sh b/bin/kafka-console-consumer.sh
index 8a76c6c..95edc9e 100755
--- a/bin/kafka-console-consumer.sh
+++ b/bin/kafka-console-consumer.sh
@@ -14,6 +14,5 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
-$base_dir/kafka-run-class.sh kafka.consumer.ConsoleConsumer $@
+export KAFKA_HEAP_OPTS="-Xmx512M"
+$(dirname $0)/kafka-run-class.sh kafka.consumer.ConsoleConsumer $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-console-producer.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-console-producer.sh b/bin/kafka-console-producer.sh
index 99d2dc6..f102c22 100755
--- a/bin/kafka-console-producer.sh
+++ b/bin/kafka-console-producer.sh
@@ -14,5 +14,5 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-$base_dir/kafka-run-class.sh kafka.producer.ConsoleProducer $@
+export KAFKA_HEAP_OPTS="-Xmx512M"
+$(dirname $0)/kafka-run-class.sh kafka.producer.ConsoleProducer $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-consumer-perf-test.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-consumer-perf-test.sh b/bin/kafka-consumer-perf-test.sh
index 7745ea8..39434d1 100755
--- a/bin/kafka-consumer-perf-test.sh
+++ b/bin/kafka-consumer-perf-test.sh
@@ -14,4 +14,5 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+export KAFKA_HEAP_OPTS="-Xmx512M"
 $(dirname $0)/kafka-run-class.sh kafka.perf.ConsumerPerformance $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-create-topic.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-create-topic.sh b/bin/kafka-create-topic.sh
index fccda7b..7a5f297 100755
--- a/bin/kafka-create-topic.sh
+++ b/bin/kafka-create-topic.sh
@@ -14,6 +14,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
-$base_dir/kafka-run-class.sh kafka.admin.CreateTopicCommand $@
+$(dirname $0)/kafka-run-class.sh kafka.admin.CreateTopicCommand $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-delete-topic.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-delete-topic.sh b/bin/kafka-delete-topic.sh
deleted file mode 100755
index f266ae3..0000000
--- a/bin/kafka-delete-topic.sh
+++ /dev/null
@@ -1,19 +0,0 @@
-#!/bin/bash
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements.  See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License.  You may obtain a copy of the License at
-# 
-#    http://www.apache.org/licenses/LICENSE-2.0
-# 
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-base_dir=$(dirname $0)
-export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
-$base_dir/kafka-run-class.sh kafka.admin.DeleteTopicCommand $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-list-topic.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-list-topic.sh b/bin/kafka-list-topic.sh
index 1235ad0..bb2dde5 100755
--- a/bin/kafka-list-topic.sh
+++ b/bin/kafka-list-topic.sh
@@ -14,6 +14,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/kafka-console-consumer-log4j.properties"
-$base_dir/kafka-run-class.sh kafka.admin.ListTopicCommand $@
+$(dirname $0)/kafka-run-class.sh kafka.admin.ListTopicCommand $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-preferred-replica-election.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-preferred-replica-election.sh b/bin/kafka-preferred-replica-election.sh
index 9c63e43..6416dc1 100755
--- a/bin/kafka-preferred-replica-election.sh
+++ b/bin/kafka-preferred-replica-election.sh
@@ -14,5 +14,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-$base_dir/kafka-run-class.sh kafka.admin.PreferredReplicaLeaderElectionCommand $@
+$(dirname $0)/kafka-run-class.sh kafka.admin.PreferredReplicaLeaderElectionCommand $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-producer-perf-test.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-producer-perf-test.sh b/bin/kafka-producer-perf-test.sh
index 25c4dce..1935522 100755
--- a/bin/kafka-producer-perf-test.sh
+++ b/bin/kafka-producer-perf-test.sh
@@ -14,4 +14,5 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+export KAFKA_HEAP_OPTS="-Xmx512M"
 $(dirname $0)/kafka-run-class.sh kafka.perf.ProducerPerformance $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-reassign-partitions.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-reassign-partitions.sh b/bin/kafka-reassign-partitions.sh
index 162872c..a6bd148 100755
--- a/bin/kafka-reassign-partitions.sh
+++ b/bin/kafka-reassign-partitions.sh
@@ -14,5 +14,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-$base_dir/kafka-run-class.sh kafka.admin.ReassignPartitionsCommand $@
+$(dirname $0)/kafka-run-class.sh kafka.admin.ReassignPartitionsCommand $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-replay-log-producer.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-replay-log-producer.sh b/bin/kafka-replay-log-producer.sh
index f36e0e2..86943ee 100755
--- a/bin/kafka-replay-log-producer.sh
+++ b/bin/kafka-replay-log-producer.sh
@@ -14,6 +14,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-base_dir=$(dirname $0)
-export KAFKA_OPTS="-Xmx512M -server -Dcom.sun.management.jmxremote -Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
-$base_dir/kafka-run-class.sh kafka.tools.ReplayLogProducer $@
+$(dirname $0)/kafka-run-class.sh kafka.tools.ReplayLogProducer $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-run-class.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-run-class.sh b/bin/kafka-run-class.sh
index 1b66655..eb6ff1b 100755
--- a/bin/kafka-run-class.sh
+++ b/bin/kafka-run-class.sh
@@ -22,10 +22,18 @@ fi
 
 base_dir=$(dirname $0)/..
 
-SCALA_VERSION=2.8.0
+# create logs directory
+LOG_DIR=$base_dir/logs
+if [ ! -d $LOG_DIR ]; then
+	mkdir $LOG_DIR
+fi
+
+if [ -z "$SCALA_VERSION" ]; then
+	SCALA_VERSION=2.8.0
+fi
 
 # assume all dependencies have been packaged into one jar with sbt-assembly's task "assembly-package-dependency"
-for file in $base_dir/core/target/scala-2.8.0/*.jar;
+for file in $base_dir/core/target/scala-${SCALA_VERSION}/*.jar;
 do
   CLASSPATH=$CLASSPATH:$file
 done
@@ -46,30 +54,59 @@ do
   CLASSPATH=$CLASSPATH:$file
 done
 
+# JMX settings
 if [ -z "$KAFKA_JMX_OPTS" ]; then
   KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false
 -Dcom.sun.management.jmxremote.ssl=false "
 fi
 
-if [ -z "$KAFKA_OPTS" ]; then
-  KAFKA_OPTS="-Xmx512M -server  -Dlog4j.configuration=file:$base_dir/config/log4j.properties"
-fi
-
+# JMX port to use
 if [  $JMX_PORT ]; then
   KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
 fi
 
+# Log4j settings
+if [ -z "$KAFKA_LOG4J_OPTS" ]; then
+  KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/tools-log4j.properties"
+fi
+
+# Generic jvm settings you want to add
+if [ -z "$KAFKA_OPTS" ]; then
+  KAFKA_OPTS=""
+fi
+
+# Which java to use
 if [ -z "$JAVA_HOME" ]; then
   JAVA="java"
 else
   JAVA="$JAVA_HOME/bin/java"
 fi
 
-$JAVA $KAFKA_OPTS $KAFKA_JMX_OPTS -cp $CLASSPATH "$@"
+# Memory options
+if [ -z "$KAFKA_HEAP_OPTS" ]; then
+  KAFKA_HEAP_OPTS="-Xmx256M"
+fi
+
+# JVM performance options
+if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then
+  KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseCompressedOops -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC"
+fi
+
+# GC options
+GC_FILE_SUFFIX='-gc.log'
+GC_LOG_FILE_NAME=''
+if [ "$1" = "daemon" ] && [ -z "$KAFKA_GC_LOG_OPTS"] ; then
+  shift
+  GC_LOG_FILE_NAME=$1$GC_FILE_SUFFIX
+  shift
+  KAFKA_GC_LOG_OPTS="-Xloggc:$LOG_DIR/$GC_LOG_FILE_NAME -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps "
+fi
+
+$JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS
-cp $CLASSPATH $KAFKA_OPTS "$@"
 
 exitval=$?
 
 if [ $exitval -eq "1" ] ; then 
-	$JAVA $KAFKA_OPTS $KAFKA_JMX_OPTS -cp $CLASSPATH "$@" >& exception.txt
+	$JAVA $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_OPTS
-cp $CLASSPATH $KAFKA_OPTS "$@" >& exception.txt
 	exception=`cat exception.txt`
 	noBuildMessage='Please build the project using sbt. Documentation is available at http://kafka.apache.org/'
 	pattern="(Could not find or load main class)|(java\.lang\.NoClassDefFoundError)"

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-server-start.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-server-start.sh b/bin/kafka-server-start.sh
index 307f76c..4005289 100755
--- a/bin/kafka-server-start.sh
+++ b/bin/kafka-server-start.sh
@@ -19,7 +19,7 @@ then
 	echo "USAGE: $0 server.properties"
 	exit 1
 fi
-
-export JMX_PORT=${JMX_PORT:-9999}
-
-$(dirname $0)/kafka-run-class.sh kafka.Kafka $@
+base_dir=$(dirname $0)
+export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
+export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
+$base_dir/kafka-run-class.sh daemon kafkaServer kafka.Kafka $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/kafka-simple-consumer-perf-test.sh
----------------------------------------------------------------------
diff --git a/bin/kafka-simple-consumer-perf-test.sh b/bin/kafka-simple-consumer-perf-test.sh
index a9246cb..94c9ed0 100755
--- a/bin/kafka-simple-consumer-perf-test.sh
+++ b/bin/kafka-simple-consumer-perf-test.sh
@@ -14,4 +14,5 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+export KAFKA_HEAP_OPTS="-Xmx512M"
 $(dirname $0)/kafka-run-class.sh kafka.perf.SimpleConsumerPerformance $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/bin/zookeeper-server-start.sh
----------------------------------------------------------------------
diff --git a/bin/zookeeper-server-start.sh b/bin/zookeeper-server-start.sh
index 184a10b..e4746a8 100755
--- a/bin/zookeeper-server-start.sh
+++ b/bin/zookeeper-server-start.sh
@@ -19,5 +19,8 @@ then
 	echo "USAGE: $0 zookeeper.properties"
 	exit 1
 fi
+base_dir=$(dirname $0)
+export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
+export KAFKA_HEAP_OPTS="-Xmx512M -Xms512M"
+$base_dir/kafka-run-class.sh daemon zookeeper org.apache.zookeeper.server.quorum.QuorumPeerMain
$@
 
-$(dirname $0)/kafka-run-class.sh org.apache.zookeeper.server.quorum.QuorumPeerMain $@

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/log4j.properties
----------------------------------------------------------------------
diff --git a/config/log4j.properties b/config/log4j.properties
index b76bc94..dcf48f5 100644
--- a/config/log4j.properties
+++ b/config/log4j.properties
@@ -20,25 +20,25 @@ log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
 
 log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
 log4j.appender.kafkaAppender.DatePattern='.'yyyy-MM-dd-HH
-log4j.appender.kafkaAppender.File=server.log
+log4j.appender.kafkaAppender.File=logs/server.log
 log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
 log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
 
 log4j.appender.stateChangeAppender=org.apache.log4j.DailyRollingFileAppender
 log4j.appender.stateChangeAppender.DatePattern='.'yyyy-MM-dd-HH
-log4j.appender.stateChangeAppender.File=state-change.log
+log4j.appender.stateChangeAppender.File=logs/state-change.log
 log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
 log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
 
 log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender
 log4j.appender.requestAppender.DatePattern='.'yyyy-MM-dd-HH
-log4j.appender.requestAppender.File=kafka-request.log
+log4j.appender.requestAppender.File=logs/kafka-request.log
 log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
 log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
 
 log4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender
 log4j.appender.controllerAppender.DatePattern='.'yyyy-MM-dd-HH
-log4j.appender.controllerAppender.File=controller.log
+log4j.appender.controllerAppender.File=logs/controller.log
 log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
 log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
 

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/producer.properties
----------------------------------------------------------------------
diff --git a/config/producer.properties b/config/producer.properties
index 162b8a6..52a7611 100644
--- a/config/producer.properties
+++ b/config/producer.properties
@@ -16,7 +16,7 @@
 
 ############################# Producer Basics #############################
 
-# list of brokers used for bootstrapping
+# list of brokers used for bootstrapping knowledge about the rest of the cluster
 # format: host1:port1,host2:port2 ...
 metadata.broker.list=localhost:9092
 

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/server.properties
----------------------------------------------------------------------
diff --git a/config/server.properties b/config/server.properties
index 0589c71..977b8da 100644
--- a/config/server.properties
+++ b/config/server.properties
@@ -47,8 +47,8 @@ socket.request.max.bytes=104857600
 
 ############################# Log Basics #############################
 
-# The directory under which to store log files
-log.dir=/tmp/kafka-logs
+# A comma seperated list of directories under which to store log files
+log.dirs=/tmp/kafka-logs
 
 # The number of logical partitions per topic per server. More partitions allow greater parallelism
 # for consumption, but also mean more files.
@@ -56,12 +56,12 @@ num.partitions=2
 
 ############################# Log Flush Policy #############################
 
-# The following configurations control the flush of data to disk. This is the most
+# The following configurations control the flush of data to disk. This is among the most
 # important performance knob in kafka.
 # There are a few important trade-offs here:
-#    1. Durability: Unflushed data is at greater risk of loss in the event of a crash.
-#    2. Latency: Data is not made available to consumers until it is flushed (which adds
latency).
-#    3. Throughput: The flush is generally the most expensive operation. 
+#    1. Durability: Unflushed data may be lost if you are not using replication.
+#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does
occur as there will be a lot of data to flush.
+#    3. Throughput: The flush is generally the most expensive operation, and a small flush
interval may lead to exceessive seeks. 
 # The settings below allow one to configure the flush policy to flush data after a period
of time or
 # every N messages (or both). This can be done globally and overridden on a per-topic basis.
 
@@ -107,10 +107,4 @@ zookeeper.connect=localhost:2181
 # Timeout in ms for connecting to zookeeper
 zookeeper.connection.timeout.ms=1000000
 
-# metrics reporter properties
-kafka.metrics.polling.interval.secs=5
-kafka.metrics.reporters=kafka.metrics.KafkaCSVMetricsReporter
-kafka.csv.metrics.dir=/tmp/kafka_metrics
-# Disable csv reporting by default.
-kafka.csv.metrics.reporter.enabled=false
 

http://git-wip-us.apache.org/repos/asf/kafka/blob/14af7132/config/tools-log4j.properties
----------------------------------------------------------------------
diff --git a/config/tools-log4j.properties b/config/tools-log4j.properties
new file mode 100644
index 0000000..7924049
--- /dev/null
+++ b/config/tools-log4j.properties
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+log4j.rootLogger=WARN, stdout 
+
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
+
+


Mime
View raw message