kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nehanarkh...@apache.org
Subject svn commit: r1579981 - /kafka/site/081/ops.html
Date Fri, 21 Mar 2014 17:03:17 GMT
Author: nehanarkhede
Date: Fri Mar 21 17:03:17 2014
New Revision: 1579981

URL: http://svn.apache.org/r1579981
Log:
Included doc on increasing replication factor of a partition

Modified:
    kafka/site/081/ops.html

Modified: kafka/site/081/ops.html
URL: http://svn.apache.org/viewvc/kafka/site/081/ops.html?rev=1579981&r1=1579980&r2=1579981&view=diff
==============================================================================
--- kafka/site/081/ops.html (original)
+++ kafka/site/081/ops.html Fri Mar 21 17:03:17 2014
@@ -128,7 +128,7 @@ The partition reassignment tool can run 
 <h5>Automatically migrating data to new machines</h5>
 The partition reassignment tool can be used to move some topics off of the current set of
brokers to the newly added brokers. This is typically useful while expanding an existing cluster
since it is easier to move entire topics to the new set of brokers, than moving one partition
at a time. When used to do this, the user should provide a list of topics that should be moved
to the new set of brokers and a target list of new brokers. The tool then evenly distributes
all partitions for the given list of topics across the new set of brokers. During this move,
the replication factor of the topic is kept constant. Effectively the replicas for all partitions
for the input list of topics are moved from the old set of brokers to the newly added brokers.

 <p>
-For example, the following example will move all partitions for topics foo1,foo2 to the new
set of brokers 5,6. At the end of this move, all partitions for topics foo1 and foo2 will
<i>only</i> exist on brokers 5,6
+For instance, the following example will move all partitions for topics foo1,foo2 to the
new set of brokers 5,6. At the end of this move, all partitions for topics foo1 and foo2 will
<i>only</i> exist on brokers 5,6
 <p>
 Since, the tool accepts the input list of topics as a json file, you first need to identify
the topics you want to move and create the json file as follows-
 <pre>
@@ -177,7 +177,7 @@ Reassignment of partition [foo2,2] compl
 <h5>Custom partition assignment and migration</h5>
 The partition reassignment tool can also be used to selectively move replicas of a partition
to a specific set of brokers. When used in this manner, it is assumed that the user knows
the reassignment plan and does not require the tool to generate a candidate reassignment,
effectively skipping the --generate step and moving straight to the --execute step
 <p>
-For example, the following example moves partition 0 of topic foo1 to brokers 5,6 and partition
1 of topic foo2 to brokers 2,3
+For instance, the following example moves partition 0 of topic foo1 to brokers 5,6 and partition
1 of topic foo2 to brokers 2,3
 <p>
 The first step is to hand craft the custom reassignment plan in a json file-
 <pre>
@@ -207,6 +207,41 @@ Reassignment of partition [foo2,1] compl
 <h5>Decommissioning machines</h5>
 The partition reassignment tool does not have the ability to automatically generate a reassignment
plan for decommissioning brokers yet. As such, the admin has to come up with a reassignment
plan to move the replica for all partitions hosted on the broker to be decommissioned, to
the rest of the brokers. This can be relatively tedious as the reassignment needs to ensure
that all the replicas are not moved from the decommissioned broker to only one other broker.
To make this process effortless, we plan to add tooling support for decommissioning brokers
in 0.8.2.
 
+<h5>Increasing replication factor of an existing partition</h5>
+Increasing the replication factor of an existing partition is easy. Just specify the extra
replicas in the custom reassignment json file and use it with the --execute option to increase
the replication factor of the specified partitions. 
+<p>
+For instance, the following example increases the replication factor of partition 0 of topic
foo from 1 to 3. Before increasing the replication factor, the partition's only replica existed
on broker 5. As part of increasing the replication factor, we will add more replicas on brokers
6 and 7.
+<p>
+The first step is to hand craft the custom reassignment plan in a json file-
+<pre>
+> cat increase-replication-factor.json
+{"version":1,"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}
+</pre>
+Then, use the json file with the --execute option to start the reassignment process-
+<pre>
+> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file
increase-replication-factor.json --execute
+Current partition replica assignment
+
+{"version":1,"partitions":[{"topic":"foo","partition":0,"replicas":[5]}]}
+
+Save this to use as the --reassignment-json-file option during rollback
+Successfully started reassignment of partitions
+{"version":1,"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}
+</pre>
+<p>
+The --verify option can be used with the tool to check the status of the partition reassignment.
Note that the same increase-replication-factor.json (used with the --execute option) should
be used with the --verify option
+<pre>
+bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json
--verify
+Status of partition reassignment:
+Reassignment of partition [foo,0] completed successfully
+</pre>
+You can also verify the increase in replication factor with the kafka-topics tool-
+<pre>
+> bin/kafka-topics.sh --zookeeper localhost:2181 --topic foo --describe
+Topic:foo	PartitionCount:1	ReplicationFactor:3	Configs:
+	Topic: foo	Partition: 0	Leader: 5	Replicas: 5,6,7	Isr: 5,6,7
+</pre>
+
 <h3><a id="datacenters">6.2 Datacenters</a></h3>
 
 Some deployments will need to manage a data pipeline that spans multiple datacenters. Our
recommended approach to this is to deploy a local Kafka cluster in each datacenter with application
instances in each datacenter interacting only with their local cluster and mirroring between
clusters (see the documentation on the <a href="#basic_ops_mirror_maker">mirror maker
tool</a> for how to do this).



Mime
View raw message