kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From guozh...@apache.org
Subject [kafka] branch trunk updated: KAFKA-8219: add doc changes for static membership release (#6790)
Date Mon, 27 May 2019 21:25:58 GMT
This is an automated email from the ASF dual-hosted git repository.

guozhang pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/kafka.git


The following commit(s) were added to refs/heads/trunk by this push:
     new 27943e3  KAFKA-8219: add doc changes for static membership release (#6790)
27943e3 is described below

commit 27943e39876efad87fce7b558af3419f1a7a025c
Author: Boyang Chen <boyang@confluent.io>
AuthorDate: Mon May 27 14:25:47 2019 -0700

    KAFKA-8219: add doc changes for static membership release (#6790)
    
    * add doc changes for static membership release
    
    * address comments
    
    * address comments for a separate page
    
    * address Matthias' comment
---
 docs/design.html                | 22 ++++++++++++++++++++++
 docs/streams/upgrade-guide.html |  7 +++++++
 docs/upgrade.html               |  8 ++++++++
 3 files changed, 37 insertions(+)

diff --git a/docs/design.html b/docs/design.html
index eadf29d..4c4e796 100644
--- a/docs/design.html
+++ b/docs/design.html
@@ -216,6 +216,28 @@
     In the case of Hadoop we parallelize the data load by splitting the load over individual
map tasks, one for each node/topic/partition combination, allowing full parallelism in the
loading. Hadoop provides the task
     management, and tasks which fail can restart without danger of duplicate data&mdash;they
simply restart from their original position.
 
+    <h4><a id="static_membership" href="#static_membership">Static Membership</a></h4>
+    Static membership aims to improve the availability of stream applications, consumer groups
and other applications built on top of the group rebalance protocol.
+    The rebalance protocol relies on the group coordinator to allocate entity ids to group
members. These generated ids are ephemeral and will change when members restart and rejoin.
+    For consumer based apps, this "dynamic membership" can cause a large percentage of tasks
re-assigned to different instances during administrative operations
+    such as code deploys, configuration updates and periodic restarts. For large state applications,
shuffled tasks need a long time to recover their local states before processing
+    and cause applications to be partially or entirely unavailable. Motivated by this observation,
Kafka’s group management protocol allows group members to provide persistent entity ids.
+    Group membership remains unchanged based on those ids, thus no rebalance will be triggered.
+    <p>
+    If you want to use static membership,
+    <ul>
+        <li>Upgrade both broker cluster and client apps to 2.3 or beyond, and also
make sure the upgraded brokers are using <code>inter.broker.protocol.version</code>
+            of 2.3 or beyond as well.</li>
+        <li>Set the config <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code>
to a unique value for each consumer instance under one group.</li>
+        <li>For Kafka Streams applications, it is sufficient to set a unique <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code>
per KafkaStreams instance,
+            independent of the number of used threads for an instance.</li>
+    </ul>
+    If your broker is on an older version than 2.3, but you choose to set <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code>
on the client side, the application will detect
+    the broker version and then throws an UnsupportedException. If you accidentally configure
duplicate ids for different instances,
+    a fencing mechanism on broker side will inform your duplicate client to shutdown immediately
by triggering a <code>org.apache.kafka.common.errors.FencedInstanceIdException</code>.
+    For more details, see
+    <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances">KIP-345</a>
+
     <h3><a id="semantics" href="#semantics">4.6 Message Delivery Semantics</a></h3>
     <p>
     Now that we understand a little about how producers and consumers work, let's discuss
the semantic guarantees Kafka provides between producer and consumer. Clearly there are multiple
possible message delivery
diff --git a/docs/streams/upgrade-guide.html b/docs/streams/upgrade-guide.html
index f4c5d48..4b3dac2 100644
--- a/docs/streams/upgrade-guide.html
+++ b/docs/streams/upgrade-guide.html
@@ -241,6 +241,13 @@
         details, see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-312%3A+Add+Overloaded+StreamsBuilder+Build+Method+to+Accept+java.util.Properties">KIP-312</a>
     </p>
 
+    <p>
+        We are introducing static membership towards Kafka Streams user. This feature reduces
unnecessary rebalances during normal application upgrades or rolling bounces.
+        For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static
membership design</a>.
+        Note, Kafka Streams uses the same <code>ConsumerConfig#GROUP_INSTANCE_ID_CONFIG</code>,
and you only need to make sure it is uniquely defined across
+        different stream instances in one application.
+    </p>
+
     <h3><a id="streams_api_changes_200" href="#streams_api_changes_200">Streams
API changes in 2.0.0</a></h3>
     <p>
         In 2.0.0 we have added a few new APIs on the <code>ReadOnlyWindowStore</code>
interface (for details please read <a href="#streams_api_changes_200">Streams API changes</a>
below).
diff --git a/docs/upgrade.html b/docs/upgrade.html
index be9886c..d9232fa 100644
--- a/docs/upgrade.html
+++ b/docs/upgrade.html
@@ -71,6 +71,14 @@
     </li>
 </ol>
 
+<h5><a id="upgrade_230_notable" href="#upgrade_230_notable">Notable changes in
2.3.0</a></h5>
+<ul>
+    <li>
+        We are introducing static membership towards consumer user. This feature reduces
unnecessary rebalances during normal application upgrades or rolling bounces.
+        For more details on how to use it, checkout <a href="/{{version}}/design/#static_membership">static
membership design</a>.
+    </li>
+</ul>
+
 <h5><a id="upgrade_220_notable" href="#upgrade_220_notable">Notable changes in
2.2.0</a></h5>
 <ul>
     <li>The default consumer group id has been changed from the empty string (<code>""</code>)
to <code>null</code>. Consumers who use the new default group id will not be able
to subscribe to topics,


Mime
View raw message