kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jjko...@apache.org
Subject svn commit: r1656966 - /kafka/site/082/implementation.html
Date Wed, 04 Feb 2015 01:17:49 GMT
Author: jjkoshy
Date: Wed Feb  4 01:17:49 2015
New Revision: 1656966

URL: http://svn.apache.org/r1656966
Log:
KAFKA-1729; follow-up documentation patch on offset management

Modified:
    kafka/site/082/implementation.html

Modified: kafka/site/082/implementation.html
URL: http://svn.apache.org/viewvc/kafka/site/082/implementation.html?rev=1656966&r1=1656965&r2=1656966&view=diff
==============================================================================
--- kafka/site/082/implementation.html (original)
+++ kafka/site/082/implementation.html Wed Feb  4 01:17:49 2015
@@ -227,7 +227,7 @@ Note that two kinds of corruption must b
 <h3><a id="distributionimpl">5.6 Distribution</a></h3>
 <h4>Consumer Offset Tracking</h4>
 <p>
-The high-level consumer tracks the maximum offset it has consumed in each partition and periodically
commits its offset vector so that it can resume from those offsets in the event of a restart.
Kafka provides the option to store all the offsets for a given consumer group in a designated
broker (for that group) called the <i>offset manager</i>. i.e., any consumer instance
in that consumer group should send its offset commits and fetches to that offset manager (broker).
The high-level consumer handles this automatically. If you use the simple consumer you will
need to discover the offset manager and explicitly commit or fetch offsets. A consumer can
look up its offset manager by issuing a ConsumerMetadataRequest to any Kafka broker and reading
the ConsumerMetadataResponse which will contain the offset manager. The consumer can then
proceed to commit or fetch offsets from the offsets manager broker. In case the offset manager
moves, the consumer will need to rediscover the offset mana
 ger. If you wish to manage your offsets manually, you can take a look at these <a href="https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka">code
samples that explain how to issue OffsetCommitRequest and OffsetFetchRequest</a>.
+The high-level consumer tracks the maximum offset it has consumed in each partition and periodically
commits its offset vector so that it can resume from those offsets in the event of a restart.
Kafka provides the option to store all the offsets for a given consumer group in a designated
broker (for that group) called the <i>offset manager</i>. i.e., any consumer instance
in that consumer group should send its offset commits and fetches to that offset manager (broker).
The high-level consumer handles this automatically. If you use the simple consumer you will
need to manage offsets manually. This is currently unsupported in the Java simple consumer
which can only commit or fetch offsets in ZooKeeper. If you use the Scala simple consumer
you can discover the offset manager and explicitly commit or fetch offsets to the offset manager.
A consumer can look up its offset manager by issuing a ConsumerMetadataRequest to any Kafka
broker and reading the ConsumerMetadataResponse which will c
 ontain the offset manager. The consumer can then proceed to commit or fetch offsets from
the offsets manager broker. In case the offset manager moves, the consumer will need to rediscover
the offset manager. If you wish to manage your offsets manually, you can take a look at these
<a href="https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka">code
samples that explain how to issue OffsetCommitRequest and OffsetFetchRequest</a>.
 </p>
 
 <p>



Mime
View raw message