kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From guozh...@apache.org
Subject kafka git commit: MINOR: added upgrade and API changes to docs
Date Fri, 03 Feb 2017 05:36:38 GMT
Repository: kafka
Updated Branches:
  refs/heads/0.10.2 7b5bceeb4 -> d952d94e5


MINOR: added upgrade and API changes to docs

Author: Matthias J. Sax <matthias@confluent.io>

Reviewers: Ismael Juma, Will Marshall, Damian Guy, Guozhang Wang, Michael G. Noll

Closes #2461 from mjsax/addStreamsUpdateSecton

(cherry picked from commit ab4ebb4bb93fbaab432d3f91dbc362d18796c83c)
Signed-off-by: Guozhang Wang <wangguoz@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka/commit/d952d94e
Tree: http://git-wip-us.apache.org/repos/asf/kafka/tree/d952d94e
Diff: http://git-wip-us.apache.org/repos/asf/kafka/diff/d952d94e

Branch: refs/heads/0.10.2
Commit: d952d94e5f0dad08af3ec935fe43e3a17ea97359
Parents: 7b5bcee
Author: Matthias J. Sax <matthias@confluent.io>
Authored: Thu Feb 2 21:36:27 2017 -0800
Committer: Guozhang Wang <wangguoz@gmail.com>
Committed: Thu Feb 2 21:36:35 2017 -0800

----------------------------------------------------------------------
 docs/streams.html | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++
 docs/upgrade.html | 20 +++++++-----
 2 files changed, 96 insertions(+), 8 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kafka/blob/d952d94e/docs/streams.html
----------------------------------------------------------------------
diff --git a/docs/streams.html b/docs/streams.html
index af1b2b0..3e9334b 100644
--- a/docs/streams.html
+++ b/docs/streams.html
@@ -35,6 +35,9 @@
                     <li><a href="#streams_dsl">High-level streams DSL</a>
                 </ul>
             </li>
+            <li>
+                <a href="#streams_upgrade">Upgrade guide and API changes</a>
+            </li>
         </ol>
 
         <h2><a id="streams_overview" href="#streams_overview">Overview</a></h2>
@@ -466,7 +469,88 @@
         in <code>StreamsConfig</code> before running it. A complete list of
         Kafka Streams configs can be found <a href="#streamsconfigs"><b>here</b></a>.
         </p>
+
+        <h2><a id="streams_upgrade" href="#upgrade">Upgrade guide and API changes</a></h2>
+
+        <h3><a id="streams_upgrade_1020" href="#upgrade_1020">Upgrading a Kafka
Streams Application</a></h3>
+
+        <h4>Upgrading from 0.10.1.x to 0.10.2.0</h4>
+        <p>
+        See <a href="../#upgrade_1020_streams">Upgrade Section</a> for details.
+        </p>
+
+        <h3><a id="streams_api_changes" href="#api_changes">Streams API changes
in 0.10.2.0</a></h3>
+        <li> New methods in <code>KafkaStreams</code>:
+            <ul>
+                <li> set a listener to react on application state change via <code>#setStateListener(StateListener
listener)</code> </li>
+                <li> retrieve the current application state via <code>#state()</code>
</li>
+                <li> retrieve the global metrics registry via <code>#metrics()</code>
</li>
+                <li> apply a timeout when closing an application via <code>#close(long
timeout, TimeUnit timeUnit)</code> </li>
+                <li> specify a custom indent when retrieving Kafka Streams information
via <code>#toString(String indent)</code> </li>
+            </ul>
+        </li>
+        <li> Parameter updates in <code>StreamsConfig</code>:
+            <ul>
+                <li> parameter <code>zookeeper.connect</code> was deprecated
</li>
+                <ul>
+                    <li> a Kafka Streams application does no longer interact with Zookeeper
for topic management but uses the new broker admin protocol
+                         (cf. <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-TopicAdminSchema.1">KIP-4,
Section "Topic Admin Schema"</a>) </li>
+                    <li> thus, parameter "zookeeper.connect" is ignored in 0.10.2 and
should be removed from <code>StreamsConfig</code> </li>
+                </ul>
+                <li> added many new parameters for metrics, security, and client configurations
</li>
+            </ul>
+        </li>
+        <li> Changes in <code>StreamsMetrics</code> interface:
+            <ul>
+                <li> removed methods: <code>#addLatencySensor()</code>
</li>
+                <li> added methods: <code>#addLatencyAndThroughputSensor()</code>,
<code>#addThroughputSensor()</code>, <code>#recordThroughput()</code>,
+                     <code>#addSensor()</code>, <code>#removeSensor()</code>
</li>
+            </ul>
+        </li>
+        <li> New methods in <code>TopologyBuilder</code>:
+            <ul>
+                <li> added overloads for <code>#addSource()</code> that
allow to define a <code>auto.offset.reset</code> policy per source node </li>
+                <li> added methods <code>#addGlobalStore()</code> to add
global <code>StateStore</code>s </li>
+            </ul>
+        </li>
+        <li> New methods in <code>KStreamBuilder</code>:
+            <ul>
+                <li> added overloads for <code>#stream()</code> and <code>#table()</code>
that allow to define a <code>auto.offset.reset</code> policy per input stream/table
</li>
+                <li> <code>#table()</code> always requires store name </li>
+                <li> added method <code>#globalKTable()</code> to create
a <code>GlobalKTable</code> </li>
+            </ul>
+        </li>
+        <li> New joins for <code>KStream</code>:
+            <ul>
+                <li> added overloads for <code>#join()</code> to join with
<code>KTable</code> </li>
+                <li> added overloads for <code>#join()</code> and <code>leftJoin()</code>
to join with <code>GlobalKTable</code> </li>
+            </ul>
+        </li>
+        <li> Aligned <code>null</code>-key handling for <code>KTable</code>
joins:
+            <ul>
+                <li> like all other KTable operations, <code>KTable-KTable</code>
joins do not throw an exception on <code>null</code> key records anymore, but
drop those records silently </li>
+            </ul>
+        </li>
+        <li> New window type <em>Session Windows</em>:
+            <ul>
+                <li> added class <code>SessionWindows</code> to specify
session windows </li>
+                <li> added overloads for <code>KGroupedStream</code> methods
<code>#count()</code>, <code>#reduce()</code>, and <code>#aggregate()</code>
+                     to allow session window aggregations </li>
+            </ul>
+        </li>
+        <li> Changes to <code>TimestampExtractor</code>:
+            <ul>
+                <li> method <code>#extract()</code> has a second parameter
now </li>
+                <li> new default timestamp extractor class <code>FailOnInvalidTimestamp</code>
+                     (it gives the same behavior as old (and removed) default extractor <code>ConsumerRecordTimestampExtractor</code>)
</li>
+                <li> new alternative timestamp extractor classes <code>LogAndSkipOnInvalidTimestamp</code>
and <code>UsePreviousTimeOnInvalidTimestamps</code> </li>
+            </ul>
+        </li>
+        <li> Relaxed type constraints of many DSL interfaces, classes, and methods
(cf. <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+Relax+Type+constraints+in+Kafka+Streams+API">KIP-100</a>).
</li>
+    </ul>
 </pre>
+
+
 </script>
 
 <!--#include virtual="../includes/_header.htm" -->

http://git-wip-us.apache.org/repos/asf/kafka/blob/d952d94e/docs/upgrade.html
----------------------------------------------------------------------
diff --git a/docs/upgrade.html b/docs/upgrade.html
index ebc61db..79c376d 100644
--- a/docs/upgrade.html
+++ b/docs/upgrade.html
@@ -47,14 +47,16 @@ Kafka cluster before upgrading your clients. Version 0.10.2 brokers support
0.8.
 
 <p><b>Note:</b> Bumping the protocol version and restarting can be done
any time after the brokers were upgraded. It does not have to be immediately after.
 
-<ol>
-    <li>Upgrading a Kafka Streams Applications:
-        <ul>
-            <li>You need to recompile your code. Just swapping the jar file will not
work and will break your appliation.</li>
-            <li>If you use a custom timestamp extractor, you will need to update this
code, because the <code>TimestampExtractor</code> interface was changed.</li>
-        </ul>
-    </li>
-</ol>
+<h5><a id="upgrade_1020_streams" href="#upgrade_1020_streams">Upgrading a Kafka
Streams Application</a></h5>
+<ul>
+    <li> Upgrading your Streams application from 0.10.1 to 0.10.2 does not require
a broker upgrade.
+         A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it
is not possible to connect to 0.10.0 brokers though). </li>
+    <li> You need to recompile your code. Just swapping the Kafka Streams library jar
file will not work and will break your application. </li>
+    <li> <code>KStreamBuilder#table()</code> always requires a store name.
</li>
+    <li> <code>KTable#through()</code> always requires a store name. </li>
+    <li> If you use a custom (i.e., user implemented) timestamp extractor, you will
need to update this code, because the <code>TimestampExtractor</code> interface
was changed. </li>
+    <li> If you register custom metrics, you will need to update this code, because
the <code>StreamsMetric</code> interface was changed. </li>
+</ul>
 
 <h5><a id="upgrade_1020_notable" href="#upgrade_1020_notable">Notable changes
in 0.10.2.0</a></h5>
 <ul>
@@ -67,6 +69,8 @@ Kafka cluster before upgrading your clients. Version 0.10.2 brokers support
0.8.
         A new close API with timeout has been added to <code>KafkaConsumer</code>
to control the maximum wait time.</li>
     <li>Multiple regular expressions separated by commas can be passed to MirrorMaker
with the new Java consumer via the --whitelist option. This
         makes the behaviour consistent with MirrorMaker when used the old Scala consumer.</li>
+    <li>Upgrading your Streams application from 0.10.1 to 0.10.2 does not require a
broker upgrade.
+        A Kafka Streams 0.10.2 application can connect to 0.10.2 and 0.10.1 brokers (it is
not possible to connect to 0.10.0 brokers though).</li>
     <li>The Zookeeper dependency was removed from the Streams API. The Streams API
now uses the Kafka protocol to manage internal topics instead of
         modifying Zookeeper directly. This eliminates the need for privileges to access Zookeeper
directly and "StreamsConfig.ZOOKEEPER_CONFIG"
         should not be set in the Streams app any more. If the Kafka cluster is secured, Streams
apps must have the required security privileges to create new topics.</li>


Mime
View raw message