kafka-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gwens...@apache.org
Subject kafka-site git commit: SMT example, correct classnames in the generated transformations list and more Connect in TOC
Date Thu, 06 Apr 2017 22:45:26 GMT
Repository: kafka-site
Updated Branches:
  refs/heads/asf-site 3c617059d -> 749b548f3


SMT example, correct classnames in the generated transformations list and more Connect in
TOC


Project: http://git-wip-us.apache.org/repos/asf/kafka-site/repo
Commit: http://git-wip-us.apache.org/repos/asf/kafka-site/commit/749b548f
Tree: http://git-wip-us.apache.org/repos/asf/kafka-site/tree/749b548f
Diff: http://git-wip-us.apache.org/repos/asf/kafka-site/diff/749b548f

Branch: refs/heads/asf-site
Commit: 749b548f3c9b4b2d2e8478c0de78a45d78da889c
Parents: 3c61705
Author: Gwen Shapira <cshapi@gmail.com>
Authored: Thu Apr 6 11:44:48 2017 -0700
Committer: Gwen Shapira <cshapi@gmail.com>
Committed: Thu Apr 6 11:44:48 2017 -0700

----------------------------------------------------------------------
 0102/connect.html                      | 68 ++++++++++++++++++++++++++++-
 0102/generated/connect_transforms.html | 12 ++---
 0102/toc.html                          |  6 +++
 3 files changed, 79 insertions(+), 7 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kafka-site/blob/749b548f/0102/connect.html
----------------------------------------------------------------------
diff --git a/0102/connect.html b/0102/connect.html
index 1af5ed9..6ad06b8 100644
--- a/0102/connect.html
+++ b/0102/connect.html
@@ -102,7 +102,7 @@
 
     <h4><a id="connect_transforms" href="#connect_transforms">Transformations</a></h4>
 
-    Connectors can be configured with transformations to make lightweight message-at-a-time
modifications. They can be convenient for minor data massaging and routing changes.
+    Connectors can be configured with transformations to make lightweight message-at-a-time
modifications. They can be convenient for data massaging and event routing.
 
     A transformation chain can be specified in the connector configuration.
 
@@ -112,8 +112,74 @@
         <li><code>transforms.$alias.$transformationSpecificConfig</code>
Configuration properties for the transformation</li>
     </ul>
 
+    <p>For example, lets take the built-in file source connector and use a transformation
to add a static field.</p>
+
+    <p>Throughout the example we'll use schemaless JSON data format. To use schemaless
format, we changed the following two lines in <code>connect-standalone.properties</code>
from true to false:</p>
+
+    <pre>
+        key.converter.schemas.enable
+        value.converter.schemas.enable
+    </pre>
+
+    The file source connector reads each line as a String. We will wrap each line in a Map
and then add a second field to identify the origin of the event. To do this, we use two transformations:
+    <ul>
+        <li><b>HoistField</b> to place the input line inside a Map</li>
+        <li><b>InsertField</b> to add the static field. In this example
we'll indicate that the record came from a file connector</li>
+    </ul>
+
+    After adding the transformations, <code>connect-file-source.properties</code>
file looks as following:
+
+    <pre>
+        name=local-file-source
+        connector.class=FileStreamSource
+        tasks.max=1
+        file=test.txt
+        topic=connect-test
+        transforms=MakeMap, InsertSource
+        transforms.MakeMap.type=org.apache.kafka.connect.transforms.HoistField$Value
+        transforms.MakeMap.field=line
+        transforms.InsertSource.type=org.apache.kafka.connect.transforms.InsertField$Value
+        transforms.InsertSource.static.field=data_source
+        transforms.InsertSource.static.value=test-file-source
+    </pre>
+
+    <p>All the lines starting with <code>transforms</code> were added for
the transformations. You can see the two transformations we created: "InsertSource" and "MakeMap"
are aliases that we chose to give the transformations. The transformation types are based
on the list of built-in transformations you can see below. Each transformation type has additional
configuration: HoistField requires a configuration called "field", which is the name of the
field in the map that will include the original String from the file. InsertField transformation
lets us specify the field name and the value that we are adding.</p>
+
+    When we ran the file source connector on my sample file without the transformations,
and then read them using <code>kafka-console-consumer.sh</code>, the results were:
+
+    <pre>
+        "foo"
+        "bar"
+        "hello world"
+   </pre>
+
+    We then create a new file connector, this time after adding the transformations to the
configuration file. This time, the results will be:
+
+    <pre>
+        {"line":"foo","data_source":"test-file-source"}
+        {"line":"bar","data_source":"test-file-source"}
+        {"line":"hello world","data_source":"test-file-source"}
+    </pre>
+
+    You can see that the lines we've read are now part of a JSON map, and there is an extra
field with the static value we specified. This is just one example of what you can do with
transformations.
+
     Several widely-applicable data and routing transformations are included with Kafka Connect:
 
+    <ul>
+        <li>InsertField - Add a field using either static data or record metadata</li>
+        <li>ReplaceField - Filter or rename fields</li>
+        <li>MaskField - Replace field with valid null value for the type (0, empty
string, etc)</li>
+        <li>ValueToKey</li>
+        <li>HoistField - Wrap the entire event as a single field inside a Struct or
a Map</li>
+        <li>ExtractField - Extract a specific field from Struct and Map and include
only this field in results</li>
+        <li>SetSchemaMetadata - modify the schema name or version</li>
+        <li>TimestampRouter - Modify the topic of a record based on original topic
and timestamp. Useful when using a sink that needs to write to different tables or indexes
based on timestamps</li>
+        <li>RegexpRouter - modify the topic of a record based on original topic, replacement
string and a regular expression</li>
+    </ul>
+
+    Details on how to configure each transformation are listed below:
+
+
     <!--#include virtual="generated/connect_transforms.html" -->
 
     <h4><a id="connect_rest" href="#connect_rest">REST API</a></h4>

http://git-wip-us.apache.org/repos/asf/kafka-site/blob/749b548f/0102/generated/connect_transforms.html
----------------------------------------------------------------------
diff --git a/0102/generated/connect_transforms.html b/0102/generated/connect_transforms.html
index 624bec4..924c8da 100644
--- a/0102/generated/connect_transforms.html
+++ b/0102/generated/connect_transforms.html
@@ -1,6 +1,6 @@
 <div id="org.apache.kafka.connect.transforms.InsertField">
 <h5>org.apache.kafka.connect.transforms.InsertField</h5>
-Insert field(s) using attributes from the record metadata or a configured static value.<p/>Use
the concrete transformation type designed for the record key (<code>org.apache.kafka.connect.transforms.InsertField.Key</code>)
or value (<code>org.apache.kafka.connect.transforms.InsertField.Value</code>).
+Insert field(s) using attributes from the record metadata or a configured static value.<p/>Use
the concrete transformation type designed for the record key (<code>org.apache.kafka.connect.transforms.InsertField$Key</code>)
or value (<code>org.apache.kafka.connect.transforms.InsertField$Value</code>).
 <p/>
 <table class="data-table"><tbody>
 <tr>
@@ -27,7 +27,7 @@ Insert field(s) using attributes from the record metadata or a configured
static
 </div>
 <div id="org.apache.kafka.connect.transforms.ReplaceField">
 <h5>org.apache.kafka.connect.transforms.ReplaceField</h5>
-Filter or rename fields.
+Filter or rename fields.<p/>Use the concrete transformation type designed for the record
key (<code>org.apache.kafka.connect.transforms.ReplaceField$Key</code>) or value
(<code>org.apache.kafka.connect.transforms.ReplaceField$Value</code>).
 <p/>
 <table class="data-table"><tbody>
 <tr>
@@ -48,7 +48,7 @@ Filter or rename fields.
 </div>
 <div id="org.apache.kafka.connect.transforms.MaskField">
 <h5>org.apache.kafka.connect.transforms.MaskField</h5>
-Mask specified fields with a valid null value for the field type (i.e. 0, false, empty string,
and so on).<p/>Use the concrete transformation type designed for the record key (<code>org.apache.kafka.connect.transforms.MaskField.Key</code>)
or value (<code>org.apache.kafka.connect.transforms.MaskField.Value</code>).
+Mask specified fields with a valid null value for the field type (i.e. 0, false, empty string,
and so on).<p/>Use the concrete transformation type designed for the record key (<code>org.apache.kafka.connect.transforms.MaskField$Key</code>)
or value (<code>org.apache.kafka.connect.transforms.MaskField$Value</code>).
 <p/>
 <table class="data-table"><tbody>
 <tr>
@@ -82,7 +82,7 @@ Replace the record key with a new key formed from a subset of fields in
the reco
 </div>
 <div id="org.apache.kafka.connect.transforms.HoistField">
 <h5>org.apache.kafka.connect.transforms.HoistField</h5>
-Wrap data using the specified field name in a Struct when schema present, or a Map in the
case of schemaless data.<p/>Use the concrete transformation type designed for the record
key (<code>org.apache.kafka.connect.transforms.HoistField.Key</code>) or value
(<code>org.apache.kafka.connect.transforms.HoistField.Value</code>).
+Wrap data using the specified field name in a Struct when schema present, or a Map in the
case of schemaless data.<p/>Use the concrete transformation type designed for the record
key (<code>org.apache.kafka.connect.transforms.HoistField$Key</code>) or value
(<code>org.apache.kafka.connect.transforms.HoistField$Value</code>).
 <p/>
 <table class="data-table"><tbody>
 <tr>
@@ -99,7 +99,7 @@ Wrap data using the specified field name in a Struct when schema present,
or a M
 </div>
 <div id="org.apache.kafka.connect.transforms.ExtractField">
 <h5>org.apache.kafka.connect.transforms.ExtractField</h5>
-Extract the specified field from a Struct when schema present, or a Map in the case of schemaless
data.<p/>Use the concrete transformation type designed for the record key (<code>org.apache.kafka.connect.transforms.ExtractField.Key</code>)
or value (<code>org.apache.kafka.connect.transforms.ExtractField.Value</code>).
+Extract the specified field from a Struct when schema present, or a Map in the case of schemaless
data.<p/>Use the concrete transformation type designed for the record key (<code>org.apache.kafka.connect.transforms.ExtractField$Key</code>)
or value (<code>org.apache.kafka.connect.transforms.ExtractField$Value</code>).
 <p/>
 <table class="data-table"><tbody>
 <tr>
@@ -116,7 +116,7 @@ Extract the specified field from a Struct when schema present, or a Map
in the c
 </div>
 <div id="org.apache.kafka.connect.transforms.SetSchemaMetadata">
 <h5>org.apache.kafka.connect.transforms.SetSchemaMetadata</h5>
-Set the schema name, version or both on the record's key (<code>org.apache.kafka.connect.transforms.SetSchemaMetadata.Key</code>)
or value (<code>org.apache.kafka.connect.transforms.SetSchemaMetadata.Value</code>)
schema.
+Set the schema name, version or both on the record's key (<code>org.apache.kafka.connect.transforms.SetSchemaMetadata$Key</code>)
or value (<code>org.apache.kafka.connect.transforms.SetSchemaMetadata$Value</code>)
schema.
 <p/>
 <table class="data-table"><tbody>
 <tr>

http://git-wip-us.apache.org/repos/asf/kafka-site/blob/749b548f/0102/toc.html
----------------------------------------------------------------------
diff --git a/0102/toc.html b/0102/toc.html
index 787153d..935703b 100644
--- a/0102/toc.html
+++ b/0102/toc.html
@@ -130,6 +130,12 @@
             <ul>
                 <li><a href="#connect_overview">8.1 Overview</a></li>
                 <li><a href="#connect_user">8.2 User Guide</a></li>
+                <ul>
+                    <li><a href="#connect_running">Running Kafka Connect</a></li>
+                    <li><a href="#connect_configuring">Configuring Connectors</a></li>
+                    <li><a href="#connect_transforms">Transformations</a></li>
+                    <li><a href="#connect_rest">REST API</a></li>
+                </ul>
                 <li><a href="#connect_development">8.3 Connector Development
Guide</a></li>
             </ul>
         </li>


Mime
View raw message