flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: [Flume 1.7.0] 'parseAsFlumeEvent = false' works in Kafka channel?
Date Fri, 03 Jun 2016 09:06:03 GMT
The explanation could be clearer but it boils down to this:

-If you use parseAsEvent=true (default)
 Then the events in the channel are avro FlumeEvents. So you have to
read/write avro but you have all the object metadata (timestamp, headers,
etc)

-If you use parseAsEvent=false
  Then the events in the channel is just text (no metadata)

You have to choose either to use pure text or have a richer FlumeEvent
object.


On 3 June 2016 at 09:48, George M. <qazw3456@gmail.com> wrote:

> Hi Gonzalo,
>
>
> Seems the morphline is transforming the event into one without "body", are
> you converting the event body into headers?
>
>
> Yes. I am converting the event body into headers with Kite grok command
> with regex patterns. For example, for the event commented in previous
> messages, i'am creating unixtime, address, size, hostip, line_type and
> event_route headers.
>
>
> parseAsFlumeEvent only handles the body, the headers are lost.
>
>
> I do not find in the user guide any reference about that.
>
> "Expecting Avro datums with FlumeEvent schema in the channel. This should
> be true if Flume source is writing to the channel And false if other
> producers are writing into the topic that the channel is using Flume source
> messages to Kafka can be parsed outside of Flume by using
> org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk
> artifact"
>
> Maybe, is the description a little confusing?
>
> Are the headers lost? In the kafka message below, the headers appear but
> in a 'curious' way.
>
> Regards.
>
>
>
>
> On 2 June 2016 at 16:59, George M. <qazw3456@gmail.com> wrote:
>
> With the morphlines and without multiplexing
> ============================
> ------------------------------------------
> parseAsFlumeEvent = true
> ------------------------------------------
> unixtime 1420119195000address,example@example.cosiz1060
> <eruiz@kubiwireless.cosiz1060>
>                                                             hostip
> 10.1.0.170 line_type sender id 938223BD77C event_route SENDERmessage´┐Ż 2015
> Jan 1 14:33:15 10.1.0.170 postfix-dsp/qmgr[5605]: 938223BD77C: from=<
> example@example.com>, size=1060, nrcpt=1 (queue active)
>
> -------------------------------------------
> parseAsFlumeEvent = false
> -------------------------------------------
> Only new lines
>
>
>
>
>

Mime
View raw message