flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "George M." <qazw3...@gmail.com>
Subject Re: [Flume 1.7.0] 'parseAsFlumeEvent = false' works in Kafka channel?
Date Fri, 03 Jun 2016 08:48:44 GMT
Hi Gonzalo,

> Seems the morphline is transforming the event into one without "body",
> are you converting the event body into headers?

Yes. I am converting the event body into headers with Kite grok command
with regex patterns. For example, for the event commented in previous
messages, i'am creating unixtime, address, size, hostip, line_type and
event_route headers.

> parseAsFlumeEvent only handles the body, the headers are lost.

I do not find in the user guide any reference about that.

"Expecting Avro datums with FlumeEvent schema in the channel. This
should be true if Flume source is writing to the channel And false if
other producers are writing into the topic that the channel is using
Flume source messages to Kafka can be parsed outside of Flume by using
org.apache.flume.source.avro.AvroFlumeEvent provided by the flume-ng-sdk

Maybe, is the description a little confusing?

Are the headers lost? In the kafka message below, the headers appear but
in a 'curious' way.


> On 2 June 2016 at 16:59, George M. <qazw3456@gmail.com> wrote:
>         With the morphlines and without multiplexing
>         ============================
>         ------------------------------------------
>         parseAsFlumeEvent = true
>         ------------------------------------------
>         unixtime1420119195000address,example@example.cosiz1060
>         hostip10.1.0.170line_type senderid938223BD77Cevent_route
>         SENDERmessage´┐Ż2015 Jan 1 14:33:15
>         postfix-dsp/qmgr[5605]: 938223BD77C:
>         from=<example@example.com>, size=1060, nrcpt=1 (queue active)
>         -------------------------------------------
>         parseAsFlumeEvent = false
>         -------------------------------------------
>         Only new lines

View raw message