flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Ryan <jur...@ziprealty.com>
Subject Re: Avro source: could not find schema for event
Date Tue, 08 Mar 2016 21:05:26 GMT
Hiya folks, still struggling with this, is anyone on the list familiar with
AvroEventSerializer$Builder ?

While I have gotten past my outright failure, I¹ve only done so by adopting
a fairly inflexible schema, which seems counter to the goal of using avro.
Particularly frustrating is that flume simply needs to pass the existing
message along, though I understand it likely needs to grok to separate
messages.  I can¹t even find Kafka consumer code which is capable of being

From:  Justin Ryan <juryan@ziprealty.com>
Reply-To:  <user@flume.apache.org>
Date:  Thursday, March 3, 2016 at 2:08 PM
To:  <user@flume.apache.org>
Subject:  Re: Avro source: could not find schema for event


So, I changed my serializer from
org.apache.flume.sink.hdfs.AvroEventSerializer$Builder to avro_event, and
this started working.  Well, working-ish, the data is a little funky but
it¹s arriving, being delivered to HDFS, and I can pull a file and examine it

I seem to remember that I had the former based on some things I read about
not having to specify a schema, since the schema is embedded in the avro

So I¹m confused, it seems that my previous configuration should have worked
without any special attention to the schema, but I got complaints that the
schema couldn¹t be found.

If anyone could shed a bit of light here, it would be much appreciated.

From:  Justin Ryan <juryan@ziprealty.com>
Reply-To:  <user@flume.apache.org>
Date:  Monday, February 29, 2016 at 2:52 PM
To:  "user@flume.apache.org" <user@flume.apache.org>
Subject:  Avro source: could not find schema for event


I¹ve got a fairly simply flume agent pulling events from kafka and landing
them in HDFS.  For plain text messages, this works fine.

I created a topic specifically for the purpose of testing sending avro
messages through kafka to land in HDFS, which I¹m having some trouble with.

I noted from 
p/ the example of flume¹s default avro schema[0], which will do for my
testing, and set up my python-avro producer to send messages with this
schema.  Unfortunately, I still have flume looping this message in its¹ log:

  org.apache.flume.FlumeException: Could not find schema for event

I¹m running out of assumptions to rethink / verify here, would appreciate
any guidance on what I may be missing..

Thanks in advance,


[0] {
 "type": "record",
 "name": "Event",
 "fields": [{
   "name": "headers",
   "type": {
     "type": "map",
     "values": "string"
 }, {
   "name": "body",
   "type": "bytes"

View raw message