flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gonzalo Herreros <gherre...@gmail.com>
Subject Re: Avro source and sink
Date Tue, 15 Sep 2015 18:47:39 GMT
I'm not sure if I understand your topology and what you mean exactly by
"used Kafka channel/sink", it would help if you send the configuration.

My best guess about the error is that you are pointing the kafka source to
a topic that is used by a channel and not by a kafka sink


On Sep 15, 2015 6:42 PM, "Buntu Dev" <buntudev@gmail.com> wrote:

> Currently I have a single flume agent that converts apache logs into Avro
> and writes to HDFS sink. I'm looking for ways to create tiered topology and
> want to have the Avro records available to other flume agents. I used Kafka
> channel/sink to write these Avro records but was running into this error
> when using the Kafka source to read the records:
>  Caused by: java.io.IOException: Not a data file.
>     at
> org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105)
>     at org.apache.avro.file.DataFileReader.<init>(DataFileReader.java:97)
> For using tiered topology, should I be using Avro sink and write to
> host/port for other flume agent to read using Avro source? or is there any
> other data format that I should consider if I want to stick with Kafka as
> the channel/sink?
> Thanks!

View raw message