flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Meyer, Dennis" <dennis.me...@adtech.com>
Subject Re: Using Flume agent on binary Avro file
Date Thu, 20 Oct 2011 07:25:37 GMT

Would it be possible to write directly to the Flume TCP socket using the binary AVRO format?
This could be an option in our use-case.


Von: Eric Sammer <esammer@cloudera.com<mailto:esammer@cloudera.com>>
Antworten an: <flume-user@incubator.apache.org<mailto:flume-user@incubator.apache.org>>
Datum: Wed, 19 Oct 2011 10:55:38 -0700
An: <flume-user@incubator.apache.org<mailto:flume-user@incubator.apache.org>>
Betreff: Re: Using Flume agent on binary Avro file


There's no good way to tail an Avro file without writing a custom source. Your options are
probably to either move the complete file (without Flume) or have the application write directly
to the Avro source (using the log4j appender or something similar).

On Wed, Oct 19, 2011 at 4:22 AM, Meyer, Dennis <dennis.meyer@adtech.com<mailto:dennis.meyer@adtech.com>>

We've got some use case I couldn't find good information about:

We use AVRO binary logging format and log the data out in binary compressed AVRO format as
we want to kep the data small. Those files we'd like a Flume agent to listen and stream the
data into hadoop. The current process needs the intermediate file – so no direct to Hadoop
streaming is possible.

Is there any way to do this, would wonder if it's not supported, but can't find infos?


Eric Sammer
twitter: esammer
data: www.cloudera.com<http://www.cloudera.com>

View raw message