flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Sammer <esam...@cloudera.com>
Subject Re: Using Flume agent on binary Avro file
Date Wed, 19 Oct 2011 17:55:38 GMT

There's no good way to tail an Avro file without writing a custom source.
Your options are probably to either move the complete file (without Flume)
or have the application write directly to the Avro source (using the log4j
appender or something similar).

On Wed, Oct 19, 2011 at 4:22 AM, Meyer, Dennis <dennis.meyer@adtech.com>wrote:

>   Hi,
>  We've got some use case I couldn't find good information about:
>  We use AVRO binary logging format and log the data out in binary
> compressed AVRO format as we want to kep the data small. Those files we'd
> like a Flume agent to listen and stream the data into hadoop. The current
> process needs the intermediate file – so no direct to Hadoop streaming is
> possible.
>  Is there any way to do this, would wonder if it's not supported, but
> can't find infos?
>  Thanks,
> Dennis

Eric Sammer
twitter: esammer
data: www.cloudera.com

View raw message