flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harish Mandala <mvharish14...@gmail.com>
Subject Re: Transfering compressed (gzip) files
Date Mon, 22 Oct 2012 15:27:06 GMT

Which of the flume sources are you trying to use?


On Mon, Oct 22, 2012 at 11:18 AM, Sadananda Hegde <saduhegde@gmail.com>wrote:

> My application servers produce data files that are in compressed format
> (gzip). I am planning to use flume ng (1.2.0) to collect those files and
> transfer them to hadoop cluster (write to HDFS). Is it possible to read and
> transfer them without uncomressing first? My sink would be HDFS and there
> are options to compress before writing to HDFS. That would work fine if my
> source is uncompressed text file and need to store hdfs file in compressed
> format. But in my case, the source itself is compressed. What would be the
> best options to handle such cases?
> Thanks for your help.
> Sadu

View raw message