flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Sicker <boa...@gmail.com>
Subject Re: Append existing Avro file - HDFS Sink
Date Fri, 20 Apr 2018 19:13:11 GMT
It's not a Flume native solution, but an alternative I used in the past was
Kafka Connect using the HDFS connector plugin. That plugin provides
configuration regarding how often to roll over Avro files.

On 20 April 2018 at 13:49, Nitin Kumar <nitin.kumar2512@gmail.com> wrote:

> Hi All,
> I am using Flume v1.8 in which Flume agent comprises of Kafka Channel &
> HDFS Sink.
> I am able to write data in Avro file on HDFS into a external HIVE table,
> but the problem is whenever Flume gets restarted it closes that file and
> open a new file because of which I can see many small files. (Data is
> partition on the basis of date)
> Can't Flume append to existing file to avoid creation of new file?
> Also, how can I solve this problem which leads to creation of too many
> small files?
> Any help would be appreciated.
> --
> *Regards,Nitin Kumar*

Matt Sicker <boards@gmail.com>

View raw message