flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Kumar <nitin.kumar2...@gmail.com>
Subject Append existing Avro file - HDFS Sink
Date Fri, 20 Apr 2018 18:49:32 GMT
Hi All,

I am using Flume v1.8 in which Flume agent comprises of Kafka Channel &
HDFS Sink.
I am able to write data in Avro file on HDFS into a external HIVE table,
but the problem is whenever Flume gets restarted it closes that file and
open a new file because of which I can see many small files. (Data is
partition on the basis of date)

Can't Flume append to existing file to avoid creation of new file?
Also, how can I solve this problem which leads to creation of too many
small files?

Any help would be appreciated.


*Regards,Nitin Kumar*

View raw message