flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Kumar <nitin.kumar2...@gmail.com>
Subject Re: Append existing Avro file - HDFS Sink
Date Wed, 02 May 2018 17:38:52 GMT
Thanks Matt

On Sat, Apr 21, 2018 at 12:43 AM, Matt Sicker <boards@gmail.com> wrote:

> It's not a Flume native solution, but an alternative I used in the past
> was Kafka Connect using the HDFS connector plugin. That plugin provides
> configuration regarding how often to roll over Avro files.
>
> On 20 April 2018 at 13:49, Nitin Kumar <nitin.kumar2512@gmail.com> wrote:
>
>> Hi All,
>>
>> I am using Flume v1.8 in which Flume agent comprises of Kafka Channel &
>> HDFS Sink.
>> I am able to write data in Avro file on HDFS into a external HIVE table,
>> but the problem is whenever Flume gets restarted it closes that file and
>> open a new file because of which I can see many small files. (Data is
>> partition on the basis of date)
>>
>> Can't Flume append to existing file to avoid creation of new file?
>> Also, how can I solve this problem which leads to creation of too many
>> small files?
>>
>> Any help would be appreciated.
>>
>> --
>>
>> *Regards,Nitin Kumar*
>>
>
>
>
> --
> Matt Sicker <boards@gmail.com>
>



-- 
*Regards,Nitin Kumar Choudhary*

Mime
View raw message