flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Workman <justinjwork...@gmail.com>
Subject Re: hdfs.idleTime
Date Tue, 17 Jan 2017 08:00:06 GMT
This was posted and resolved on the user thread. Typo in my configuration was the issue. 

Thanks
Justin

Sent from my iPhone

> On Jan 17, 2017, at 12:42 AM, Tristan Stevens <tristan@cloudera.com> wrote:
> 
> Hi Justin,
> Please can you post your agent config and also any HDFS logs? Ideally you should be seeing
INFO logs as follows: “Closing Idle Bucketwriter”.
> 
> Tristan
> 
> Tristan Stevens
> Senior Solutions Architect
> Cloudera, Inc. | www.cloudera.com
> m +44(0)7808 986422 | tristan@cloudera.com
> 
> <hadoop10.png>	Celebrating a decade of community accomplishments
> cloudera.com/hadoop10
> #hadoop10
> 
>> On 12 January 2017 at 19:23:18, Justin Workman (justinjworkman@gmail.com) wrote:
>> 
>> More details 
>> 
>> Flume 1.6 - Core Apache version. 
>> KafkaSource (0.8.2) -> File Channel -> HDFS Sink (CDH5.5.2). 
>> 
>> On Thu, Jan 12, 2017 at 12:20 PM, Justin Workman <justinjworkman@gmail.com>

>> wrote: 
>> 
>> > sorry for cross posting to user and dev. I have recently set up a flume 
>> > configuration where we are using the regex_extractor interceptor to parse 
>> > the actual event date from the record flowing through the Flume source, 
>> > then using that date to build the HDFS sink bucket path. However, it 
>> > appears that the hdfs.idleTimeout value is not honored in this 
>> > configuration. It does work when using the timestamp interceptor you build 
>> > the output path. 
>> > 
>> > I have set the hdfs.idleTimeout value for the HDFS sink, but the files are 
>> > never closed or renamed until I restart or shutdown Flume. Our flume is 
>> > configured to roll based on size or output path, and the files 
>> > rename/close/roll fine based on size, however the last file in each output 
>> > path is always left with the .tmp extension until we restart Flume. I would

>> > expect that the file would be renamed and closed if there are no records 
>> > written to this file after the idleTimeout is reached. 
>> > 
>> > Could I be missing something, or is this a known bug with the 
>> > regex_extract interceptor? 
>> > 
>> > Thanks 
>> > Justin 
>> > 

Mime
View raw message