flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjie Lai <mjla...@gmail.com>
Subject Re: flume flush every 4k when use escapedFormatDfs
Date Mon, 10 Oct 2011 18:27:23 GMT
roll sink only closes and reopen its subsink. It's not aware of the 
implementation of subsink.

Right now, everytime escapedFormatDfs is opened, it just overwrites an 
existing file instead of appending it if the given file exists.

I remember reading from somewhere that the hdfs append will be supported 
in flume in (near) future (after apache hadoop starts to support 
append). But not now.

You can write to a new file by the combination of roll and hdfs by:

roll(1000) [ console, 
escapedCustomDfs("hdfs://namenode/flume/file-%{rolltag}") ]

-mingjie

On 10/10/2011 02:05 AM, 勇哲桂 wrote:
> when use
>
> roll(15000),
>
> reopen the logfile, will overwrite the file,
>
> can roll append the file ?
>
> 2011/10/10 勇哲桂 <peanutgyz@gmail.com <mailto:peanutgyz@gmail.com>>
>
>     i use escapedFormatDfs("file:://mylogdir","logfilename", raw)
>
>
>     then , when buffer is full, logfile will be udpated , each time 4k
>     content .
>
>
>     i thought there should be a period time , i want file to be updated
>     every 15s.
>
>
>     can someone tell me how to config this time ?
>
>
>

Mime
View raw message