flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From larryzhang <zhangt...@gmail.com>
Subject Exec source doesn't flush the last data
Date Wed, 13 Mar 2013 03:33:38 GMT
    I did a simple test about exec source, and found it didn't flush the 
last data. Here's the steps:
*a. create the source file 1.test, which has sequence number from 1 to 
15, like this:*
*b. create the configure file flume_simple.conf like this:*
        a1.sources = r1
a1.channels = c1
a1.sinks = k1

a1.sources.r1.type = exec
a1.sources.r1.command = tail -n +0 -F 
a1.sources.r1.channels = c1
a1.sources.r1.batchSize = 10

a1.channels.c1.type = memory

a1.sinks.k1.type = file_roll
a1.sinks.k1.channel = c1
a1.sinks.k1.sink.directory = /opt/scripts/tvhadoop/flume/flume-1.3.0/sink
*c. run flume with command: *
              bin/flume-ng agent --conf conf -f conf/flume_simple.conf 
-Dflume.root.logger=DEBUG,console -n a1

      After more than 1 minute(file roll interval), I check the output 
directory, there are 2 files, one has number from 1 to 10, and the other 
has nothing.
*I think this is because the batchSize was set to 10, the the last 5 
numbers didn't get flushed and lost.* Even I apply the patch in 
'https://issues.apache.org/jira/browse/FLUME-1819', nothing changed. If 
I debug into the code, *I found the red codes outside while clause never 
get executed*.
           while ((line = reader.readLine()) != null) {
             if(eventList.size() >= bufferCount) {
if(!eventList.isEmpty()) {
     In my scenario, the source log files are divided by hour, so I need 
to change the file name in flume configure file. Because of the above 
bug, I can only set the batchsize of execSource to 1, which 
significantly slowdown the through pass.   I wonder how to solve this 
problem. Any suggestions are most welcomed.
Best Regards,

View raw message