flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Majid Alfifi <majid.alf...@gmail.com>
Subject HDFS Sink keeping last temp file open
Date Tue, 24 Feb 2015 10:47:08 GMT
I have the following HDFS Sink configuration which rolls files based on
size. I am not able to get flume to close the last temp file before it
moves to the next directory.

Do the configuration options below seem right?

agent.sinks.HDFSSink.hdfs.rollInterval = 0
agent.sinks.HDFSSink.hdfs.rollSize = 512000000
agent.sinks.HDFSSink.hdfs.rollCount = 0
agent.sinks.HDFSSink.hdfs.batchSize = 10000
agent.sinks.HDFSSink.hdfs.fileType = CompressedStream
agent.sinks.HDFSSink.hdfs.codeC = snappy
agent.sinks.HDFSSink.hdfs.maxOpenFiles = 50
agent.sinks.HDFSSink.hdfs.appendTimeout = 10000
agent.sinks.HDFSSink.hdfs.callTimeout = 100
agent.sinks.HDFSSink.hdfs.threadsPoolSize = 100
agent.sinks.HDFSSink.hdfs.rollTimerPoolSize = 1Listing the files in
HDFS for two directories look like the following:


[majid@srv01 ~]$ hadoop fs -ls /user/monitor/incoming/2015/02/22/am/ | tail -5
-rw-r--r--   3 flume flume  129204066 2015-02-22 11:24
/user/monitor/incoming/2015/02/22/am/FlumeData.1424563206488.snappy
-rw-r--r--   3 flume flume  129129935 2015-02-22 11:33
/user/monitor/incoming/2015/02/22/am/FlumeData.1424563206489.snappy
-rw-r--r--   3 flume flume  129224836 2015-02-22 11:43
/user/monitor/incoming/2015/02/22/am/FlumeData.1424563206490.snappy
-rw-r--r--   3 flume flume  130160914 2015-02-22 11:54
/user/monitor/incoming/2015/02/22/am/FlumeData.1424563206491.snappy
-rw-r--r--   3 flume flume       5123 2015-02-22 11:54
/user/monitor/incoming/2015/02/22/am/FlumeData.1424563206492.snappy.tmp
[majid@srv01 ~]$ hadoop fs -ls /user/monitor/incoming/2015/02/22/pm/ | tail -5
-rw-r--r--   3 flume flume  128659488 2015-02-22 23:19
/user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408953.snappy
-rw-r--r--   3 flume flume  127512784 2015-02-22 23:30
/user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408954.snappy
-rw-r--r--   3 flume flume  128234258 2015-02-22 23:41
/user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408955.snappy
-rw-r--r--   3 flume flume  128191069 2015-02-22 23:53
/user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408956.snappy
-rw-r--r--   3 flume flume     818575 2015-02-22 23:53
/user/monitor/incoming/2015/02/22/pm/FlumeData.1424606408957.snappy.tmp


Thanks,
Majid

Mime
View raw message