flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mungeol Heo <mungeol....@gmail.com>
Subject Is there any "1 hour ago" like option at HDFS sink?
Date Wed, 24 Dec 2014 09:01:51 GMT
Hello,

I try to use the configuration, which listed below, to transfer logs to HDFS.

...
agent01.sources.source01.interceptors = interceptor01
agent01.sources.source01.interceptors.interceptor01.type = timestamp
...
agent01.sinks.sink01.type = hdfs
agent01.sinks.sink01.hdfs.path = /access/%Y%m%d/%H
...

The log files I have is like below, which have hourly logs.

...
access.2014122400
access.2014122401
access.2014122402
...

If the 'access.2014122402' file is transferred at 03:10, the HDFS path
will be '/access/20141224/03/"file name"
However, I need the path to be '/access/20141224/02/"file name"
instead of '/access/20141224/03/"file name".
I mean the hour information of HDFS path should be same with the hour
information of the file name or logs.

Is there any way to get it done except changing 'hdfs.timeZone' setting?
Like, "1 hour ago" option of the date command in linux.
Any help will be great.
Thanks.

- Mungeol

Mime
View raw message