flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Chavez <pcha...@ntent.com>
Subject Re: Is there any "1 hour ago" like option at HDFS sink?
Date Wed, 24 Dec 2014 16:52:09 GMT
You can use a regex extractor interceptor to create the time stamp header from your data.


> On Dec 24, 2014, at 1:03 AM, Mungeol Heo <mungeol.heo@gmail.com> wrote:
> 
> Hello,
> 
> I try to use the configuration, which listed below, to transfer logs to HDFS.
> 
> ...
> agent01.sources.source01.interceptors = interceptor01
> agent01.sources.source01.interceptors.interceptor01.type = timestamp
> ...
> agent01.sinks.sink01.type = hdfs
> agent01.sinks.sink01.hdfs.path = /access/%Y%m%d/%H
> ...
> 
> The log files I have is like below, which have hourly logs.
> 
> ...
> access.2014122400
> access.2014122401
> access.2014122402
> ...
> 
> If the 'access.2014122402' file is transferred at 03:10, the HDFS path
> will be '/access/20141224/03/"file name"
> However, I need the path to be '/access/20141224/02/"file name"
> instead of '/access/20141224/03/"file name".
> I mean the hour information of HDFS path should be same with the hour
> information of the file name or logs.
> 
> Is there any way to get it done except changing 'hdfs.timeZone' setting?
> Like, "1 hour ago" option of the date command in linux.
> Any help will be great.
> Thanks.
> 
> - Mungeol

Mime
View raw message