flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mungeol Heo <mungeol....@gmail.com>
Subject Re: Is there any "1 hour ago" like option at HDFS sink?
Date Fri, 26 Dec 2014 02:34:16 GMT
Thank you, Paul.

It is working.

On Thu, Dec 25, 2014 at 1:52 AM, Paul Chavez <pchavez@ntent.com> wrote:
> You can use a regex extractor interceptor to create the time stamp header from your data.
>
>
>> On Dec 24, 2014, at 1:03 AM, Mungeol Heo <mungeol.heo@gmail.com> wrote:
>>
>> Hello,
>>
>> I try to use the configuration, which listed below, to transfer logs to HDFS.
>>
>> ...
>> agent01.sources.source01.interceptors = interceptor01
>> agent01.sources.source01.interceptors.interceptor01.type = timestamp
>> ...
>> agent01.sinks.sink01.type = hdfs
>> agent01.sinks.sink01.hdfs.path = /access/%Y%m%d/%H
>> ...
>>
>> The log files I have is like below, which have hourly logs.
>>
>> ...
>> access.2014122400
>> access.2014122401
>> access.2014122402
>> ...
>>
>> If the 'access.2014122402' file is transferred at 03:10, the HDFS path
>> will be '/access/20141224/03/"file name"
>> However, I need the path to be '/access/20141224/02/"file name"
>> instead of '/access/20141224/03/"file name".
>> I mean the hour information of HDFS path should be same with the hour
>> information of the file name or logs.
>>
>> Is there any way to get it done except changing 'hdfs.timeZone' setting?
>> Like, "1 hour ago" option of the date command in linux.
>> Any help will be great.
>> Thanks.
>>
>> - Mungeol

Mime
View raw message