flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Li <litao.bupt...@gmail.com>
Subject Re: [KafkaSource] Write timestamp in the header
Date Tue, 14 Apr 2015 17:15:14 GMT
@Gwen Shapira, I think the feature customarize timestamp is useful,
especialy with HDFSEventSink. Do you consider to add this feature in the
future?

2015-04-15 0:55 GMT+08:00 Tao Li <litao.buptsse@gmail.com>:

> Code like this, KafkaSource will simply put "System.currentTimeMillis()"
> as "timestamp"
>
> 2015-04-15 0:52 GMT+08:00 Gwen Shapira <gshapira@cloudera.com>:
>
>> Are you sure KafkaSource does that? Or do you have an Interceptor?
>>
>> On Tue, Apr 14, 2015 at 9:50 AM, Tao Li <litao.buptsse@gmail.com> wrote:
>> > Hi all:
>> >
>> > I found that KafkaSource will put "System.currentTimeMillis()" as
>> > "timestamp" field in header. If we use HDFSEventSink, the HDFS log can
>> be
>> > rotate by the timestamp.
>> >
>> > But the web server log usually contains "timestamp" string, and it
>> indicate
>> > the true timestamp, other than "System.currentTimeMillis()". So I think
>> if
>> > have a feature to allow to customarize the "timestamp" field, will be
>> > better.
>>
>
>

Mime
View raw message