flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shara Shi" <shiruih...@dhgate.com>
Subject 答复: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG
Date Wed, 04 Jul 2012 04:54:05 GMT
Hi 

 

It works well , thanks

 

Ruihong Shi

 

发件人: Juhani Connolly [mailto:juhani_connolly@cyberagent.co.jp] 
发送时间: 2012年7月4日 10:20
收件人: flume-user@incubator.apache.org
主题: Re: HOW TO USE %Y%m%d ESCAPE SEQUENCE IN FLUME-NG

 

To use the date related escape sequences you need to provide a timestamp
header.

The easiest way to do this would be configuring a timestamp interceptor onto
your source, ours looks something like:

sender1.sources.tail.interceptors = ts
sender1.sources.tail.interceptors.ts.type =
org.apache.flume.interceptor.TimestampInterceptor$Builder


On 07/04/2012 11:00 AM, Shara Shi wrote:

HI 

 

I failed to use /%Y/%m/%d/ ESCAPE SEQUENCE to roll file in hdfs. 

I got following error messages

 

12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed

java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)

        at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)

        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)

        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception
follows.

org.apache.flume.EventDeliveryException: java.lang.NumberFormatException:
null

        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)

        at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)

        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)

        at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)

        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        ... 3 more

12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed

java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)

        at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)

        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)

        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

12/07/04 09:59:16 ERROR flume.SinkRunner: Unable to deliver event. Exception
follows.

org.apache.flume.EventDeliveryException: java.lang.NumberFormatException:
null

        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)

        at
org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java
:68)

        at
org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at
org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.jav
a:220)

        at
org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:31
0)

        at
org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)

        ... 3 more

12/07/04 09:59:16 ERROR hdfs.HDFSEventSink: process failed

java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

 

=============

 

collector2.channels.ch2.type = memory

 

collector2.sources.avro-source1.channels = ch2

collector2.sources.avro-source1.type = avro

collector2.sources.avro-source1.bind = 0.0.0.0

collector2.sources.avro-source1.port = 41414

 

collector2.sinks.hdfs.channel = ch2

collector2.sinks.hdfs.type= hdfs

collector2.sinks.hdfs.hdfs.path=hdfs://namenode:8020/data/log/%Y/%m/%d/

collector2.sinks.hdfs.batchsize=1

collector2.sinks.hdfs.runner.type=polling

collector2.sinks.hdfs.runner.polling.interval = 60

 

collector2.sinks.hdfs.hdfs.rollInterval = 300

collector2.sinks.hdfs.hdfs.rollSize = 0

collector2.sinks.hdfs.hdfs.rollCount = 0

 

collector2.channels = ch2

collector2.sources = avro-source1

collector2.sinks = hdfs

 

 

Ruihong Shi

 

 


Mime
View raw message