flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Schroer <cschr...@autoscout24.com>
Subject Problems with time variables in HDFS path
Date Wed, 11 Jul 2012 09:41:32 GMT
Hi,

we are running into a strange problem using Flume-NG 1.10 from CDH 4.0.1.

Setup:
Flume-NG opens a TCP syslog port, collects all messages and forwards them directly into HDFS.
This works fine until the point where we want to forward MS IIS Logs in W3C format. The reason
seems to be a " - " inside the log message. I could reproduce the problem using rsyslogd forwarding
all syslog messages to flume:

logger "Hello this is a test" => Works fine :)

logger "hello - this will break" => breaks flume :(

If I remove the time variables from the HDFS path in our configuration (attached) everything
is working fine...

Exception:

2012-07-11 11:08:18,292 ERROR hdfs.HDFSEventSink: process failed
java.lang.NumberFormatException: null
        at java.lang.Long.parseLong(Long.java:375)
        at java.lang.Long.valueOf(Long.java:525)
        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
2012-07-11 11:08:18,294 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.NumberFormatException: null
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:469)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NumberFormatException: null
        at java.lang.Long.parseLong(Long.java:375)
        at java.lang.Long.valueOf(Long.java:525)
        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:220)
        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:310)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:402)
        ... 3 more

I attached our configuration in case something is broken in there.

Best regards,

Christian Schroer

-- 
AutoScout24 GmbH
Dingolfinger Str. 1-15
81673 München

E-Mail    cschroer@autoscout24.com
Internet www.autoscout24.de
________________________________________________________________________________________
Geschäftsführer: Alberto Sanz de Lama, André Stark
Vorsitzender des Beirats: Dr. Martin Enderle | Amtsgericht München HRB 128 701






Mime
View raw message