flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Lee <defeng...@yottaa.com>
Subject How to write full event include headers to hdfs sink
Date Sat, 01 Sep 2012 09:32:33 GMT
Hi all,

When i use Flume-OG, hdfs sink will get full event like follow text.

head -1 varnishncsa.log | python -mjson.tool
{
    "body": "xxxxxxxxx",
    "fields": {
        "rolltag": "20120427-234205010-0400.418723114906325.00004966",
        "tailSrcFile": "varnishncsa.log"
    },
    "host": "ip-10-170-147-62.compute.internal",
    "nanos": 5164871621338860,
    "pri": "INFO",
    "timestamp": 1335584717134
}
But when i using flume-NG. If i’m use a logger sink, it will give me full event, like bellow.

collector config:

agent.sinks.hdfs-write.type = logger
agent config:

agent.sources.exec-agent.interceptors.hostname.type = org.apache.flume.interceptor.HostInterceptor$Builder
agent.sources.exec-agent.interceptors.hostname.preserveExisting = true
agent.sources.exec-agent.interceptors.hostname.useIP = false
12/09/01 07:51:41 INFO sink.LoggerSink: Event: { headers:{timestamp=1346485898919, host=ip-10-34-4-55.ec2.internal}
body: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 aaaaaaaaaaaaaaaa }

But when i use hdfs sink, i can’t get the full event, thanks in advance for any help.

Thanks, 
- Kevin
Mime
View raw message