flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhun Shen <shenzhunal...@gmail.com>
Subject Add hostname when sinking nginx log to kafka
Date Sun, 19 Jul 2015 10:10:39 GMT
Hello,
I'm using Flume 1.6 to collect nginx log then sink the log to kafka. I want
to add hostname of nginx when I sink the log to kafka in order to analyse
the web traffic of different hosts.

Here is my flume configuration file:
a1.sources = r1
a1.channels = c1

a1.sources.r1.type = exec
a1.sources.r1.channels = c1
a1.sources.r1.command = tail -F /data/tmp/cs.log
a1.sources.r1.interceptors = i1
a1.sources.r1.interceptors.i1.type = host
a1.sources.r1.interceptors.i1.hostHeader = hostname

a1.sinks = s1
a1.sinks.s1.channel = c1
a1.sinks.s1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.s1.zk.connect = 10.5.2.129:2181,10.5.2.127:2181,10.5.2.131:2181,
10.5.2.130:2181,10.5.2.132:2181
a1.sinks.s1.batchsize = 10
a1.sinks.s1.topic = testflume
a1.sinks.s1.key = test
a1.sinks.s1.brokerList = 10.5.2.112:9092,10.5.2.113:9092
a1.sinks.s1.requiredAcks = -1

a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

But it did not work, the key was still null. I could not find the hostname
anywhere.

{Event} testflume : null : 83.38.189.80 - - [12/Nov/2014:05:05:02 -0800]
"GET /es/index.php?main_page=checkout_login HTTP/1.0" 302 0 "
http://www.lightinthebox.com/es/index.php?main_page=shopping_cart"
"Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/38.0.2125.111 Safari/537.36"

If you know how to solve it, let me know.

Many thanks.


--
Shen Zhun (Allen)
Data Mining at LightnInTheBox.com
Email: shenzhunallen@gmail.com
GitHub: https://github.com/shenzhun
LinkedIn: http://www.linkedin.com/in/shenzhun

Mime
View raw message