flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From higkoohk <higko...@gmail.com>
Subject What does the file header mean ? Flume always add headers to file header
Date Wed, 15 May 2013 10:05:36 GMT
My flume.conf

tengine.sources = tengine
> tengine.sources.tengine.type = exec
> tengine.sources.tengine.command = tail -n +0 -F
> /data/log/tengine/access.log
> tengine.sources.tengine.channels = file4log
> tengine.sinks = hdfs4log
> tengine.sinks.hdfs4log.type = hdfs
> tengine.sinks.hdfs4log.channel = file4log
> tengine.sinks.hdfs4log.serializer = avro_event
> tengine.sinks.hdfs4log.hdfs.path = hdfs://
> hdfs.kisops.org:8020/flume/tengine
> tengine.sinks.hdfs4log.hdfs.filePrefix = access
> tengine.sinks.hdfs4log.hdfs.fileSuffix = .log
> tengine.sinks.hdfs4log.hdfs.rollInterval = 0
> tengine.sinks.hdfs4log.hdfs.rollCount = 0
> tengine.sinks.hdfs4log.hdfs.rollSize = 134217728
> tengine.sinks.hdfs4log.hdfs.batchSize = 1024
> tengine.sinks.hdfs4log.hdfs.threadsPoolSize = 1
> tengine.sinks.hdfs4log.hdfs.fileType = DataStream
> tengine.sinks.hdfs4log.hdfs.writeFormat = Text
> tengine.channels = file4log
> tengine.channels.file4log.type = file
> tengine.channels.file4log.capacity = 4096
> tengine.channels.file4log.transactionCapacity = 1024
> tengine.channels.file4log.checkpointDir = /data/log/hdfs
> tengine.channels.file4log.dataDirs = /data/log/loadrunner


When I see the logs in hdfs , there are same headers in files which not
creater by app :

>
> Objavro.codecnullavro.schema�{"type":"record","name":"Event","fields":[{"name":"headers","type":{"type":"map","values":"string"}},{"name":"body","type":"bytes"}]}�"�,�)��E����5�Y���
> ��agent25.kisops.org|10.20.216.20|1368610557.341|200|207|255|GET
> /status?00000005 HTTP/1.1|0.000|52033467��


See the image :
[image: 内嵌图片 1]

What does it mean , how to remove it or when and how to use this info ?

Many thanks !

Mime
View raw message