flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 周梦想 <abloz...@gmail.com>
Subject Re: why flume agent do not support --fileheader
Date Fri, 22 Feb 2013 04:30:01 GMT
Yes,I mean all event headers send from a specific agent. some headers need
for all events, not only hostname. now I have to configure it on every
source time and time again.
for example:
if i configure 10 sources, every source need 5 header,  I have to configure
50 times.
other configure of the same type source have the same problem.

agent1.sources =gamelog src1 src2 ... src10

agent1.sources.gamelog.interceptors = i1 i2 i3
#for %{host} org.apache.flume.interceptor.HostInterceptor$Builder
agent1.sources.gamelog.interceptors.i1.type = host
agent1.sources.gamelog.interceptors.i2.type = timestamp
agent1.sources.gamelog.interceptors.i3.type = static
agent1.sources.gamelog.interceptors.i3.key = filename
agent1.sources.gamelog.interceptors.i3.value = gamelog

*repleat configure for src1,...src10 for 10 or even more times.*

it's very boring.
maybe we could group the sources,sinks and configure them one or several

Best Regards,

2013/2/22 Juhani Connolly <juhani_connolly@cyberagent.co.jp>

> If I understand your question, using a static interceptor is the expected
> way to do this.
> Not sure what you mean by agent header? Do you mean all event headers sent
> from a specific agent? I don't imagine we will be adding a command-line
> parameter to do this, it wouldn't be consistent and would be superfluous.
> If you want to reuse a configuration file, and add a header to inform what
> agent it came from, perhaps you could use the hostname interceptor?
> On 02/22/2013 12:06 PM, 周梦想 wrote:
>> I want add some key/value to the agent header, but it's not convenient to
>> do so. why not flume agent support --headerFile just like flume avro-client?
>> my requirement is :
>> I want use spooling source to send files to another flume node to write
>> to hdfs.
>> and i want to name hdfs file to have the original file name prefix, not
>> the full path of the orgin file.
>> now I have to add interceptors of conf to do that.
>> so I have the question, why not flume agent support --headerFile just
>> like flume avro-client?
>> Best Regards,
>> Andy

View raw message