flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shang Wu (CS)" <Shang...@symantec.com>
Subject Re: Define and leverage custom constants in agent configuration
Date Thu, 01 Dec 2016 18:33:33 GMT
Hi Robert,
Looks like what you want is more like a global configuration, and based on my limited knowledge
in Apache Flume, it is missing now.
From: Roberto Coluccio <roberto.coluccio@eng.it>
Reply-To: "user@flume.apache.org" <user@flume.apache.org>
Date: Thursday, December 1, 2016 at 2:59 AM
To: "user@flume.apache.org" <user@flume.apache.org>
Subject: Re: Define and leverage custom constants in agent configuration

Hi Silvio,

thanks for your suggestion. It gives me a workaround, but it's not exactly what I was hoping/asking

I'm going to assume what I'd want to do being just not doable.

Thank you,


On 29/11/2016 09:26, Silvio Di gregorio wrote:
with the "static interceptor" you can add key/value pairs in your flume event header:

agentOne.sources.s1.interceptors = interOne
agentOne.sources.s1.interceptors.interOne.type = static
agentOne.sources.s1.interceptors.interOne.key = costante
agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi

I hope it is useful
have nice day

2016-11-28 18:26 GMT+01:00 Roberto Coluccio <roberto.coluccio@eng.it<mailto:roberto.coluccio@eng.it>>:
Hello folks,

I was wondering if it's possible to define custom constants in a Flume agent configuration
file, so to avoid errors and replicated stuff when not needed. An example to make my point

assume I want to launch an agent of type myAgent configured with file channel and hdfs sink.

I want the file channel to persist data and checkpoint into "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"

Also, I want the hdfs sink to write files with prefix "myAgent1"

In this scenario, I should harcorde the "myAgent1" string 3 times, one for each configuration
parameter. This is error prone and kinda frustrating.

Is there a way to define a constant like "myConstant" in the configuration file, and leverage
it as parameter in the other configuration lines?


myConstant = myAgent1
myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}

It seems like the use of flume events header keys, but I don't want to add interceptors or
other components just to have parameters in my config file.

Thanks for your help.

Best regards,




Roberto Coluccio
Solution Designer
Direzione Ricerca e Innovazione
Interno 604135 | Esterno +39 0510435135

Engineering Ingegneria Informatica spa
Via Marconi, 10, 40122 Bologna

View raw message