flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roberto Coluccio <roberto.coluc...@eng.it>
Subject Define and leverage custom constants in agent configuration
Date Mon, 28 Nov 2016 17:26:05 GMT
Hello folks,

I was wondering if it's possible to define custom constants in a Flume 
agent configuration file, so to avoid errors and replicated stuff when 
not needed. An example to make my point clear:

assume I want to launch an agent of type /myAgent/ configured with file 
channel and hdfs sink.

I want the file channel to persist data and checkpoint into 
"~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"

Also, I want the hdfs sink to write files with prefix "myAgent1"

In this scenario, I should harcorde the "myAgent1" string 3 times, one 
for each configuration parameter. This is error prone and kinda frustrating.

Is there a way to define a constant like "myConstant" in the 
configuration file, and leverage it as parameter in the other 
configuration lines?

e.g.

myConstant = myAgent1
myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}

It seems like the use of flume events header keys, but I don't want to 
add interceptors or other components just to have parameters in my 
config file.

Thanks for your help.

Best regards,

Roberto



Mime
View raw message