flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Silvio Di gregorio <silvio.digrego...@gmail.com>
Subject Re: Define and leverage custom constants in agent configuration
Date Tue, 29 Nov 2016 08:26:09 GMT
with the "static interceptor" you can add key/value pairs in your flume
event header:

agentOne.sources.s1.interceptors = interOne
agentOne.sources.s1.interceptors.interOne.type = static
agentOne.sources.s1.interceptors.interOne.key = costante
agentOne.sources.s1.interceptors.interOne.value = quello_che_vuoi

I hope it is useful
have nice day
Silvio

2016-11-28 18:26 GMT+01:00 Roberto Coluccio <roberto.coluccio@eng.it>:

> Hello folks,
>
> I was wondering if it's possible to define custom constants in a Flume
> agent configuration file, so to avoid errors and replicated stuff when not
> needed. An example to make my point clear:
>
> assume I want to launch an agent of type *myAgent* configured with file
> channel and hdfs sink.
>
> I want the file channel to persist data and checkpoint into
> "~/flume/myAgent1/data" and "~/flume/myAgent1/checkpoint"
>
> Also, I want the hdfs sink to write files with prefix "myAgent1"
>
> In this scenario, I should harcorde the "myAgent1" string 3 times, one for
> each configuration parameter. This is error prone and kinda frustrating.
>
> Is there a way to define a constant like "myConstant" in the configuration
> file, and leverage it as parameter in the other configuration lines?
>
> e.g.
>
> myConstant = myAgent1
> myAgent.channels.myChannel.dataDirs = ~/flume/${myConstant}/data
> myAgent.sinks.mySink.hdfs.filePrefix = ${myConstant}
>
> It seems like the use of flume events header keys, but I don't want to add
> interceptors or other components just to have parameters in my config file.
>
> Thanks for your help.
>
> Best regards,
>
> Roberto
>
>
>

Mime
View raw message