flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Wise <m...@nextdoor.com>
Subject Dynamic Key=Value Parsing with an Interceptor?
Date Sun, 10 Nov 2013 01:28:09 GMT
Hey we'd like to set up a default format for all of our logging systems...
perhaps looking like this:


With this pattern, we'd allow developers to define any key/value pairs they
want to log, and separate them with a common separator.

If we did this, what do we need to do in Flume to get Flume to parse out
the key=value pairs into dynamic headers? We pass our data from Flume into
both HDFS and ElasticSearch sinks. We would really like to have these
fields dynamically sent to the sinks for much easier parsing and analysis

Any thoughts on this? I know that we can define a unique interceptor for
each service that looks for explicit field names ... but thats a nightmare
to manage. I really want something truly dynamic.

Matt Wise
Sr. Systems Architect

View raw message