flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuang <shu...@open42.com>
Subject Re: Output bucketing based on custom data / logic
Date Tue, 29 Nov 2011 18:45:57 GMT
You can use the regex decorator to parse out "ABC" and put it in a meta
data field for example named "customer", then in your collectorSink,  use
%{customer} in the destination path.


On Tue, Nov 29, 2011 at 6:10 AM, Srinivasan Subramanian <
ssrini_vasan@hotmail.com> wrote:

>  Just got flume installed on Centos and the basic setup is working great.
>  I have got a Master, collector and a few agents.
> One additional requirement i have is to do with Output Bucketing.  Based
> on data that is present in the log message that is being sent to flume, the
> data has to be slotted into different buckets.  For eg assume that the log
> files are to be separated into different folders based on a customer name
> that is present in the log message, how would one go about doing this?
> Lets say the log message sent to Flume is "ABC : Some message".  The
> collector needs to put this into a folder like /var/log/ABC.  How can this
> be achieved?
> Alternatively,  Can I use metadata?  As far as i could make out meta data
> is static and not dynamic as i would need it to be.
> Thanks for all the help.
> Regards
> Srini

View raw message