flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Li <litao.bupt...@gmail.com>
Subject Re: How about support custom counter for flume component
Date Thu, 30 Apr 2015 17:25:36 GMT
Now my implementation is kind of ugly. I have to modify ScribeSource source
code to support my new TimedSourceCounter. If flume can better support
custom counter, for example I can add my custom counter to ScribeSource
through config file, it will be nice to me.
[image: 内嵌图片 1]
[image: 内嵌图片 2]

2015-05-01 1:18 GMT+08:00 Tao Li <litao.buptsse@gmail.com>:

> Hi all:
> I use flume to transfer web server log to hdfs (a lzo file every 5min).
> I have to two import monitoring point:
>     1. if any message is lost
>     2. the hdfs lzo complete delay
> But now the flume default source or sink counter can't satisfy my
> requirements.
> *I think that if flume can support to custom counter would be better. Now
> I have to modify ScribeSource or HDFSEventSink source code, because the
> only implementation is SourceCounter and SinkCounter.*
> For example, I have implement a TimedSouceCounter and TimedSinkCounter, it
> can aggregate the counters within five minutes and it record the final
> update timestmap.
> So it would be easy for me to calculate how many message is lost from
> source to sink, and calculate final lzo file delay.
> For flume config file. I add a "customSourceCounterType" field
> [image: 内嵌图片 3]
> Add a new metrics like "EventAcceptedCountInFiveMinJson"
> [image: 内嵌图片 1]

View raw message