flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tao Li <litao.bupt...@gmail.com>
Subject How about support custom counter for flume component
Date Thu, 30 Apr 2015 17:18:56 GMT
Hi all:

I use flume to transfer web server log to hdfs (a lzo file every 5min).
I have to two import monitoring point:
    1. if any message is lost
    2. the hdfs lzo complete delay
But now the flume default source or sink counter can't satisfy my

*I think that if flume can support to custom counter would be better. Now I
have to modify ScribeSource or HDFSEventSink source code, because the only
implementation is SourceCounter and SinkCounter.*

For example, I have implement a TimedSouceCounter and TimedSinkCounter, it
can aggregate the counters within five minutes and it record the final
update timestmap.
So it would be easy for me to calculate how many message is lost from
source to sink, and calculate final lzo file delay.

For flume config file. I add a "customSourceCounterType" field
[image: 内嵌图片 3]

Add a new metrics like "EventAcceptedCountInFiveMinJson"
[image: 内嵌图片 1]

View raw message