flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From iain wright <iainw...@gmail.com>
Subject Re: How can i get flume performance metrix? Jmx
Date Wed, 05 Aug 2015 22:42:06 GMT
For CPU/mem/GC/etc. you should be able to add this to JAVA_OPTS in
flume-env.sh (im not 100% sure hortonworks has this, or that it hasn't been
abstracted/pulled into an init script or something)

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321

JAVA_OPTS="-Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote

For tracking events through the pipeline, im not sure if those are in JMX,
i don't believe they are, we use the /metrics endpoint flume serves up:

[root@ip-172-16-1-119 ec2-user]# curl -s http://localhost:16742/metrics |
python -mjson.tool
    "CHANNEL.channel1": {
        "ChannelCapacity": "100000000",
        "ChannelFillPercentage": "0.0",
        "ChannelSize": "0",
        "EventPutAttemptCount": "42355657",
        "EventPutSuccessCount": "42355657",
        "EventTakeAttemptCount": "42391346",
        "EventTakeSuccessCount": "42355657",
        "StartTime": "1438627774367",
        "StopTime": "0",
        "Type": "CHANNEL"
    "SINK.sink1": {
        "BatchCompleteCount": "423556",
        "BatchEmptyCount": "35688",
        "BatchUnderflowCount": "1",
        "ConnectionClosedCount": "1634",
        "ConnectionCreatedCount": "1639",
        "ConnectionFailedCount": "20",
        "EventDrainAttemptCount": "42355657",
        "EventDrainSuccessCount": "42355657",
        "StartTime": "1438627774370",
        "StopTime": "0",
        "Type": "SINK"
    "SOURCE.source1": {
        "AppendAcceptedCount": "0",
        "AppendBatchAcceptedCount": "423562",
        "AppendBatchReceivedCount": "423562",
        "AppendReceivedCount": "0",
        "EventAcceptedCount": "42355657",
        "EventReceivedCount": "42355657",
        "OpenConnectionCount": "0",
        "StartTime": "1438627774387",
        "StopTime": "0",
        "Type": "SOURCE"

You can poll this, awk/grep/parse/etc. & store values of channelsize or
take deltas of event drained success to extrapolate an average throughput

Note that flume would need to be started with these properties for /metrics
to work:

-Dflume.monitoring.type=HTTP -Dflume.monitoring.port=16742

Hope that helps

Iain Wright

This email message is confidential, intended only for the recipient(s)
named above and may contain information that is privileged, exempt from
disclosure under applicable law. If you are not the intended recipient, do
not disclose or disseminate the message to anyone except the intended
recipient. If you have received this message in error, or are not the named
recipient(s), please immediately notify the sender by return email, and
delete all copies of this message.

On Wed, Aug 5, 2015 at 2:39 PM, Sutanu Das <sd2302@att.com> wrote:

> Hi Team,
> Can you please advise how can I see the Flume jmx performance metrix in
> our production?
> Our management is interested to know Flume performance usage as far as
> cpu, mem, gc, events, etc…..
> We use Hortonworks Flume

View raw message