flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: How can i get flume performance metrix? Jmx
Date Thu, 06 Aug 2015 00:22:23 GMT
If you are using Hortonworks distro and installed Flume via Ambari .. Then you should be  able
to monitor much of that from Ambari.
If you used command line installation, then you can do what Iain suggested below.

From: iain wright <iainwrig@gmail.com<mailto:iainwrig@gmail.com>>
Reply-To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Date: Wednesday, August 5, 2015 3:42 PM
To: "user@flume.apache.org<mailto:user@flume.apache.org>" <user@flume.apache.org<mailto:user@flume.apache.org>>
Subject: Re: How can i get flume performance metrix? Jmx

For CPU/mem/GC/etc. you should be able to add this to JAVA_OPTS in flume-env.sh (im not 100%
sure hortonworks has this, or that it hasn't been abstracted/pulled into an init script or

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321 -Dcom.sun.management.jmxremote.authenticate=false

JAVA_OPTS="-Xms1024m -Xmx2048m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=54321
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"

For tracking events through the pipeline, im not sure if those are in JMX, i don't believe
they are, we use the /metrics endpoint flume serves up:

[root@ip-172-16-1-119 ec2-user]# curl -s http://localhost:16742/metrics | python -mjson.tool
    "CHANNEL.channel1": {
        "ChannelCapacity": "100000000",
        "ChannelFillPercentage": "0.0",
        "ChannelSize": "0",
        "EventPutAttemptCount": "42355657",
        "EventPutSuccessCount": "42355657",
        "EventTakeAttemptCount": "42391346",
        "EventTakeSuccessCount": "42355657",
        "StartTime": "1438627774367",
        "StopTime": "0",
        "Type": "CHANNEL"
    "SINK.sink1": {
        "BatchCompleteCount": "423556",
        "BatchEmptyCount": "35688",
        "BatchUnderflowCount": "1",
        "ConnectionClosedCount": "1634",
        "ConnectionCreatedCount": "1639",
        "ConnectionFailedCount": "20",
        "EventDrainAttemptCount": "42355657",
        "EventDrainSuccessCount": "42355657",
        "StartTime": "1438627774370",
        "StopTime": "0",
        "Type": "SINK"
    "SOURCE.source1": {
        "AppendAcceptedCount": "0",
        "AppendBatchAcceptedCount": "423562",
        "AppendBatchReceivedCount": "423562",
        "AppendReceivedCount": "0",
        "EventAcceptedCount": "42355657",
        "EventReceivedCount": "42355657",
        "OpenConnectionCount": "0",
        "StartTime": "1438627774387",
        "StopTime": "0",
        "Type": "SOURCE"

You can poll this, awk/grep/parse/etc. & store values of channelsize or take deltas of
event drained success to extrapolate an average throughput

Note that flume would need to be started with these properties for /metrics to work:

-Dflume.monitoring.type=HTTP -Dflume.monitoring.port=16742

Hope that helps

Iain Wright

This email message is confidential, intended only for the recipient(s) named above and may
contain information that is privileged, exempt from disclosure under applicable law. If you
are not the intended recipient, do not disclose or disseminate the message to anyone except
the intended recipient. If you have received this message in error, or are not the named recipient(s),
please immediately notify the sender by return email, and delete all copies of this message.

On Wed, Aug 5, 2015 at 2:39 PM, Sutanu Das <sd2302@att.com<mailto:sd2302@att.com>>
Hi Team,

Can you please advise how can I see the Flume jmx performance metrix in our production?

Our management is interested to know Flume performance usage as far as cpu, mem, gc, events,

We use Hortonworks Flume

View raw message