flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keane, Mike" <mke...@conversantmedia.com>
Subject RE: Determine the backlog in File channel
Date Tue, 03 Nov 2015 13:53:32 GMT
We ping our flume metrics every 5 minutes and push the results to a TSDB (http://opentsdb.net/)
database.  TSDB is great for visualizing your data rates.  Depending on your flume configuration,
you will get greatly varying rates.  If you are using spinning disks with a file channel you'll
want to make sure your checkpoint directory and  and data directory are on physically different
disks.


________________________________
From: Buntu Dev [buntudev@gmail.com]
Sent: Monday, November 02, 2015 7:09 PM
To: user@flume.apache.org
Subject: Re: Determine the backlog in File channel

Awesome thanks!

Whats the right way to calculate the throughput of the channel?

On Mon, Nov 2, 2015 at 4:10 PM, Ahmed Vila <avila@devlogic.eu<mailto:avila@devlogic.eu>>
wrote:
To add to it, here is the user manual reference:
https://flume.apache.org/FlumeUserGuide.html#json-reporting

On Tue, Nov 3, 2015 at 1:06 AM, Keane, Mike <mkeane@conversantmedia.com<mailto:mkeane@conversantmedia.com>>
wrote:
Enable monitoring and you can get the channelFillPercentage and channelSize from the metrics:

VM arguments for http monitoring:
-Dflume.monitoring.type=http -Dflume.monitoring.port=12345

For JMX
-Dcom.sun.management.jmxremote.port=12346

or whatever port you choose.


________________________________
From: Buntu Dev [buntudev@gmail.com<mailto:buntudev@gmail.com>]
Sent: Monday, November 02, 2015 4:33 PM
To: user@flume.apache.org<mailto:user@flume.apache.org>
Subject: Determine the backlog in File channel

I got a File channel with HDFS sink. In the case when the sink slows down and event taken
from the channel falls behind while the event puts continue at the same pace, how would one
go about finding the amount of backlog or time it takes to clear the backlog?

Thanks!

This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.



--

Best regards,

Ahmed Vila | Senior software developer
DevLogic | Sarajevo | Bosnia and Herzegovina

Office : +387 33 942 123<tel:%2B387%2033%20942%20123>
Mobile: +387 62 139 348<tel:%2B387%2062%20139%20348>

Website: www.devlogic.eu<http://www.devlogic.eu>
E-mail   : avila@devlogic.eu<mailto:avila@devlogic.eu>
---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipient(s) only. This
email contains confidential information. It should not be copied, disclosed to, retained or
used by, any party other than the intended recipient. Any unauthorised distribution, dissemination
or copying of this E-mail or its attachments, and/or any use of any information contained
in them, is strictly prohibited and may be illegal. If you are not an intended recipient then
please promptly delete this e-mail and any attachment and all copies and inform the sender
directly via email. Any emails that you send to us may be monitored by systems or persons
other than the named communicant for the purposes of ascertaining whether the communication
complies with the law and company policies.

---------------------------------------------------------------------
This e-mail and any attachment is for authorised use by the intended recipient(s) only. This
email contains confidential information. It should not be copied, disclosed to, retained or
used by, any party other than the intended recipient. Any unauthorised distribution, dissemination
or copying of this E-mail or its attachments, and/or any use of any information contained
in them, is strictly prohibited and may be illegal. If you are not an intended recipient then
please promptly delete this e-mail and any attachment and all copies and inform the sender
directly via email. Any emails that you send to us may be monitored by systems or persons
other than the named communicant for the purposes of ascertaining whether the communication
complies with the law and company policies.





This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.

Mime
View raw message