phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pedro Boado <pedro.bo...@gmail.com>
Subject Re: High CPU usage on Hbase region Server with GlobalMemoryManager warnings
Date Thu, 01 Feb 2018 07:48:58 GMT
Maybe the warnings are not the cause but a consequence (gc calls finalize()
and not the other way around)

Any details on memory usage? G1? Non full Vs full gc ratio, average freed
memory... Did any gc run in the 3rd RS?

Memory percentage assigned to memstores? You have an average memory
assigned of 120MB per region assuming 1 column family per table, and HBase
default memstore size  is already 256mb. You might just being running oom.

Read/write pattern? Is there an intensive write?

In my experience these kind of issues are caused by a bad HBase memory
tuning.






On 1 Feb 2018 02:41, "Jins George" <jins.george@aeris.net> wrote:

Hi,

On analyzing a prod issue of High CPU usage on Hbase Region server, I came
across warning messages from region server logs complaining about Orphaned
chunk of memory.

2018-01-30 19:16:31,565 WARN
org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk of
104000 bytes found during finalize
2018-01-30 19:16:31,565 WARN
org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk of
104000 bytes found during finalize


The high CPU usage looks like due to garbage collection and it lasted for
almost 6 hours.  And throughout 6 hours, region server logs had these
warning messages logged.

Cluster Details:
4 node( 1 master + 3 slaves)  cdh cluster
Hbase version 1.2
Phoenix version 4.7
Region Server Heap : 4G
Total Regions: ~135
Total tables : ~35

Out of 3 region servers, 2 of them had the warning logs and both suffered
high CPU. Third region server nither had High CPU nor the warning logs. Any
idea why these messages are logged and can that trigger continuous GC ?

Before this issue started( or around the same time) huge application log
files were copied to HDFS by Yarn.. But can't think of that causing issue
on Hbase Region  server.

Any help is appreciated.

Thanks,
Jins George

Mime
View raw message