phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jins George <jins.geo...@aeris.net>
Subject High CPU usage on Hbase region Server with GlobalMemoryManager warnings
Date Thu, 01 Feb 2018 02:40:42 GMT
Hi,

On analyzing a prod issue of High CPU usage on Hbase Region server, I 
came across warning messages from region server logs complaining about 
Orphaned chunk of memory.

2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk
of 104000 bytes found during finalize
2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: Orphaned chunk
of 104000 bytes found during finalize


The high CPU usage looks like due to garbage collection and it lasted 
for almost 6 hours.  And throughout 6 hours, region server logs had 
these warning messages logged.

Cluster Details:
4 node( 1 master + 3 slaves)  cdh cluster
Hbase version 1.2
Phoenix version 4.7
Region Server Heap : 4G
Total Regions: ~135
Total tables : ~35

Out of 3 region servers, 2 of them had the warning logs and both 
suffered high CPU. Third region server nither had High CPU nor the 
warning logs. Any idea why these messages are logged and can that 
trigger continuous GC ?

Before this issue started( or around the same time) huge application log 
files were copied to HDFS by Yarn.. But can't think of that causing 
issue on Hbase Region  server.

Any help is appreciated.

Thanks,
Jins George

Mime
View raw message