phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Soldatov <sergeysolda...@gmail.com>
Subject Re: High CPU usage on Hbase region Server with GlobalMemoryManager warnings
Date Thu, 01 Feb 2018 19:17:34 GMT
that kind of messages may happen when there were queries that utilize
memory manager (usually joins and group by) and they were timed out or
failed due to some reason. So the message itself is hardly related to CPU
usage or GC.
BUT. That may mean that your region servers are unable to handle properly
such kind of workload.
Since you say that this issue started after Yarn work I would suggest
checking swappiness and huge pages (there are quite a lot of resources over
the Internet how they affect HBase). It might be the case when you just run
out of HW resources.

Thanks,
Sergey

On Wed, Jan 31, 2018 at 6:40 PM, Jins George <jins.george@aeris.net> wrote:

> Hi,
>
> On analyzing a prod issue of High CPU usage on Hbase Region server, I came
> across warning messages from region server logs complaining about Orphaned
> chunk of memory.
>
> 2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: Orphaned
chunk of 104000 bytes found during finalize
> 2018-01-30 19:16:31,565 WARN org.apache.phoenix.memory.GlobalMemoryManager: Orphaned
chunk of 104000 bytes found during finalize
>
>
> The high CPU usage looks like due to garbage collection and it lasted for
> almost 6 hours.  And throughout 6 hours, region server logs had these
> warning messages logged.
>
> Cluster Details:
> 4 node( 1 master + 3 slaves)  cdh cluster
> Hbase version 1.2
> Phoenix version 4.7
> Region Server Heap : 4G
> Total Regions: ~135
> Total tables : ~35
>
> Out of 3 region servers, 2 of them had the warning logs and both suffered
> high CPU. Third region server nither had High CPU nor the warning logs. Any
> idea why these messages are logged and can that trigger continuous GC ?
>
> Before this issue started( or around the same time) huge application log
> files were copied to HDFS by Yarn.. But can't think of that causing issue
> on Hbase Region  server.
>
> Any help is appreciated.
>
> Thanks,
> Jins George
>

Mime
View raw message