hi, ALL:

   we are useing phoenix-4.7 with hbase-1.1.3,  running a 17 nodes cluster with 200+ tables and 3500 regions.

Recently there are weird phenomenon happened and I’m certain that have not done anything to the cluster:


    the RS that holding system.catalog region comes up lots of zk connection create and close logs, alost 1 million times a day. eg:

     2017-10-09 15:26:12,656 INFO  [B.defaultRpcServer.handler=129,queue=17,port=16020-SendThread(zk1:2181)] zookeeper.ClientCnxn: Session establishment complete on server zk1/zk1:2181, sessionid = 0x258206f4e13c7ea, negotiated timeout = 60000

2017-10-09 15:26:13,131 INFO  [B.defaultRpcServer.handler=129,queue=17,port=16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x258206f4e13c7ea

2017-10-09 15:26:13,136 INFO  [B.defaultRpcServer.handler=129,queue=17,port=16020] zookeeper.ZooKeeper: Session: 0x258206f4e13c7ea closed


    system.stats table reads is extremely high,  about 60K qps 

Have anyone met this before?   Or how can I fix this problem?