phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Zookeeper connection remains alive after DefaultStatisticsCollector's collecting stats done
Date Thu, 29 Dec 2016 04:45:18 GMT
This definitely sounds like a serious problem. Agreed, it should be closed
immediately. Also, a minor compaction shouldn't trigger any stats
collection logic. Please file a JIRA and we'll get this fixed in the next
release. As always, patches are welcome.
Thanks,
James

On Wed, Dec 28, 2016 at 7:37 PM 김연섭 <nilkis0@gmail.com> wrote:

> In every minor compaction job of HBase,
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidePostDepth()
> is called,
>
> and SYSTEM.CATALOG table is open to get guidepost width via
>
> htable = env.getTable(
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>                     SchemaUtil.*getPhysicalTableName*
> (PhoenixDatabaseMetaData.*SYSTEM_CATALOG_NAME_BYTES*, env
> .getConfiguration()));
>
> This function call creates one zookeeper connection to get cluster id.
>
> DefaultStatisticsCollector doesn't close this zookeeper connection
> immediately after get guidepost width, and the zookeeper connection remains
> alive until HRegion is closed.
>
> This is not a problem with small number of Regions, but when number of
> Region is large and upsert operation is frequent, the number of zookeeper
> connection gradually increases  to hundreds, and the zookeeper server nodes
> experience  short of available TCP/IP ports.
>
> I think this zookeeper connection should be closed immediately after get
> guidepost width.
>
> My Phoenix environment is,
>
> - HBase : v1.1.8
>
> - Phoenix : v4.9.0-HBase-1.1
>
> - 80 nodes of hbase
>
> The following is the Dump stack of zookeeper connection creation
>
>
>        at java.lang.Thread.dumpStack(Thread.java:1329)
>
>         at
> org.apache.hadoop.hbase.client.ZooKeeperKeepAliveConnection.<init>(ZooKeeperKeepAliveConnection.java:48)
>
>         at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveZooKeeperWatcher(ConnectionManager.java:1663)
>
>         at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:104)
>
>         at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:885)
>
>         at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:638)
>
>         at
> org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:99)
>
>         at
> org.apache.hadoop.hbase.client.CoprocessorHConnection.<init>(CoprocessorHConnection.java:89)
>
>         at
> org.apache.hadoop.hbase.client.CoprocessorHConnection.getConnectionForEnvironment(CoprocessorHConnection.java:61)
>
>         at
> org.apache.hadoop.hbase.client.HTableWrapper.createWrapper(HTableWrapper.java:69)
>
>         at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:514)
>
>         at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.getTable(CoprocessorHost.java:503)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.initGuidepostDepth(DefaultStatisticsCollector.java:123)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.init(DefaultStatisticsCollector.java:299)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.getInternalScanner(DefaultStatisticsCollector.java:293)
>
>         at
> org.apache.phoenix.schema.stats.DefaultStatisticsCollector.createCompactionScanner(DefaultStatisticsCollector.java:285)
>
>         at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:692)
>
>         at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$2.run(UngroupedAggregateRegionObserver.java:681)
>
>         at java.security.AccessController.doPrivileged(Native Method)
>
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>
>         at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:448)
>
>         at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUser(SecurityUtil.java:429)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>         at java.lang.reflect.Method.invoke(Method.java:498)
>
>         at org.apache.hadoop.hbase.util.Methods.call(Methods.java:39)
>
>         at
> org.apache.hadoop.hbase.security.User.runAsLoginUser(User.java:205)
>
>         at
> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.preCompact(UngroupedAggregateRegionObserver.java:681)
>
>         at
> org.apache.hadoop.hbase.coprocessor.BaseRegionObserver.preCompact(BaseRegionObserver.java:195)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$9.call(RegionCoprocessorHost.java:612)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1708)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1785)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperationWithResult(RegionCoprocessorHost.java:1757)
>
>         at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preCompact(RegionCoprocessorHost.java:607)
>
>         at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.postCreateCoprocScanner(Compactor.java:247)
>
>         at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:91)
>
>         at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:119)
>
>         at
> org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1221)
>
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1845)
>
>         at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:521)
>
>         at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:557)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
>
>
>

Mime
View raw message