phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "vergil" <vergilc...@foxmail.com>
Subject Forward: Re: Forward: Re: local index turn disable when region split
Date Tue, 07 Nov 2017 03:26:39 GMT
Thanks for your help.


Here is the detail log:
The region repeat the below head log" wait for xx actions" for a long time.
Then show the rest log when the region turn normal and the local index turn disable.


2017-11-06 17:59:56,517 WARN  [B.defaultRpcServer.handler=16,queue=1,port=16020] ipc.RpcServer:
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1509962336516,"responsesize":56543,"method":"Multi","param":"region=
TEST_TABLE_LOCAL,\\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee., for 1 actions and
1st row key=\\x01a2abc443ecb94a68ac9c96de6e8b83a9-0","processingtimems":60001,"client":"192.168.2.11:33120","queuetimems":0,"class":"HRegionServer"}
2017-11-06 17:59:56,639 WARN  [B.defaultRpcServer.handler=22,queue=1,port=16020] ipc.RpcServer:
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1509962336638,"responsesize":48269,"method":"Multi","param":"region=
TEST_TABLE_LOCAL,\\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee., for 1 actions and
1st row key=\\x0161eedff38f5b4440b3d2b1eacf62b1a3-0","processingtimems":60001,"client":"192.168.2.11:33120","queuetimems":0,"class":"HRegionServer"}
2017-11-06 17:59:58,390 INFO  [ht06,16020,1509939923675-index-writer--pool391-t7] client.AsyncProcess:
#394, waiting for 35  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 17:59:58,396 INFO  [ht06,16020,1509939923675-index-writer--pool391-t9] client.AsyncProcess:
#394, waiting for 34  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 17:59:58,403 INFO  [ht06,16020,1509939923675-index-writer--pool391-t8] client.AsyncProcess:
#394, waiting for 40  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 17:59:58,407 INFO  [ht06,16020,1509939923675-index-writer--pool391-t1] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 17:59:58,414 INFO  [ht06,16020,1509939923675-index-writer--pool391-t4] client.AsyncProcess:
#394, waiting for 28  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 17:59:58,435 INFO  [ht06,16020,1509939923675-index-writer--pool391-t2] client.AsyncProcess:
#394, waiting for 33  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 17:59:58,470 INFO  [ht06,16020,1509939923675-index-writer--pool391-t10] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:08,400 INFO  [ht06,16020,1509939923675-index-writer--pool391-t7] client.AsyncProcess:
#394, waiting for 35  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:08,400 INFO  [ht06,16020,1509939923675-index-writer--pool391-t9] client.AsyncProcess:
#394, waiting for 34  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:08,410 INFO  [ht06,16020,1509939923675-index-writer--pool391-t1] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:08,413 INFO  [ht06,16020,1509939923675-index-writer--pool391-t8] client.AsyncProcess:
#394, waiting for 40  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:08,418 INFO  [ht06,16020,1509939923675-index-writer--pool391-t4] client.AsyncProcess:
#394, waiting for 28  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:08,445 INFO  [ht06,16020,1509939923675-index-writer--pool391-t2] client.AsyncProcess:
#394, waiting for 33  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:08,478 INFO  [ht06,16020,1509939923675-index-writer--pool391-t10] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:18,404 INFO  [ht06,16020,1509939923675-index-writer--pool391-t9] client.AsyncProcess:
#394, waiting for 34  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:18,407 INFO  [ht06,16020,1509939923675-index-writer--pool391-t7] client.AsyncProcess:
#394, waiting for 35  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:18,414 INFO  [ht06,16020,1509939923675-index-writer--pool391-t1] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:18,415 INFO  [ht06,16020,1509939923675-index-writer--pool391-t8] client.AsyncProcess:
#394, waiting for 40  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:18,421 INFO  [ht06,16020,1509939923675-index-writer--pool391-t4] client.AsyncProcess:
#394, waiting for 28  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:18,456 INFO  [ht06,16020,1509939923675-index-writer--pool391-t2] client.AsyncProcess:
#394, waiting for 33  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:18,479 INFO  [ht06,16020,1509939923675-index-writer--pool391-t10] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:25,129 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=788.37
MB, freeSize=5.46 GB, max=6.23 GB, blockCount=26882, accesses=18497669, hits=4416532, hitRatio=23.88%,
, cachingAccesses=4453026, cachingHits=4382679, cachingHitsRatio=98.42%, evictions=2249, evicted=43278,
evictedPerRun=19.24321937561035
2017-11-06 18:00:28,405 INFO  [ht06,16020,1509939923675-index-writer--pool391-t9] client.AsyncProcess:
#394, waiting for 34  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:28,410 INFO  [ht06,16020,1509939923675-index-writer--pool391-t7] client.AsyncProcess:
#394, waiting for 35  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:28,416 INFO  [ht06,16020,1509939923675-index-writer--pool391-t1] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:28,425 INFO  [ht06,16020,1509939923675-index-writer--pool391-t4] client.AsyncProcess:
#394, waiting for 28  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:28,425 INFO  [ht06,16020,1509939923675-index-writer--pool391-t8] client.AsyncProcess:
#394, waiting for 40  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:28,465 INFO  [ht06,16020,1509939923675-index-writer--pool391-t2] client.AsyncProcess:
#394, waiting for 33  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:28,482 INFO  [ht06,16020,1509939923675-index-writer--pool391-t10] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:36,397 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: Stopping HBase metrics
system...
2017-11-06 18:00:36,398 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system
stopped.
2017-11-06 18:00:36,899 INFO  [HBase-Metrics2-1] impl.MetricsConfig: loaded properties from
hadoop-metrics2-hbase.properties
2017-11-06 18:00:36,900 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2017-11-06 18:00:36,900 INFO  [HBase-Metrics2-1] impl.MetricsSystemImpl: HBase metrics system
started
2017-11-06 18:00:38,413 INFO  [ht06,16020,1509939923675-index-writer--pool391-t7] client.AsyncProcess:
#394, waiting for 35  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:38,415 INFO  [ht06,16020,1509939923675-index-writer--pool391-t9] client.AsyncProcess:
#394, waiting for 34  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:38,418 INFO  [ht06,16020,1509939923675-index-writer--pool391-t1] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:38,428 INFO  [ht06,16020,1509939923675-index-writer--pool391-t4] client.AsyncProcess:
#394, waiting for 28  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:38,428 INFO  [ht06,16020,1509939923675-index-writer--pool391-t8] client.AsyncProcess:
#394, waiting for 40  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:38,469 INFO  [ht06,16020,1509939923675-index-writer--pool391-t2] client.AsyncProcess:
#394, waiting for 33  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:38,486 INFO  [ht06,16020,1509939923675-index-writer--pool391-t10] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:48,417 INFO  [ht06,16020,1509939923675-index-writer--pool391-t7] client.AsyncProcess:
#394, waiting for 35  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:48,421 INFO  [ht06,16020,1509939923675-index-writer--pool391-t1] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:48,425 INFO  [ht06,16020,1509939923675-index-writer--pool391-t9] client.AsyncProcess:
#394, waiting for 34  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:48,431 INFO  [ht06,16020,1509939923675-index-writer--pool391-t4] client.AsyncProcess:
#394, waiting for 28  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:48,432 INFO  [ht06,16020,1509939923675-index-writer--pool391-t8] client.AsyncProcess:
#394, waiting for 40  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:48,472 INFO  [ht06,16020,1509939923675-index-writer--pool391-t2] client.AsyncProcess:
#394, waiting for 33  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:48,488 INFO  [ht06,16020,1509939923675-index-writer--pool391-t10] client.AsyncProcess:
#394, waiting for 31  actions to finish on table: TEST_TABLE_LOCAL
2017-11-06 18:00:56,393 INFO  [hconnection-0x52dee0d4-shared--pool423-t6] client.AsyncProcess:
#394, table=TEST_TABLE_LOCAL, attempt=10/11 failed=31ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in 60000 ms. regionName=TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.,
server=ht06,16020,1509939923675
	at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8176)
	at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8162)
	at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8071)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2866)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
	at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:128)
	at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:748)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
 on ht06,16020,1509939923675, tracking started null, retrying after=10099ms, replay=31ops
2017-11-06 18:02:06,572 WARN  [hconnection-0x52dee0d4-shared--pool423-t4] client.AsyncProcess:
#394, table=TEST_TABLE_LOCAL, attempt=11/11 failed=33ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in 60000 ms. regionName=TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.,
server=ht06,16020,1509939923675
	at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8176)
	at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8162)
	at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8071)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2866)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
	at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:128)
	at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:748)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.sendMultiAction(AsyncProcess.java:1025)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.groupAndSendMultiAction(AsyncProcess.java:934)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.resubmit(AsyncProcess.java:1252)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.receiveMultiAction(AsyncProcess.java:1406)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.access$1300(AsyncProcess.java:615)
	at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:769)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
 on ht06,16020,1509939923675, tracking started Mon Nov 06 17:50:28 CST 2017; not retrying
33 - final failure
2017-11-06 18:02:06,573 INFO  [B.defaultRpcServer.handler=17,queue=2,port=16020] client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x15efee0c16a0c4e
2017-11-06 18:02:06,573 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020] util.QueryUtil:
Creating connection with the jdbc url: jdbc:phoenix:ht05,ht06,ht07:2181:/hbase;
2017-11-06 18:02:06,574 INFO  [B.defaultRpcServer.handler=17,queue=2,port=16020] zookeeper.ZooKeeper:
Session: 0x15efee0c16a0c4e closed
2017-11-06 18:02:06,575 INFO  [B.defaultRpcServer.handler=17,queue=2,port=16020-EventThread]
zookeeper.ClientCnxn: EventThread shut down
2017-11-06 18:02:06,577 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020] zookeeper.RecoverableZooKeeper:
Process identifier=hconnection-0x16e8b953 connecting to ZooKeeper ensemble=ht05:2181,ht06:2181,ht07:2181
2017-11-06 18:02:06,578 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020] zookeeper.ZooKeeper:
Initiating client connection, connectString=ht05:2181,ht06:2181,ht07:2181 sessionTimeout=90000
watcher=hconnection-0x16e8b9530x0, quorum=ht05:2181,ht06:2181,ht07:2181, baseZNode=/hbase
2017-11-06 18:02:06,579 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020-SendThread(ht06:2181)]
zookeeper.ClientCnxn: Opening socket connection to server ht06/192.168.2.6:2181. Will not
attempt to authenticate using SASL (unknown error)
2017-11-06 18:02:06,580 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020-SendThread(ht06:2181)]
zookeeper.ClientCnxn: Socket connection established to ht06/192.168.2.6:2181, initiating session
2017-11-06 18:02:06,581 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020-SendThread(ht06:2181)]
zookeeper.ClientCnxn: Session establishment complete on server ht06/192.168.2.6:2181, sessionid
= 0x25efee0c16d0bac, negotiated timeout = 40000
2017-11-06 18:02:06,582 INFO  [B.defaultRpcServer.handler=21,queue=0,port=16020] index.PhoenixIndexFailurePolicy:
Successfully disabled index TEST_INDEX_LOCAL due to an exception while writing updates.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: Failed to write
to multiple index tables
	at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:229)
	at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:193)
	at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:154)
	at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:143)
	at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:653)
	at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:610)
	at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:593)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
	at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3322)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2881)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:748)
2017-11-06 18:02:06,583 INFO  [B.defaultRpcServer.handler=21,queue=0,port=16020] client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x25efee0c16d0bab
2017-11-06 18:02:06,584 INFO  [B.defaultRpcServer.handler=21,queue=0,port=16020] zookeeper.ZooKeeper:
Session: 0x25efee0c16d0bab closed
2017-11-06 18:02:06,584 INFO  [B.defaultRpcServer.handler=21,queue=0,port=16020-EventThread]
zookeeper.ClientCnxn: EventThread shut down
2017-11-06 18:02:06,584 INFO  [B.defaultRpcServer.handler=17,queue=2,port=16020] util.IndexManagementUtil:
Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: Unable to update the following indexes:
[TEST_TABLE_LOCAL],serverTimestamp=1509961768138,
2017-11-06 18:02:06,587 WARN  [B.defaultRpcServer.handler=17,queue=2,port=16020] ipc.RpcServer:
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1509961768138,"responsesize":99299,"method":"Multi","param":"region=
TEST_TABLE_LOCAL,\\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee., for 1 actions and
1st row key=\\x012ce9caa0990347988c2723ea0eebe324-0","processingtimems":758448,"client":"192.168.2.11:33120","queuetimems":0,"class":"HRegionServer"}
2017-11-06 18:02:06,588 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020] index.PhoenixIndexFailurePolicy:
Successfully disabled index TEST_INDEX_LOCAL due to an exception while writing updates.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: Failed to write
to multiple index tables
	at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:229)
	at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:193)
	at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:154)
	at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:143)
	at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:653)
	at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:610)
	at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:593)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
	at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3322)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2881)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:748)
2017-11-06 18:02:06,588 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020] client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x25efee0c16d0bac
2017-11-06 18:02:06,589 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020] zookeeper.ZooKeeper:
Session: 0x25efee0c16d0bac closed
2017-11-06 18:02:06,589 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020-EventThread]
zookeeper.ClientCnxn: EventThread shut down
2017-11-06 18:02:06,594 INFO  [B.defaultRpcServer.handler=21,queue=0,port=16020] util.IndexManagementUtil:
Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: Unable to update the following indexes:
[TEST_TABLE_LOCAL],serverTimestamp=1509961768129,
2017-11-06 18:02:06,595 WARN  [B.defaultRpcServer.handler=21,queue=0,port=16020] ipc.RpcServer:
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1509961768128,"responsesize":79440,"method":"Multi","param":"region=
TEST_TABLE_LOCAL,\\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee., for 1 actions and
1st row key=\\x0133a193e472e24f49a1eaa53900b0ad9a-0","processingtimems":758467,"client":"192.168.2.11:33120","queuetimems":0,"class":"HRegionServer"}
2017-11-06 18:02:06,599 INFO  [B.defaultRpcServer.handler=20,queue=2,port=16020] util.IndexManagementUtil:
Rethrowing org.apache.hadoop.hbase.DoNotRetryIOException: Unable to update the following indexes:
[TEST_TABLE_LOCAL],serverTimestamp=1509961768131,
2017-11-06 18:02:06,599 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.HRegion:
Flushing 2/2 column families, memstore=14.80 MB
2017-11-06 18:02:06,600 WARN  [B.defaultRpcServer.handler=20,queue=2,port=16020] ipc.RpcServer:
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1509961768131,"responsesize":93625,"method":"Multi","param":"region=
TEST_TABLE_LOCAL,\\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee., for 1 actions and
1st row key=\\x0113f967dc89c84e54a67725cc18e9ebd0-0","processingtimems":758469,"client":"192.168.2.11:33120","queuetimems":0,"class":"HRegionServer"}
2017-11-06 18:02:06,717 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.DefaultStoreFlusher:
Flushed, sequenceid=42247, memsize=13.4 M, hasBloomFilter=false, into tmp file hdfs://192.168.2.5:9000/hbase/data/default/TEST_TABLE_LOCAL/9070a4f2ace4516f985c6c9e31427eee/.tmp/9f5734d07ab54572aa98df8bd94a4ab8
2017-11-06 18:02:06,750 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.DefaultStoreFlusher:
Flushed, sequenceid=42247, memsize=1.4 M, hasBloomFilter=false, into tmp file hdfs://192.168.2.5:9000/hbase/data/default/TEST_TABLE_LOCAL/9070a4f2ace4516f985c6c9e31427eee/.tmp/a14bd923bb624ba3b1f5afbf065e1f82
2017-11-06 18:02:06,761 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.HStore:
Added hdfs://192.168.2.5:9000/hbase/data/default/TEST_TABLE_LOCAL/9070a4f2ace4516f985c6c9e31427eee/0/9f5734d07ab54572aa98df8bd94a4ab8,
entries=63640, sequenceid=42247, filesize=2.5 M
2017-11-06 18:02:06,770 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.HStore:
Added hdfs://192.168.2.5:9000/hbase/data/default/TEST_TABLE_LOCAL/9070a4f2ace4516f985c6c9e31427eee/L#0/a14bd923bb624ba3b1f5afbf065e1f82,
entries=6308, sequenceid=42247, filesize=481.5 K
2017-11-06 18:02:06,771 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.HRegion:
Finished memstore flush of ~14.80 MB/15515168, currentsize=0 B/0 for region TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.
in 172ms, sequenceid=42247, compaction requested=true
2017-11-06 18:02:06,773 INFO  [StoreCloserThread-TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.-1]
regionserver.HStore: Closed 0
2017-11-06 18:02:06,775 INFO  [StoreCloserThread-TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.-1]
regionserver.HStore: Closed L#0
2017-11-06 18:02:06,800 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] write.TrackingParallelWriterIndexCommitter:
Shutting down TrackingParallelWriterIndexCommitter
2017-11-06 18:02:06,800 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] parallel.BaseTaskRunner:
Shutting down task runner because Indexer is being stopped
2017-11-06 18:02:06,801 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x35efee0c9eb0bd0
2017-11-06 18:02:06,802 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] zookeeper.ZooKeeper:
Session: 0x35efee0c9eb0bd0 closed
2017-11-06 18:02:06,802 INFO  [ht06,16020,1509939923675-index-writer--pool391-t4-EventThread]
zookeeper.ClientCnxn: EventThread shut down
2017-11-06 18:02:06,802 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] write.TrackingParallelWriterIndexCommitter:
Shutting down TrackingParallelWriterIndexCommitter
2017-11-06 18:02:06,802 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] parallel.BaseTaskRunner:
Shutting down task runner because Indexer is being stopped
2017-11-06 18:02:06,802 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] client.ConnectionManager$HConnectionImplementation:
Closing zookeeper sessionid=0x15efee0c16a0c2e
2017-11-06 18:02:06,803 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] zookeeper.ZooKeeper:
Session: 0x15efee0c16a0c2e closed
2017-11-06 18:02:06,803 INFO  [RS_OPEN_REGION-ht06:16020-0-EventThread] zookeeper.ClientCnxn:
EventThread shut down
2017-11-06 18:02:06,803 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.HRegion:
Closed TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.
2017-11-06 18:02:06,803 INFO  [regionserver/ht06/192.168.2.6:16020-splits-1509945613355] regionserver.SplitTransaction:
Preparing to split 10 storefiles for region TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.
using 10 threads
2017-11-06 18:02:06,807 WARN  [B.defaultRpcServer.handler=14,queue=2,port=16020] ipc.RpcServer:
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$MultiRequest)","starttimems":1509962476555,"responsesize":38647,"method":"Multi","param":"region=
TEST_TABLE_LOCAL,\\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee., for 1 actions and
1st row key=\\x0113f967dc89c84e54a67725cc18e9ebd0-0","processingtimems":50251,"client":"192.168.2.11:33120","queuetimems":0,"class":"HRegionServer"}

 




------------------ Original ------------------
From:  "rajeshbabu@apache.org";<chrajeshbabu32@gmail.com>;
Send time: Monday, Nov 6, 2017 7:11 PM
To: "user@phoenix.apache.org"<user@phoenix.apache.org>; 

Subject:  Re: Forward: Re: local index turn disable when region split



Hi Vergil,

Thanks for sharing the information. Can you please share the complete logs from where you
have collected the info?
Thanks,
Rajeshbabu.


On Mon, Nov 6, 2017 at 10:08 AM, vergil <vergilchiu@foxmail.com> wrote:
Thank for your reply.


Here is My test environment:
hbase:1.2.5
phoenix:4.12


Phoenix table with local index。


The region will wait for a long time when the region split。
The region server webapp page show below:
Start TimeDescriptionStateStatus
Mon Nov 06 17:49:27 CST 2017Closing region TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.RUNNING
(since 1mins, 46sec ago)Finished memstore flush of ~39.94 MB/41879888, currentsize=14.31 MB/15002912
for region TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee. in 856ms,
sequenceid=41861, compaction requested=true (since 1mins, 45sec ago)

The region server wait for a long time and log show below :
2017-11-06 18:02:06,450 INFO  [B.defaultRpcServer.handler=27,queue=0,port=16020] index.PhoenixIndexFailurePolicy:
Successfully disabled index TEST_INDEX_LOCAL due to an exception while writing updates.
org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: Failed to write
to multiple index tables
	at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:229)
	at org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:193)
	at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:154)
	at org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:143)
	at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:653)
	at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:610)
	at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:593)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1034)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
	at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3322)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2881)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
	at java.lang.Thread.run(Thread.java:748)



2017-11-06 18:02:06,493 WARN  [hconnection-0x52dee0d4-shared--pool423-t2] client.AsyncProcess:
#394, table=TEST_TABLE_LOCAL, attempt=11/11 failed=40ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException:
org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in 60000 ms. regionName=TEST_TABLE_LOCAL,\x01,1509961563238.9070a4f2ace4516f985c6c9e31427eee.,
server=ht06,16020,1509939923675
	at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8176)
	at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8162)
	at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8071)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2866)
	at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2823)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:758)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:720)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2168)
	at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:128)





The region will go back to normal after above exception,then table's local index turn disable.


------------------ Original ------------------
From:  "Ted Yu";<yuzhihong@gmail.com>;
Send time: Saturday, Nov 4, 2017 0:09 AM
To: "user@phoenix.apache.org"<user@phoenix.apache.org>; 

Subject:  Re: local index turn disable when region split



Can you give us more information ?


Release of hbase and Phoenix you use


bq. The local index turn disable 


Can you pastebin the related exception(s) ?


Snippet from region server log would also help.


On Thu, Nov 2, 2017 at 11:31 PM, vergil <vergilchiu@foxmail.com> wrote:
Hi,all:




Here is my test table.
create table test_table_local(id varchar primary key,f1 varchar,f2 varchar) salt_buckets=3;
Add local index on it at the first time.
create local index test_index_local on test_table_local(f1);


Then upsert data into it.
As the data increase,the region will split.
The local index turn disable when the region splits.
Local index data do not increase and it do not work.


Here is my configuration on each master and regionserver.
  <property>     <name>hbase.regionserver.wal.codec</name>     <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
  </property>   <property>   <name>hbase.region.server.rpc.scheduler.factory.class</name>
  <value>org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory</value> </property>
 <property>   <name>hbase.rpc.controllerfactory.class</name>   <value>org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory</value>
</property>




I need your help,thx!
------------------
The harder, more fortunate
Mime
View raw message