phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: ABORTING region server and following HBase cluster "crash"
Date Tue, 11 Sep 2018 01:08:13 GMT
Did you update the HBase jars on all RegionServers?

Make sure that you have all of the Regions assigned (no RITs). There 
could be a pretty simple explanation as to why the index can't be 
written to.

On 9/9/18 3:46 PM, Batyrshin Alexander wrote:
> Correct me if im wrong.
> 
> But looks like if you have A and B region server that has index and 
> primary table then possible situation like this.
> 
> A and B under writes on table with indexes
> A - crash
> B failed on index update because A is not operating then B starting aborting
> A after restart try to rebuild index from WAL but B at this time is 
> aborting then A starting aborting too
>  From this moment nothing happens (0 requests to region servers) and A 
> and B is not responsible from Master-status web interface
> 
> 
>> On 9 Sep 2018, at 04:38, Batyrshin Alexander <0x62ash@gmail.com 
>> <mailto:0x62ash@gmail.com>> wrote:
>>
>> After update we still can't recover HBase cluster. Our region servers 
>> ABORTING over and over:
>>
>> prod003:
>> Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=92,queue=2,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod003,60020,1536446665703: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:51:27 prod003 hbase[1440]: 2018-09-09 02:51:27,395 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=77,queue=7,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod003,60020,1536446665703: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:52:19 prod003 hbase[1440]: 2018-09-09 02:52:19,224 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=82,queue=2,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod003,60020,1536446665703: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:52:28 prod003 hbase[1440]: 2018-09-09 02:52:28,922 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=94,queue=4,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod003,60020,1536446665703: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:55:02 prod003 hbase[957]: 2018-09-09 02:55:02,096 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=95,queue=5,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod003,60020,1536450772841: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:55:18 prod003 hbase[957]: 2018-09-09 02:55:18,793 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=97,queue=7,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod003,60020,1536450772841: Could not update the index table, 
>> killing server region because couldn't write to an index table
>>
>> prod004:
>> Sep 09 02:52:13 prod004 hbase[4890]: 2018-09-09 02:52:13,541 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=83,queue=3,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod004,60020,1536446387325: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:52:50 prod004 hbase[4890]: 2018-09-09 02:52:50,264 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=75,queue=5,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod004,60020,1536446387325: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:53:40 prod004 hbase[4890]: 2018-09-09 02:53:40,709 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=66,queue=6,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod004,60020,1536446387325: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:54:00 prod004 hbase[4890]: 2018-09-09 02:54:00,060 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=89,queue=9,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod004,60020,1536446387325: Could not update the index table, 
>> killing server region because couldn't write to an index table
>>
>> prod005:
>> Sep 09 02:52:50 prod005 hbase[3772]: 2018-09-09 02:52:50,661 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=65,queue=5,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod005,60020,1536446400009: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:53:27 prod005 hbase[3772]: 2018-09-09 02:53:27,542 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=90,queue=0,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod005,60020,1536446400009: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:54:00 prod005 hbase[3772]: 2018-09-09 02:53:59,915 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=7,queue=7,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod005,60020,1536446400009: Could not update the index table, 
>> killing server region because couldn't write to an index table
>> Sep 09 02:54:30 prod005 hbase[3772]: 2018-09-09 02:54:30,058 FATAL 
>> [RpcServer.default.FPBQ.Fifo.handler=16,queue=6,port=60020] 
>> regionserver.HRegionServer: ABORTING region 
>> server prod005,60020,1536446400009: Could not update the index table, 
>> killing server region because couldn't write to an index table
>>
>> And so on...
>>
>> Trace is the same everywhere:
>>
>> Sep 09 02:54:30 prod005 hbase[3772]: 
>> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException: 
>>  disableIndexOnFailure=true, Failed to write to multiple index 
>> tables: [KM_IDX1, KM_IDX2, KM_HISTORY_IDX1, KM_HISTORY_IDX2, 
>> KM_HISTORY_IDX3]
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:235)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:620)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:595)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:578)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1048)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1711)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1044)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3646)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3108)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3050)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commitBatch(UngroupedAggregateRegionObserver.java:271)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commitBatchWithRetries(UngroupedAggregateRegionObserver.java:241)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.rebuildIndices(UngroupedAggregateRegionObserver.java:1068)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:386)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:239)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:287)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2843)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3080)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2354)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
>> Sep 09 02:54:30 prod005 hbase[3772]:         at 
>> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
>>
>>> On 9 Sep 2018, at 01:44, Batyrshin Alexander <0x62ash@gmail.com 
>>> <mailto:0x62ash@gmail.com>> wrote:
>>>
>>> Thank you.
>>> We're updating our cluster right now...
>>>
>>>
>>>> On 9 Sep 2018, at 01:39, Ted Yu <yuzhihong@gmail.com 
>>>> <mailto:yuzhihong@gmail.com>> wrote:
>>>>
>>>> It seems you should deploy hbase with the following fix:
>>>>
>>>> HBASE-21069 NPE in StoreScanner.updateReaders causes RS to crash
>>>>
>>>> 1.4.7 was recently released.
>>>>
>>>> FYI
>>>>
>>>> On Sat, Sep 8, 2018 at 3:32 PM Batyrshin Alexander 
>>>> <0x62ash@gmail.com <mailto:0x62ash@gmail.com>> wrote:
>>>>
>>>>      Hello,
>>>>
>>>>     We got this exception from *prod006* server
>>>>
>>>>     Sep 09 00:38:02 prod006 hbase[18907]: 2018-09-09 00:38:02,532
>>>>     FATAL [MemStoreFlusher.1] regionserver.HRegionServer: ABORTING
>>>>     region server prod006,60020,1536235102833: Replay of
>>>>     WAL required. Forcing server shutdown
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:
>>>>     org.apache.hadoop.hbase.DroppedSnapshotException:
>>>>     region: KM,c\xEF\xBF\xBD\x16I7\xEF\xBF\xBD\x0A"A\xEF\xBF\xBDd\xEF\xBF\xBD\xEF\xBF\xBD\x19\x07t,1536178245576.60c121ba50e67f2429b9ca2ba2a11bad.
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2645)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2322)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2284)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2170)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2095)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:508)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:478)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     java.lang.Thread.run(Thread.java:748)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]: Caused by:
>>>>     java.lang.NullPointerException
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     java.util.ArrayList.<init>(ArrayList.java:178)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.StoreScanner.updateReaders(StoreScanner.java:863)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HStore.notifyChangedReadersObservers(HStore.java:1172)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HStore.updateStorefiles(HStore.java:1145)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HStore.access$900(HStore.java:122)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.commit(HStore.java:2505)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2600)
>>>>     Sep 09 00:38:02 prod006 hbase[18907]:         ... 9 more
>>>>     Sep 09 00:38:02 prod006 hbase[18907]: 2018-09-09 00:38:02,532
>>>>     FATAL [MemStoreFlusher.1] regionserver.HRegionServer:
>>>>     RegionServer abort: loaded coprocessors
>>>>     are: [org.apache.hadoop.hbase.regionserver.IndexHalfStoreFileReaderGenerator,
>>>>     org.apache.phoenix.coprocessor.SequenceRegionObserver,
>>>>     org.apache.phoenix.c
>>>>
>>>>     After that we got ABORTING on almost every Region Servers in
>>>>     cluster with different reasons:
>>>>
>>>>     *prod003*
>>>>     Sep 09 01:12:11 prod003 hbase[11552]: 2018-09-09 01:12:11,799
>>>>     FATAL [PostOpenDeployTasks:88bfac1dfd807c4cd1e9c1f31b4f053f]
>>>>     regionserver.HRegionServer: ABORTING region
>>>>     server prod003,60020,1536444066291: Exception running
>>>>     postOpenDeployTasks; region=88bfac1dfd807c4cd1e9c1f31b4f053f
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:
>>>>     java.io.InterruptedIOException: #139, interrupted.
>>>>     currentNumberOfTask=8
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1853)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1823)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1899)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:250)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:213)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1484)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.client.HTable.put(HTable.java:1031)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.MetaTableAccessor.put(MetaTableAccessor.java:1033)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.MetaTableAccessor.putToMetaTable(MetaTableAccessor.java:1023)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.MetaTableAccessor.updateLocation(MetaTableAccessor.java:1433)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.MetaTableAccessor.updateRegionLocation(MetaTableAccessor.java:1400)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegionServer.postOpenDeployTasks(HRegionServer.java:2041)
>>>>     Sep 09 01:12:11 prod003 hbase[11552]:         at
>>>>     org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler$PostOpenDeployTasksThread.run(OpenRegionHandler.java:329)
>>>>
>>>>     *prod002*
>>>>     Sep 09 01:12:30 prod002 hbase[29056]: 2018-09-09 01:12:30,144
>>>>     FATAL
>>>>     [RpcServer.default.FPBQ.Fifo.handler=36,queue=6,port=60020]
>>>>     regionserver.HRegionServer: ABORTING region
>>>>     server prod002,60020,1536235138673: Could not update the index
>>>>     table, killing server region because couldn't write to an index
>>>>     table
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:
>>>>     org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
>>>>      disableIndexOnFailure=true, Failed to write to multiple index
>>>>     tables: [KM_IDX1, KM_IDX2, KM_HISTORY1, KM_HISTORY2,
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:235)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:195)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:156)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:145)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:620)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:595)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(Indexer.java:578)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(RegionCoprocessorHost.java:1048)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1711)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1789)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1745)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutateIndispensably(RegionCoprocessorHost.java:1044)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:3646)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3108)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:3050)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commitBatch(UngroupedAggregateRegionObserver.java:271)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.access$000(UngroupedAggregateRegionObserver.java:164)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver$1.doMutation(UngroupedAggregateRegionObserver.java:246)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.index.PhoenixIndexFailurePolicy.doBatchWithRetries(PhoenixIndexFailurePolicy.java:455)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.handleIndexWriteException(UngroupedAggregateRegionObserver.java:929)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.commitBatchWithRetries(UngroupedAggregateRegionObserver.java:243)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.rebuildIndices(UngroupedAggregateRegionObserver.java:1077)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostScannerOpen(UngroupedAggregateRegionObserver.java:386)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.overrideDelegate(BaseScannerRegionObserver.java:239)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.phoenix.coprocessor.BaseScannerRegionObserver$RegionScannerHolder.nextRaw(BaseScannerRegionObserver.java:287)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2843)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3080)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:36613)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2354)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
>>>>     Sep 09 01:12:30 prod002 hbase[29056]:         at
>>>>     org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
>>>>
>>>>
>>>>     And etc...
>>>>
>>>>     Master-status web interface shows that contact lost from this
>>>>     aborted servers. 
>>>>
>>>
>>
> 

Mime
View raw message