phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anupama agarwal <anu1...@gmail.com>
Subject Re: Dropping of Index can still leave some non-replayed writes Phoenix-2915
Date Wed, 15 Jun 2016 16:29:48 GMT
This didn't work Ankit. Please find below detailed logs on region server.
Please let me know what else I can try. When I try to run hbck command,
this is the error I get:

ERROR: There is a hole in the region chain between \x06 and \x07.  You need
to create a new .regioninfo and region dir in hdfs to plug the hole.


2016-06-15 21:53:41,132 ERROR org.apache.hadoop.hbase.client.AsyncProcess:
Cannot get replica 0 location for
{"totalColumns":25,"families":{"0":[{"timestamp":1465911182647,"tag":[],"qualifier":"_0","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_GROUP","vlen":8},{"timestamp":1465911182647,"tag":[],"qualifier":"0:EXTERNAL_REF_ID","vlen":10},{"timestamp":1465911182647,"tag":[],"qualifier":"0:PARTY_ID_FROM","vlen":4}]},"row":"\\x11PlatformServiceItem-mp-S177340305-OD306272384176181000-3627238417618100-FORWARD-S177340305\\x00RevenueAccrual\\x00fkmp\\x00fkmpra2016061401292e1933e3cec42"}

2016-06-15 21:53:41,140 ERROR org.apache.hadoop.hbase.client.AsyncProcess:
Cannot get replica 0 location for
{"ts":1465911182647,"totalColumns":12,"families":{"0":[{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_6","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_10","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_9","vlen":0},{"timestamp":1465911182647,"tag":[],"qualifier":"0:ACCRUAL_REF_8","vlen":0}]},"row":"\\x11PlatformServiceItem-mp-S177340305-OD306272384176181000-3627238417618100-FORWARD-S177340305\\x00RevenueAccrual\\x00fkmp\\x00fkmpra2016061401292e1933e3cec42"}

2016-06-15 21:53:41,216 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open
of
region=APL_ACCRUAL_SERVICE.ACCRUALS,\x06,1462851156716.eaee593f66a1ef282a2b5ae352624982.,
starting to roll back the global memstore size.

org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
Failed to write to multiple index tables

at
org.apache.phoenix.hbase.index.write.recovery.TrackingParallelWriterIndexCommitter.write(TrackingParallelWriterIndexCommitter.java:220)

at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:179)

at
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:169)

at org.apache.phoenix.hbase.index.Indexer.preWALRestore(Indexer.java:545)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$58.call(RegionCoprocessorHost.java:1422)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1663)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1738)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1695)

at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preWALRestore(RegionCoprocessorHost.java:1413)

at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEdits(HRegion.java:3940)

at
org.apache.hadoop.hbase.regionserver.HRegion.replayRecoveredEditsIfAny(HRegion.java:3797)

at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:969)

at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:841)

at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:814)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5828)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5794)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5765)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5721)

at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:5672)

at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:356)

at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:126)

at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2016-06-15 21:53:41,220 INFO
org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of
region {ENCODED => eaee593f66a1ef282a2b5ae352624982, NAME =>
'APL_ACCRUAL_SERVICE.ACCRUALS,\x06,1462851156716.eaee593f66a1ef282a2b5ae352624982.',
STARTKEY => '\x06', ENDKEY => '\x07'} failed, transitioning from OPENING to
FAILED_OPEN in ZK, expecting version 22

2016-06-15 21:57:50,092 INFO
org.apache.hadoop.hbase.io.hfile.LruBlockCache: totalSize=8.37 MB,
freeSize=7.97 GB, max=7.98 GB, blockCount=0, accesses=0, hits=0,
hitRatio=0, cachingAccesses=0, cachingHits=0,
cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0



On Wed, Jun 15, 2016 at 8:48 PM, Ankit Singhal <ankitsinghal59@gmail.com>
wrote:

> Yes, restart your cluster
>
> On Wed, Jun 15, 2016 at 8:17 AM, anupama agarwal <anu1307@gmail.com>
> wrote:
>
>> I have created async index with same name. But I am still getting the
>> same error. Should I restart my cluster for changes to reflect?
>> On Jun 15, 2016 8:38 PM, "Ankit Singhal" <ankitsinghal59@gmail.com>
>> wrote:
>>
>>> Hi Anupama,
>>>
>>> Option 1:-
>>> You can create a ASYNC index so that WAL can be replayed. And once your
>>> regions are up , remember to do the flush of data table before dropping the
>>> index.
>>>
>>> Option 2:-
>>> Create a table in hbase with the same name as index table name by using
>>> hbase shell.
>>>
>>> Regards,
>>> Ankit Singhal
>>>
>>>
>>> On Tue, Jun 14, 2016 at 11:19 PM, anupama agarwal <anu1307@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I have hit this error in phoenix, Phoenix-2915. It could be possible ,
>>>> that there are some index writes in WAL which are not replayed and the
>>>> index is dropped.
>>>>
>>>> And, now the table is not there, these writes cannot be replayed which
>>>> result in data table regions also to not come up. My data region is in
>>>> FAILED_TO_OPEN state. I have tried recreating the index, and still region
>>>> is not able to come up. I realise that this has been fixed in new version
>>>> of phoenix, but I am currently on phoenix 4.6 and Hbase 1.0. Can you please
>>>> suggest a solution?
>>>>
>>>
>>>
>

Mime
View raw message