phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "rajeshbabu@apache.org" <chrajeshbab...@gmail.com>
Subject Re: Forward: Re: local index turn disable when region split
Date Mon, 06 Nov 2017 11:11:52 GMT
Hi Vergil,

Thanks for sharing the information. Can you please share the complete logs
from where you have collected the info?
Thanks,
Rajeshbabu.

On Mon, Nov 6, 2017 at 10:08 AM, vergil <vergilchiu@foxmail.com> wrote:

> Thank for your reply.
>
> Here is My test environment:
> hbase:1.2.5
> phoenix:4.12
>
> Phoenix table with local index。
>
> The region will wait for a long time when the region split。
> *The region server webapp page show below:*
> Start TimeDescriptionStateStatus
> Mon Nov 06 17:49:27 CST 2017 Closing region TEST_TABLE_LOCAL,\x01,
> 1509961563238.9070a4f2ace4516f985c6c9e31427eee. RUNNING (since 1mins,
> 46sec ago) Finished memstore flush of ~39.94 MB/41879888,
> currentsize=14.31 MB/15002912 for region TEST_TABLE_LOCAL,\x01,
> 1509961563238.9070a4f2ace4516f985c6c9e31427eee. in 856ms,
> sequenceid=41861, compaction requested=true (since 1mins, 45sec ago)
> *The region server wait for a long time and log show below :*
> 2017-11-06 18:02:06,450 INFO  [B.defaultRpcServer.handler=27,queue=0,port=16020]
> index.PhoenixIndexFailurePolicy: Successfully disabled index
> TEST_INDEX_LOCAL due to an exception while writing updates.
> org.apache.phoenix.hbase.index.exception.MultiIndexWriteFailureException:
> Failed to write to multiple index tables
> at org.apache.phoenix.hbase.index.write.TrackingParallelWriterIndexCom
> mitter.write(TrackingParallelWriterIndexCommitter.java:229)
> at org.apache.phoenix.hbase.index.write.IndexWriter.write(
> IndexWriter.java:193)
> at org.apache.phoenix.hbase.index.write.IndexWriter.
> writeAndKillYourselfOnFailure(IndexWriter.java:154)
> at org.apache.phoenix.hbase.index.write.IndexWriter.
> writeAndKillYourselfOnFailure(IndexWriter.java:143)
> at org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.
> java:653)
> at org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:610)
> at org.apache.phoenix.hbase.index.Indexer.postBatchMutateIndispensably(
> Indexer.java:593)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$37.call(
> RegionCoprocessorHost.java:1034)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$
> RegionOperation.call(RegionCoprocessorHost.java:1673)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.
> execOperation(RegionCoprocessorHost.java:1749)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.
> execOperation(RegionCoprocessorHost.java:1705)
> at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.
> postBatchMutateIndispensably(RegionCoprocessorHost.java:1030)
> at org.apache.hadoop.hbase.regionserver.HRegion.
> doMiniBatchMutation(HRegion.java:3322)
> at org.apache.hadoop.hbase.regionserver.HRegion.
> batchMutate(HRegion.java:2881)
> at org.apache.hadoop.hbase.regionserver.HRegion.
> batchMutate(HRegion.java:2823)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> doBatchOp(RSRpcServices.java:758)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> doNonAtomicRegionMutation(RSRpcServices.java:720)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> multi(RSRpcServices.java:2168)
> at org.apache.hadoop.hbase.protobuf.generated.
> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33656)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2188)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
> RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:748)
>
> 2017-11-06 18:02:06,493 WARN  [hconnection-0x52dee0d4-shared--pool423-t2]
> client.AsyncProcess: #394, table=TEST_TABLE_LOCAL, attempt=11/11
> failed=40ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException:
> org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in
> 60000 ms. regionName=TEST_TABLE_LOCAL,\x01,1509961563238.
> 9070a4f2ace4516f985c6c9e31427eee., server=ht06,16020,1509939923675
> at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8176)
> at org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:8162)
> at org.apache.hadoop.hbase.regionserver.HRegion.
> startRegionOperation(HRegion.java:8071)
> at org.apache.hadoop.hbase.regionserver.HRegion.
> batchMutate(HRegion.java:2866)
> at org.apache.hadoop.hbase.regionserver.HRegion.
> batchMutate(HRegion.java:2823)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> doBatchOp(RSRpcServices.java:758)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> doNonAtomicRegionMutation(RSRpcServices.java:720)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
> multi(RSRpcServices.java:2168)
> at org.apache.hadoop.hbase.client.MultiServerCallable.
> call(MultiServerCallable.java:128)
>
>
> *The region will go back to normal after above exception,then table's
> local index turn disable.*
>
> ------------------ Original ------------------
> *From: * "Ted Yu";<yuzhihong@gmail.com>;
> *Send time:* Saturday, Nov 4, 2017 0:09 AM
> *To:* "user@phoenix.apache.org"<user@phoenix.apache.org>;
> *Subject: * Re: local index turn disable when region split
>
> Can you give us more information ?
>
> Release of hbase and Phoenix you use
>
> bq. The local index turn disable
>
> Can you pastebin the related exception(s) ?
>
> Snippet from region server log would also help.
>
> On Thu, Nov 2, 2017 at 11:31 PM, vergil <vergilchiu@foxmail.com> wrote:
>
>> Hi,all:
>>
>>
>> Here is my test table.
>> *create table test_table_local(id varchar primary key,f1 varchar,f2
>> varchar) salt_buckets=3;*
>> Add local index on it at the first time.
>> *create local index test_index_local on test_table_local(f1);*
>>
>> Then upsert data into it.
>> As the data increase,the region will split.
>> The local index turn disable when the region splits.
>> Local index data do not increase and it do not work.
>>
>> Here is my configuration on each master and regionserver.
>> <property> <name>hbase.regionserver.wal.codec</name>
>> <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
>> </property> <property> <name>hbase.region.server.rpc.scheduler.factory.class</name>
>> <value>org.apache.hadoop.hbase.ipc.PhoenixRpcSchedulerFactory</value>
>> </property> <property> <name>hbase.rpc.controllerfactory.class</name>
>> <value>org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory</value>
>> </property>
>>
>> I need your help,thx!
>> ------------------
>> The harder, more fortunate
>>
>>
>
>

Mime
View raw message