phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: can't start hbase after crash
Date Fri, 31 Oct 2014 17:25:35 GMT
Also, probably best to move this to the HBase mailing list, as this
isn't Phoenix-specific.

Thanks,
James

On Fri, Oct 31, 2014 at 10:05 AM, Jeffrey Zhong <jzhong@hortonworks.com> wrote:
>
> From the following error, it means region 69e9a7efb9ee00b1ecfe50f825e7cc5b
> is opening and can¹t serve any request. There are may some issues
> preventing the region from being opened. You can check Region Server log
> to see why 69e9a7efb9ee00b1ecfe50f825e7cc5b can¹t be opened.
>
>
> On 10/30/14, 10:05 PM, "Li Li" <fancyerii@gmail.com> wrote:
>
>>I have tried to set replication factor to 1 by hadoop fs -setrep -R 1
>>/ but no luck
>>
>>On Fri, Oct 31, 2014 at 1:03 PM, Li Li <fancyerii@gmail.com> wrote:
>>> my hbase(phoenix) crashes when the disk is full. I killed the hbase
>>> and hadoop related processes. But After that I can't start hbase
>>> anymore.
>>> I am using ubuntu 12.04 and hadoop-1.2.1 and hbase 0.98.5 with phoenix
>>>4.1.0
>>> The region server print error like:
>>>
>>> 2014-10-31 12:33:49,835 INFO
>>> [mobvoi-knowledge-graph-0,60020,1414729587199-recovery-writer--pool4-t2]
>>> client.AsyncProcess: #9, waiting for some tasks to finish. Expected
>>> max=0, tasksSent=31, tasksDone=30, currentTasksDone=30, retries=30
>>> hasError=false, tableName=BAIDUMUSIC.BAIDUMUSIC_IDX
>>>
>>> 2014-10-31 12:33:49,842 INFO  [htable-pool9-t1] client.AsyncProcess:
>>> #9, table=BAIDUMUSIC.BAIDUMUSIC_IDX, attempt=31/350 failed 1 ops, last
>>> exception: org.apache.hadoop.hbase.exceptions.RegionOpeningException:
>>> org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
>>>
>>>BAIDUMUSIC.BAIDUMUSIC_IDX,\x0D\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
>>>0\x00\x00\x00\x00\x00\x00,1414488087936.69e9a7efb9ee00b1ecfe50f825e7cc5b.
>>> is opening on mobvoi-knowledge-graph-0,60020,1414729587199
>>>
>>>         at
>>>org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName
>>>(HRegionServer.java:2692)
>>>
>>>         at
>>>org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe
>>>r.java:4139)
>>>
>>>         at
>>>org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.ja
>>>va:3363)
>>>
>>>         at
>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.c
>>>allBlockingMethod(ClientProtos.java:29593)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114
>>>)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
>>>
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>>  on mobvoi-knowledge-graph-0,60020,1414484323072, tracking started Fri
>>> Oct 31 12:26:39 CST 2014, retrying after 20119 ms, replay 1 ops.
>>>
>>> 2014-10-31 12:33:55,747 INFO
>>> [mobvoi-knowledge-graph-0,60020,1414729587199-recovery-writer--pool4-t3]
>>> client.AsyncProcess: #25, waiting for some tasks to finish. Expected
>>> max=0, tasksSent=31, tasksDone=30, currentTasksDone=30, retries=30
>>> hasError=false, tableName=BAIDUMUSIC.BAIDUMUSIC_IDX
>>>
>>> 2014-10-31 12:33:55,755 INFO  [htable-pool24-t1] client.AsyncProcess:
>>> #25, table=BAIDUMUSIC.BAIDUMUSIC_IDX, attempt=31/350 failed 2 ops,
>>> last exception:
>>> org.apache.hadoop.hbase.exceptions.RegionOpeningException:
>>> org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
>>>
>>>BAIDUMUSIC.BAIDUMUSIC_IDX,\x0C\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0
>>>0\x00\x00\x00\x00\x00\x00,1414488087936.378bef2d741a9c00761160670329fc7c.
>>> is opening on mobvoi-knowledge-graph-0,60020,1414729587199
>>>
>>>         at
>>>org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName
>>>(HRegionServer.java:2692)
>>>
>>>         at
>>>org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServe
>>>r.java:4139)
>>>
>>>         at
>>>org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.ja
>>>va:3363)
>>>
>>>         at
>>>org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.c
>>>allBlockingMethod(ClientProtos.java:29593)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114
>>>)
>>>
>>>         at
>>>org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
>>>
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>>  on mobvoi-knowledge-graph-0,60020,1414484323072, tracking started Fri
>>> Oct 31 12:26:45 CST 2014, retrying after 20001 ms, replay 2 ops.
>>>
>>> I also find warns like
>>> wal.FSHLog: HDFS pipeline error detected. Found 1 replicas but
>>> expecting no less than 3 replicas.  Requesting close of hlog.
>>> But my hbase cluster is pesudo cluster that all processes are running
>>> in a single machine and the hdfs (I have set dfs.replication=1 in
>>> hdfs-site.xml)
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

Mime
View raw message