phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Li <fancye...@gmail.com>
Subject can't start hbase after crash
Date Fri, 31 Oct 2014 05:03:04 GMT
my hbase(phoenix) crashes when the disk is full. I killed the hbase
and hadoop related processes. But After that I can't start hbase
anymore.
I am using ubuntu 12.04 and hadoop-1.2.1 and hbase 0.98.5 with phoenix 4.1.0
The region server print error like:

2014-10-31 12:33:49,835 INFO
[mobvoi-knowledge-graph-0,60020,1414729587199-recovery-writer--pool4-t2]
client.AsyncProcess: #9, waiting for some tasks to finish. Expected
max=0, tasksSent=31, tasksDone=30, currentTasksDone=30, retries=30
hasError=false, tableName=BAIDUMUSIC.BAIDUMUSIC_IDX

2014-10-31 12:33:49,842 INFO  [htable-pool9-t1] client.AsyncProcess:
#9, table=BAIDUMUSIC.BAIDUMUSIC_IDX, attempt=31/350 failed 1 ops, last
exception: org.apache.hadoop.hbase.exceptions.RegionOpeningException:
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
BAIDUMUSIC.BAIDUMUSIC_IDX,\x0D\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1414488087936.69e9a7efb9ee00b1ecfe50f825e7cc5b.
is opening on mobvoi-knowledge-graph-0,60020,1414729587199

        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2692)

        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4139)

        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3363)

        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)

        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)

        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)

        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)

        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)

        at java.lang.Thread.run(Thread.java:745)

 on mobvoi-knowledge-graph-0,60020,1414484323072, tracking started Fri
Oct 31 12:26:39 CST 2014, retrying after 20119 ms, replay 1 ops.

2014-10-31 12:33:55,747 INFO
[mobvoi-knowledge-graph-0,60020,1414729587199-recovery-writer--pool4-t3]
client.AsyncProcess: #25, waiting for some tasks to finish. Expected
max=0, tasksSent=31, tasksDone=30, currentTasksDone=30, retries=30
hasError=false, tableName=BAIDUMUSIC.BAIDUMUSIC_IDX

2014-10-31 12:33:55,755 INFO  [htable-pool24-t1] client.AsyncProcess:
#25, table=BAIDUMUSIC.BAIDUMUSIC_IDX, attempt=31/350 failed 2 ops,
last exception:
org.apache.hadoop.hbase.exceptions.RegionOpeningException:
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region
BAIDUMUSIC.BAIDUMUSIC_IDX,\x0C\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1414488087936.378bef2d741a9c00761160670329fc7c.
is opening on mobvoi-knowledge-graph-0,60020,1414729587199

        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2692)

        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4139)

        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3363)

        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)

        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)

        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)

        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)

        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)

        at java.lang.Thread.run(Thread.java:745)

 on mobvoi-knowledge-graph-0,60020,1414484323072, tracking started Fri
Oct 31 12:26:45 CST 2014, retrying after 20001 ms, replay 2 ops.

I also find warns like
wal.FSHLog: HDFS pipeline error detected. Found 1 replicas but
expecting no less than 3 replicas.  Requesting close of hlog.
But my hbase cluster is pesudo cluster that all processes are running
in a single machine and the hdfs (I have set dfs.replication=1 in
hdfs-site.xml)

Mime
View raw message