phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bahubali Jain <bahub...@gmail.com>
Subject Region server crash
Date Mon, 03 Aug 2015 18:47:29 GMT
Hi,
I had a pretty well running setup until  I ran into a disk issue, after
which things changed for bad.
I have CDH 5.4 ,Hbase 1.0
I had to freshly install hbase , but now I am running into unexpected
issues.
When the sqlline.py connects for the first time to hbase , it creates
SYSTEM.CATALOG and SYSTEM.SEQUENCE.
During the processing of SYSTEM.SEQUENCE the region servers are crashing.

Below is the state from hbase master page(60010 port) from the "Regions in
Transition" section.
SYSTEM.SEQUENCE,`\x00\x00\x00,1438627204424.287e13aeb42e1310074052bcb60f147f.
state=PENDING_OPEN, ts=Mon Aug 03 18:40:07 UTC 2015 (21s ago),
server=hbasenode01.apac1.cis.trcloud,60020,1438627043841

Below is the snippet from region server logs, I dont see any exception as
such.
2015-08-03 18:40:12,738 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
SYSTEM.SEQUENCE,W\x00\x00\x00,1438627204424.7f5a88b5a5e35d88d464cb6473363ce1.
with server=hbasenode01.apac1.cis.trcloud,60020,1438627043841
2015-08-03 18:40:12,753 INFO org.apache.hadoop.hbase.io.hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=2, currentSize=23464,
freeSize=20266482, maxSize=20289946, heapSize=23464, minSize=19275448,
minFactor=0.95, multiSize=9637724, multiFactor=0.5, singleSize=4818862,
singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false,
cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2015-08-03 18:40:12,754 INFO
org.apache.hadoop.hbase.regionserver.compactions.CompactionConfiguration:
size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000;
off-peak ratio 5.000000; throttle point 2684354560; major period 604800000,
major jitter 0.500000, min locality to compact 0.000000
2015-08-03 18:40:13,028 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.SequenceRegionObserver from HTD
of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,029 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.hbase.index.Indexer from HTD of
SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,343 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Onlined 6ef5ea0883e5aceda0959dc631434675; next sequenceid=2
2015-08-03 18:40:13,344 INFO org.apache.phoenix.hbase.index.Indexer: Found
some outstanding index updates that didn't succeed during WAL replay -
attempting to replay now.
2015-08-03 18:40:13,344 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.ServerCachingEndpointImpl from
HTD of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,344 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver
from HTD of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,344 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
from HTD of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,344 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.ScanRegionObserver from HTD of
SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,347 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Post open deploy tasks
for
SYSTEM.SEQUENCE,Y\x00\x00\x00,1438627204424.6ef5ea0883e5aceda0959dc631434675.
2015-08-03 18:40:13,349 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.SequenceRegionObserver from HTD
of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,350 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.hbase.index.Indexer from HTD of
SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,350 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.ServerCachingEndpointImpl from
HTD of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,350 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver
from HTD of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,350 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
from HTD of SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,351 INFO
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost: Loaded
coprocessor org.apache.phoenix.coprocessor.ScanRegionObserver from HTD of
SYSTEM.SEQUENCE successfully.
2015-08-03 18:40:13,609 INFO org.apache.hadoop.hbase.MetaTableAccessor:
Updated row
SYSTEM.SEQUENCE,Y\x00\x00\x00,1438627204424.6ef5ea0883e5aceda0959dc631434675.
with server=hbasenode01.apac1.cis.trcloud,60020,1438627043841
2015-08-03 18:40:21,755 WARN
org.apache.hadoop.hbase.io.util.HeapMemorySizeUtil:
hbase.regionserver.global.memstore.upperLimit is deprecated by
hbase.regionserver.global.memstore.size
2015-08-03 18:40:21,886 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.5-cdh5.4.2--1, built on 05/19/2015 23:58
GMT
2015-08-03 18:40:21,886 INFO org.apache.zookeeper.ZooKeeper: Client
environment:host.name=hbasenode01.apac1.cis.trcloud
2015-08-03 18:40:21,886 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.version=1.7.0_67
2015-08-03 18:40:21,887 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2015-08-03 18:40:21,887 INFO org.apache.zookeeper.ZooKeeper: Client
environment:java.home=/usr/java/jdk1.7.0_67-cloudera/jre

Could any one please help on this...

Thanks,
Baahu

Mime
View raw message