phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eisenhut, Roman" <roman.eisen...@tum.de>
Subject RE: sqlline.py kills all regionservers
Date Sun, 19 Nov 2017 20:25:23 GMT
It was a version issue indeed. After upgrading the cluster to the following versions, I can
connect to sqlline.py without killing every Regionserver.

Apache Phoenix 4.13
HBase 0.98.24
Hadoop 2.9.0

Thank you very much!


From: James Taylor [mailto:jamestaylor@apache.org]
Sent: Samstag, 18. November 2017 22:39
To: user@phoenix.apache.org
Subject: Re: sqlline.py kills all regionservers

That’s quite an old version of 0.98 and we no longer support Hadoop 1. Would it be possible
for you to upgrade your cluster and use Hadoop 2 instead?

On Sat, Nov 18, 2017 at 1:16 PM Eisenhut, Roman <roman.eisenhut@tum.de<mailto:roman.eisenhut@tum.de>>
wrote:
Dear Phoenix community,

I’m trying to implement apache-phoenix-4.13.0-HBase-0.98-bin on Hadoop-1.2.1 and HBase-0.98.6.1
on a 4 node cluster, running the following services:

  *   roman-master:

     *   NameNode
     *   SecondaryNamenode
     *   HMaster
     *   HQuorumPeer

  *   roman-rs-1:

     *   DataNode
     *   HRegionServer

  *   roman-rs-2:

     *   DataNode
     *   HRegionServer

  *   roman-rs-3:

     *   DataNode
     *   HRegionServer

I have copied the corresponding server.jar into the lib directory of my HBase installation
on every server.

ubuntu@roman-master:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15 phoenix-4.13.0-HBase-0.98-server.jar

ubuntu@roman-rs-1:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15 phoenix-4.13.0-HBase-0.98-server.jar

ubuntu@roman-rs-2:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16 phoenix-4.13.0-HBase-0.98-server.jar

ubuntu@roman-rs-3:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16 phoenix-4.13.0-HBase-0.98-server.jar

I can:

  *   Start Hadoop
  *   Start HBase

     *   Create a table in HBase
     *   Put data
     *   Get data

  *   Browse

     *   http://roman-master:50070/dfshealth.jsp
     *   http://roman-master:60010/master-status

jps shows all the processes are up and running on each machine and the http interfaces show
everything is running properly. It seems like the system is in perfect health. However, as
soon as I try to connect to phoenix via  /usr/local/ apache-phoenix-4.13.0-HBase-0.98-bin/bin/sqlline.py
roman-master:2181 on roman-master, where roman-maste is my zookeeper quorum, it kills every
regionservers on the entire cluster. After a long time, I don’t know exactly how long, sqlline
will return an output (can be found under [3]).

Connection call
ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin$<mailto:ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin$>
./sqlline.py roman-master:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:roman-master:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:roman-master:2181
17/11/18 19:24:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable

Underneath you’ll find the following files

  *   Conf files for

     *   Phoenix

        *   env.sh

     *   Hadoop

        *   hadoop-env.sh
        *   core-site.xml
        *   hdfs-site.xml

     *   HBase

        *   hbase-site.xml
        *   hbase-env.sh

  *   [1] master log on roman-master
  *   [2] regionserver log roman-rs-1
  *   [3] zookeeper log on roman-master
  *   [4] output of sqlline.py

I’m pretty sure there’s a reasonable explanation for this behavior. Nevertheless, I’ve
been trying to get phoenix up and running for days now and I can’t figure out what I’m
doing wrong. I hope someone can point me into the right direction.

Best regards,
Roman





Phoenix conf:
ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin/config$<mailto:ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin/config$>
cat env.sh
JAVA_HOME=/usr/lib/jvm/java-8-oracle/
HBASE_PATH=/usr/local/hbase-0.98.6.1-hadoop1/

Hadoop conf:
Hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
_EOF_

<?xml version="1.0"?>
<!-- core-site.xml -->

<configuration>
        <property>
                <name>fs.default.name<http://fs.default.name></name>
                <value>hdfs://roman-master/</value>
                <final>true</final>
        </property>
                <property>
                               <name>hadoop.tmp.dir</name>
                               <value>/usr/local/hadoop_tmp_dir</value>
                </property>
</configuration>

<?xml version="1.0"?>
<!-- hdfs-site.xml -->
<configuration>
                <property>
                               <name>dfs.name.dir</name>
                               <value>/usr/local/dfs/namenode</value>
                               <final>true</final>
                </property>
                <property>
                               <name>dfs.data.dir</name>
                               <value>/usr/local/dfs/datanode</value>
                               <final>true</final>
                </property>
                <property>
                               <name>fs.checkpoint.dir</name>
                               <value>/usr/local/dfs/checkpoint</value>
                               <final>true</final>
                </property>
</configuration>

HBase conf

hbase.env.sh<http://hbase.env.sh>
export HBASE_HEAPSIZE=4096
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"


hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
                <property>
                               <name>hbase.zookeeper.quorum</name>
                               <value>roman-master</value>
                </property>
<!--
        <property>
                <name>hbase.zookeeper.property.clientPort</name>
                <value>5000</value>
        </property>
-->
                <property>
                               <name>hbase.zookeeper.property.dataDir</name>
                               <value>/usr/local/zookeeper</value>
                </property>
                <property>
                               <name>hbase.rootdir</name>
                               <value>hdfs://roman-master/hbase</value>
                </property>
                <property>
                               <name>hbase.cluster.distributed</name>
                               <value>true</value>
                </property>
</configuration>


[1]
2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion:
Instantiated SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion:
Closing SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.: disabling compactions
& flushes
2017-11-18 19:42:08,790 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion:
Updates disabled for region SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:08,790 INFO  [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion:
Closed SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:08,801 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] catalog.MetaEditor:
Added 1
2017-11-18 19:42:08,801 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager:
Bulk assigning 1 region(s) across 3 server(s), round-robin=true
2017-11-18 19:42:08,802 DEBUG [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner:
Timeout-on-RIT=151000
2017-11-18 19:42:08,802 DEBUG [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.AssignmentManager:
Assigning 1 region(s) to roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,803 DEBUG [roman-master,60000,1511033853960-GeneralBulkAssigner-2] zookeeper.ZKAssign:
master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Async create of
unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:08,809 DEBUG [main-EventThread] master.OfflineCallback: rs={6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034128801, server=null}, server=roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,810 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback: rs={6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034128801, server=null}, server=roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,813 INFO  [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.AssignmentManager:
roman-rs-3,60020,1511033854738 unassigned znodes=1 of total=1
2017-11-18 19:42:08,813 INFO  [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.RegionStates:
Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034128803, server=null}
to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034128813, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:08,826 DEBUG [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.AssignmentManager:
Bulk assigning done for roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,826 DEBUG [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner:
bulk assigning total 1 regions to 3 servers, took 24ms, with 1 regions still in transition
2017-11-18 19:42:08,826 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager:
Bulk assigning done
2017-11-18 19:42:08,841 DEBUG [MASTER_TABLE_OPERATIONS-roman-master:60000-0] lock.ZKInterProcessLockBase:
Released /hbase/table-lock/SYSTEM.CATALOG/write-master:600000000000000
2017-11-18 19:42:08,841 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] handler.CreateTableHandler:
failed. null
2017-11-18 19:42:08,848 DEBUG [AM.ZK.Worker-pool2-t10] master.AssignmentManager: Handling
RS_ZK_REGION_OPENING, server=roman-rs-3,60020,1511033854738, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034128813, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:08,848 INFO  [AM.ZK.Worker-pool2-t10] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034128813, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4
state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:08,946 ERROR [FifoRpcScheduler.handler1-thread-12] master.HMaster: Region
server roman-rs-3,60020,1511033854738 reported a fatal error:
ABORTING region server roman-rs-3,60020,1511033854738: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl
threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
Cause:
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more

2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Handling
RS_ZK_REGION_FAILED_OPEN, server=roman-rs-3,60020,1511033854738, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:11,245 INFO  [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4
state=CLOSED, ts=1511034131245, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-3,60020,1511033854738 accepted as a dest server = false
2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2) available servers, forceNewPlan=true
2017-11-18 19:42:11,246 DEBUG [AM.ZK.Worker-pool2-t11] handler.ClosedRegionHandler: Handling
CLOSED event for 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Using pre-existing
plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-2,60020,1511033854813
2017-11-18 19:42:11,247 INFO  [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=CLOSED, ts=1511034131246, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034131247, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000,
quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
with OFFLINE state
2017-11-18 19:42:11,256 INFO  [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Assigning
SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813
2017-11-18 19:42:11,257 INFO  [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034131247, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,257 DEBUG [AM.ZK.Worker-pool2-t11] master.ServerManager: New admin connection
to roman-rs-2,60020,1511033854813
2017-11-18 19:42:11,424 DEBUG [AM.ZK.Worker-pool2-t12] master.AssignmentManager: Handling
M_ZK_REGION_OFFLINE, server=roman-rs-2,60020,1511033854813, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,444 DEBUG [AM.ZK.Worker-pool2-t13] master.AssignmentManager: Handling
RS_ZK_REGION_OPENING, server=roman-rs-2,60020,1511033854813, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,444 INFO  [AM.ZK.Worker-pool2-t13] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,591 ERROR [FifoRpcScheduler.handler1-thread-14] master.HMaster: Region
server roman-rs-2,60020,1511033854813 reported a fatal error:
ABORTING region server roman-rs-2,60020,1511033854813: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl
threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
Cause:
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more

2017-11-18 19:42:13,874 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Handling
RS_ZK_REGION_FAILED_OPEN, server=roman-rs-2,60020,1511033854813, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:13,874 INFO  [AM.ZK.Worker-pool2-t14] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=CLOSED, ts=1511034133874, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-2,60020,1511033854813 accepted as a dest server = false
2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=2) available servers, forceNewPlan=true
2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] handler.ClosedRegionHandler: Handling
CLOSED event for 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Using pre-existing
plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-1,60020,1511033854754
2017-11-18 19:42:13,888 INFO  [AM.ZK.Worker-pool2-t14] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=CLOSED, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034133888, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000,
quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
with OFFLINE state
2017-11-18 19:42:13,896 INFO  [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Assigning
SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:13,896 INFO  [AM.ZK.Worker-pool2-t14] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:13,896 DEBUG [AM.ZK.Worker-pool2-t14] master.ServerManager: New admin connection
to roman-rs-1,60020,1511033854754
2017-11-18 19:42:14,055 DEBUG [AM.ZK.Worker-pool2-t15] master.AssignmentManager: Handling
M_ZK_REGION_OFFLINE, server=roman-rs-1,60020,1511033854754, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:14,073 DEBUG [AM.ZK.Worker-pool2-t16] master.AssignmentManager: Handling
RS_ZK_REGION_OPENING, server=roman-rs-1,60020,1511033854754, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:14,073 INFO  [AM.ZK.Worker-pool2-t16] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:14,171 ERROR [FifoRpcScheduler.handler1-thread-16] master.HMaster: Region
server roman-rs-1,60020,1511033854754 reported a fatal error:
ABORTING region server roman-rs-1,60020,1511033854754: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl
threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
Cause:
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more

2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Handling
RS_ZK_REGION_FAILED_OPEN, server=roman-rs-1,60020,1511033854754, region=6608080647cc1ca0c6610b92de5ae0c4,
current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,457 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
state=CLOSED, ts=1511034136457, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-1,60020,1511033854754 accepted as a dest server = false
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2) available servers, forceNewPlan=true
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] handler.ClosedRegionHandler: Handling
CLOSED event for 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Using pre-existing
plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,478 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=CLOSED, ts=1511034136457, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136478, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,478 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000,
quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
with OFFLINE state
2017-11-18 19:42:16,488 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning
SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,488 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136478, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034136488, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,491 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment
of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813,
trying to assign elsewhere instead; try=1 of 10
java.io.IOException: Unexpected closed connection
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1450)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,494 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034136488, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000,
quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
with OFFLINE state
2017-11-18 19:42:16,502 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning
SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:16,502 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034136502, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,503 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment
of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754,
trying to assign elsewhere instead; try=2 of 10
java.io.IOException: Call to roman-rs-1/172.24.62.221:60020<http://172.24.62.221:60020>
failed on local exception: java.io.EOFException
                at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1489)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1461)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
                at java.io.DataInputStream.readInt(DataInputStream.java:392)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1076)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,504 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034136502, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000,
quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
with OFFLINE state
2017-11-18 19:42:16,513 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning
SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,513 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034136513, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,515 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment
of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813,
trying to assign elsewhere instead; try=3 of 10
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
                at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an
existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous
transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,515 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034136513, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000,
quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
with OFFLINE state
2017-11-18 19:42:16,538 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning
SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:16,538 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
state=PENDING_OPEN, ts=1511034136538, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,539 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment
of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754,
trying to assign elsewhere instead; try=4 of 10
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
                at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
                at org.apache.hadoop.hba
Mime
View raw message