phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eisenhut, Roman" <roman.eisen...@tum.de>
Subject sqlline.py kills all regionservers
Date Sat, 18 Nov 2017 21:15:57 GMT
Dear Phoenix community,

I'm trying to implement apache-phoenix-4.13.0-HBase-0.98-bin on Hadoop-1.2.1 and HBase-0.98.6.1 on a 4 node cluster, running the following services:

  *   roman-master:
     *   NameNode
     *   SecondaryNamenode
     *   HMaster
     *   HQuorumPeer
  *   roman-rs-1:
     *   DataNode
     *   HRegionServer
  *   roman-rs-2:
     *   DataNode
     *   HRegionServer
  *   roman-rs-3:
     *   DataNode
     *   HRegionServer

I have copied the corresponding server.jar into the lib directory of my HBase installation on every server.

ubuntu@roman-master:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15 phoenix-4.13.0-HBase-0.98-server.jar

ubuntu@roman-rs-1:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15 phoenix-4.13.0-HBase-0.98-server.jar

ubuntu@roman-rs-2:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16 phoenix-4.13.0-HBase-0.98-server.jar

ubuntu@roman-rs-3:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep phoenix
-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16 phoenix-4.13.0-HBase-0.98-server.jar

I can:

  *   Start Hadoop
  *   Start HBase
     *   Create a table in HBase
     *   Put data
     *   Get data
  *   Browse
     *   http://roman-master:50070/dfshealth.jsp
     *   http://roman-master:60010/master-status

jps shows all the processes are up and running on each machine and the http interfaces show everything is running properly. It seems like the system is in perfect health. However, as soon as I try to connect to phoenix via  /usr/local/ apache-phoenix-4.13.0-HBase-0.98-bin/bin/sqlline.py roman-master:2181 on roman-master, where roman-maste is my zookeeper quorum, it kills every regionservers on the entire cluster. After a long time, I don't know exactly how long, sqlline will return an output (can be found under [3]).

Connection call
ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin$<mailto:ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin$> ./sqlline.py roman-master:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:roman-master:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:roman-master:2181
17/11/18 19:24:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Underneath you'll find the following files

  *   Conf files for
     *   Phoenix
        *   env.sh
     *   Hadoop
        *   hadoop-env.sh
        *   core-site.xml
        *   hdfs-site.xml
     *   HBase
        *   hbase-site.xml
        *   hbase-env.sh
  *   [1] master log on roman-master
  *   [2] regionserver log roman-rs-1
  *   [3] zookeeper log on roman-master
  *   [4] output of sqlline.py

I'm pretty sure there's a reasonable explanation for this behavior. Nevertheless, I've been trying to get phoenix up and running for days now and I can't figure out what I'm doing wrong. I hope someone can point me into the right direction.

Best regards,
Roman





Phoenix conf:
ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin/config$<mailto:ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin/config$> cat env.sh
JAVA_HOME=/usr/lib/jvm/java-8-oracle/
HBASE_PATH=/usr/local/hbase-0.98.6.1-hadoop1/

Hadoop conf:
Hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-oracle
# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000
# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_NAMENODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote $HADOOP_DATANODE_OPTS"
export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_BALANCER_OPTS"
export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote $HADOOP_JOBTRACKER_OPTS"
_EOF_

<?xml version="1.0"?>
<!-- core-site.xml -->

<configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://roman-master/</value>
                <final>true</final>
        </property>
                <property>
                               <name>hadoop.tmp.dir</name>
                               <value>/usr/local/hadoop_tmp_dir</value>
                </property>
</configuration>

<?xml version="1.0"?>
<!-- hdfs-site.xml -->
<configuration>
                <property>
                               <name>dfs.name.dir</name>
                               <value>/usr/local/dfs/namenode</value>
                               <final>true</final>
                </property>
                <property>
                               <name>dfs.data.dir</name>
                               <value>/usr/local/dfs/datanode</value>
                               <final>true</final>
                </property>
                <property>
                               <name>fs.checkpoint.dir</name>
                               <value>/usr/local/dfs/checkpoint</value>
                               <final>true</final>
                </property>
</configuration>

HBase conf

hbase.env.sh
export HBASE_HEAPSIZE=4096
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"


hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
                <property>
                               <name>hbase.zookeeper.quorum</name>
                               <value>roman-master</value>
                </property>
<!--
        <property>
                <name>hbase.zookeeper.property.clientPort</name>
                <value>5000</value>
        </property>
-->
                <property>
                               <name>hbase.zookeeper.property.dataDir</name>
                               <value>/usr/local/zookeeper</value>
                </property>
                <property>
                               <name>hbase.rootdir</name>
                               <value>hdfs://roman-master/hbase</value>
                </property>
                <property>
                               <name>hbase.cluster.distributed</name>
                               <value>true</value>
                </property>
</configuration>


[1]
2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion: Instantiated SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion: Closing SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.: disabling compactions & flushes
2017-11-18 19:42:08,790 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion: Updates disabled for region SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:08,790 INFO  [RegionOpenAndInitThread-SYSTEM.CATALOG-1] regionserver.HRegion: Closed SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:08,801 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] catalog.MetaEditor: Added 1
2017-11-18 19:42:08,801 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager: Bulk assigning 1 region(s) across 3 server(s), round-robin=true
2017-11-18 19:42:08,802 DEBUG [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner: Timeout-on-RIT=151000
2017-11-18 19:42:08,802 DEBUG [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.AssignmentManager: Assigning 1 region(s) to roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,803 DEBUG [roman-master,60000,1511033853960-GeneralBulkAssigner-2] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Async create of unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:08,809 DEBUG [main-EventThread] master.OfflineCallback: rs={6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034128801, server=null}, server=roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,810 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback: rs={6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034128801, server=null}, server=roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,813 INFO  [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.AssignmentManager: roman-rs-3,60020,1511033854738 unassigned znodes=1 of total=1
2017-11-18 19:42:08,813 INFO  [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034128803, server=null} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034128813, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:08,826 DEBUG [roman-master,60000,1511033853960-GeneralBulkAssigner-2] master.AssignmentManager: Bulk assigning done for roman-rs-3,60020,1511033854738
2017-11-18 19:42:08,826 DEBUG [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner: bulk assigning total 1 regions to 3 servers, took 24ms, with 1 regions still in transition
2017-11-18 19:42:08,826 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager: Bulk assigning done
2017-11-18 19:42:08,841 DEBUG [MASTER_TABLE_OPERATIONS-roman-master:60000-0] lock.ZKInterProcessLockBase: Released /hbase/table-lock/SYSTEM.CATALOG/write-master:600000000000000
2017-11-18 19:42:08,841 INFO  [MASTER_TABLE_OPERATIONS-roman-master:60000-0] handler.CreateTableHandler: failed. null
2017-11-18 19:42:08,848 DEBUG [AM.ZK.Worker-pool2-t10] master.AssignmentManager: Handling RS_ZK_REGION_OPENING, server=roman-rs-3,60020,1511033854738, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034128813, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:08,848 INFO  [AM.ZK.Worker-pool2-t10] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034128813, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:08,946 ERROR [FifoRpcScheduler.handler1-thread-12] master.HMaster: Region server roman-rs-3,60020,1511033854738 reported a fatal error:
ABORTING region server roman-rs-3,60020,1511033854738: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
Cause:
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more

2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN, server=roman-rs-3,60020,1511033854738, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:11,245 INFO  [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034131245, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-3,60020,1511033854738 accepted as a dest server = false
2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2) available servers, forceNewPlan=true
2017-11-18 19:42:11,246 DEBUG [AM.ZK.Worker-pool2-t11] handler.ClosedRegionHandler: Handling CLOSED event for 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Using pre-existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-2,60020,1511033854813
2017-11-18 19:42:11,247 INFO  [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034131246, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034131247, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:11,256 INFO  [AM.ZK.Worker-pool2-t11] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813
2017-11-18 19:42:11,257 INFO  [AM.ZK.Worker-pool2-t11] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034131247, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,257 DEBUG [AM.ZK.Worker-pool2-t11] master.ServerManager: New admin connection to roman-rs-2,60020,1511033854813
2017-11-18 19:42:11,424 DEBUG [AM.ZK.Worker-pool2-t12] master.AssignmentManager: Handling M_ZK_REGION_OFFLINE, server=roman-rs-2,60020,1511033854813, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,444 DEBUG [AM.ZK.Worker-pool2-t13] master.AssignmentManager: Handling RS_ZK_REGION_OPENING, server=roman-rs-2,60020,1511033854813, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,444 INFO  [AM.ZK.Worker-pool2-t13] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:11,591 ERROR [FifoRpcScheduler.handler1-thread-14] master.HMaster: Region server roman-rs-2,60020,1511033854813 reported a fatal error:
ABORTING region server roman-rs-2,60020,1511033854813: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
Cause:
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more

2017-11-18 19:42:13,874 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN, server=roman-rs-2,60020,1511033854813, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:13,874 INFO  [AM.ZK.Worker-pool2-t14] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034133874, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-2,60020,1511033854813 accepted as a dest server = false
2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=2) available servers, forceNewPlan=true
2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14] handler.ClosedRegionHandler: Handling CLOSED event for 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Using pre-existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-1,60020,1511033854754
2017-11-18 19:42:13,888 INFO  [AM.ZK.Worker-pool2-t14] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034133888, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:13,896 INFO  [AM.ZK.Worker-pool2-t14] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:13,896 INFO  [AM.ZK.Worker-pool2-t14] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:13,896 DEBUG [AM.ZK.Worker-pool2-t14] master.ServerManager: New admin connection to roman-rs-1,60020,1511033854754
2017-11-18 19:42:14,055 DEBUG [AM.ZK.Worker-pool2-t15] master.AssignmentManager: Handling M_ZK_REGION_OFFLINE, server=roman-rs-1,60020,1511033854754, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:14,073 DEBUG [AM.ZK.Worker-pool2-t16] master.AssignmentManager: Handling RS_ZK_REGION_OPENING, server=roman-rs-1,60020,1511033854754, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:14,073 INFO  [AM.ZK.Worker-pool2-t16] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:14,171 ERROR [FifoRpcScheduler.handler1-thread-16] master.HMaster: Region server roman-rs-1,60020,1511033854754 reported a fatal error:
ABORTING region server roman-rs-1,60020,1511033854754: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
Cause:
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more

2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN, server=roman-rs-1,60020,1511033854754, region=6608080647cc1ca0c6610b92de5ae0c4, current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,457 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034136457, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-1,60020,1511033854754 accepted as a dest server = false
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2) available servers, forceNewPlan=true
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] handler.ClosedRegionHandler: Handling CLOSED event for 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Using pre-existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,478 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034136457, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136478, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,478 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:16,488 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,488 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136478, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136488, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,491 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813, trying to assign elsewhere instead; try=1 of 10
java.io.IOException: Unexpected closed connection
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1450)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,494 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136488, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:16,502 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:16,502 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136502, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,503 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=2 of 10
java.io.IOException: Call to roman-rs-1/172.24.62.221:60020 failed on local exception: java.io.EOFException
                at org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1489)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1461)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException
                at java.io.DataInputStream.readInt(DataInputStream.java:392)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1076)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,504 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136502, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:16,513 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,513 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136513, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,515 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813, trying to assign elsewhere instead; try=3 of 10
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
                at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,515 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136513, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:16,538 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:16,538 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136538, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,539 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=4 of 10
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
                at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,539 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
2017-11-18 19:42:16,540 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,540 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:16,540 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=5 of 10
org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-1/172.24.62.221:60020
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,540 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
2017-11-18 19:42:16,540 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-3,60020,1511033854738; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,540 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136540, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136540, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,540 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:16,552 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-3,60020,1511033854738
2017-11-18 19:42:16,552 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136540, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136552, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:16,554 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-3,60020,1511033854738, trying to assign elsewhere instead; try=6 of 10
java.io.IOException: Unexpected closed connection
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1450)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,554 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-3,60020,1511033854738 accepted as a dest server = true
2017-11-18 19:42:16,554 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,554 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136552, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136554, server=roman-rs-3,60020,1511033854738}
2017-11-18 19:42:16,554 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:16,562 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,562 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136554, server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136562, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,562 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-2,60020,1511033854813, trying to assign elsewhere instead; try=7 of 10
org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-2/172.24.62.222:60020
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,563 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
2017-11-18 19:42:16,563 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,563 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136562, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136563, server=roman-rs-2,60020,1511033854813}
2017-11-18 19:42:16,563 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
2017-11-18 19:42:16,570 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:16,570 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136563, server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136570, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:16,570 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=8 of 10
org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-1/172.24.62.221:60020
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:853)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:16,570 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Found an existing plan for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
2017-11-18 19:42:16,570 DEBUG [AM.ZK.Worker-pool2-t17] master.AssignmentManager: No previous transition plan found (or ignoring an existing plan) for SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated random plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3) available servers, forceNewPlan=true
2017-11-18 19:42:16,570 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Trying to re-assign SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to the same failed server.
2017-11-18 19:42:16,632 INFO  [main-EventThread] zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [roman-rs-2,60020,1511033854813]
2017-11-18 19:42:16,638 DEBUG [main-EventThread] master.AssignmentManager: based on AM, current region=hbase:meta,,1.1588230740 is on server=roman-rs-3,60020,1511033854738 server being checked: roman-rs-2,60020,1511033854813
2017-11-18 19:42:16,649 DEBUG [main-EventThread] master.ServerManager: Added=roman-rs-2,60020,1511033854813 to dead servers, submitted shutdown handler to be executed meta=false
2017-11-18 19:42:16,667 DEBUG [main-EventThread] zookeeper.RegionServerTracker: RS node: /hbase/rs/roman-rs-3,60020,1511033854738 data: null
2017-11-18 19:42:16,667 INFO  [main-EventThread] zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [roman-rs-1,60020,1511033854754]
2017-11-18 19:42:16,668 DEBUG [main-EventThread] master.AssignmentManager: based on AM, current region=hbase:meta,,1.1588230740 is on server=roman-rs-3,60020,1511033854738 server being checked: roman-rs-1,60020,1511033854754
2017-11-18 19:42:16,669 DEBUG [main-EventThread] master.ServerManager: Added=roman-rs-1,60020,1511033854754 to dead servers, submitted shutdown handler to be executed meta=false
2017-11-18 19:42:16,669 INFO  [main-EventThread] zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [roman-rs-3,60020,1511033854738]
2017-11-18 19:42:16,676 DEBUG [main-EventThread] master.AssignmentManager: based on AM, current region=hbase:meta,,1.1588230740 is on server=roman-rs-3,60020,1511033854738 server being checked: roman-rs-3,60020,1511033854738
2017-11-18 19:42:16,677 DEBUG [main-EventThread] master.ServerManager: Added=roman-rs-3,60020,1511033854738 to dead servers, submitted shutdown handler to be executed meta=true
2017-11-18 19:42:16,679 INFO  [MASTER_META_SERVER_OPERATIONS-roman-master:60000-0] handler.MetaServerShutdownHandler: Splitting hbase:meta logs for roman-rs-3,60020,1511033854738
2017-11-18 19:42:16,680 DEBUG [MASTER_SERVER_OPERATIONS-roman-master:60000-1] client.HConnectionManager$HConnectionImplementation: Removed all cached region locations that map to roman-rs-3,60020,1511033854738
2017-11-18 19:42:16,686 DEBUG [MASTER_META_SERVER_OPERATIONS-roman-master:60000-0] master.MasterFileSystem: Renamed region directory: hdfs://roman-master/hbase/WALs/roman-rs-3,60020,1511033854738-splitting
2017-11-18 19:42:16,686 INFO  [MASTER_META_SERVER_OPERATIONS-roman-master:60000-0] master.SplitLogManager: dead splitlog workers [roman-rs-3,60020,1511033854738]
2017-11-18 19:42:16,689 DEBUG [MASTER_META_SERVER_OPERATIONS-roman-master:60000-0] master.SplitLogManager: Scheduling batch of logs to split
2017-11-18 19:42:16,690 INFO  [MASTER_META_SERVER_OPERATIONS-roman-master:60000-0] master.SplitLogManager: started splitting 1 logs in [hdfs://roman-master/hbase/WALs/roman-rs-3,60020,1511033854738-splitting]
2017-11-18 19:42:16,704 DEBUG [main-EventThread] master.SplitLogManager: put up splitlog task at znode /hbase/splitWAL/WALs%2Froman-rs-3%2C60020%2C1511033854738-splitting%2Froman-rs-3%252C60020%252C1511033854738.1511033861810.meta
2017-11-18 19:42:16,706 DEBUG [main-EventThread] master.SplitLogManager: task not yet acquired /hbase/splitWAL/WALs%2Froman-rs-3%2C60020%2C1511033854738-splitting%2Froman-rs-3%252C60020%252C1511033854738.1511033861810.meta ver = 0
2017-11-18 19:42:16,818 INFO  [roman-master,60000,1511033853960.splitLogManagerTimeoutMonitor] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Froman-rs-3%2C60020%2C1511033854738-splitting%2Froman-rs-3%252C60020%252C1511033854738.1511033861810.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-18 19:42:18,572 INFO  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Assigning SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754
2017-11-18 19:42:18,572 DEBUG [AM.ZK.Worker-pool2-t17] master.ServerManager: New admin connection to roman-rs-1,60020,1511033854754
2017-11-18 19:42:18,574 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Failed assignment of SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=9 of 10
java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
                at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
                at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
                at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
                at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
                at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
                at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
                at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
                at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:18,575 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Can't move 6608080647cc1ca0c6610b92de5ae0c4, there is no destination server available.
2017-11-18 19:42:18,575 WARN  [AM.ZK.Worker-pool2-t17] master.RegionStates: Failed to open/close 6608080647cc1ca0c6610b92de5ae0c4 on roman-rs-1,60020,1511033854754, set to FAILED_OPEN
2017-11-18 19:42:18,575 INFO  [AM.ZK.Worker-pool2-t17] master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034138572, server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4 state=FAILED_OPEN, ts=1511034138575, server=roman-rs-1,60020,1511033854754}
2017-11-18 19:42:18,575 WARN  [AM.ZK.Worker-pool2-t17] master.AssignmentManager: Unable to find a viable location to assign region SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:18,576 WARN  [AM.ZK.Worker-pool2-t18] master.AssignmentManager: Attempted to handle region transition for server but it is not online: 6608080647cc1ca0c6610b92de5ae0c4, region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., state=M_ZK_REGION_OFFLINE, servername=roman-rs-1,60020,1511033854754, createTime=1511034136563, payload.length=0
2017-11-18 19:42:18,577 WARN  [AM.ZK.Worker-pool2-t19] master.AssignmentManager: Attempted to handle region transition for server but it is not online: 6608080647cc1ca0c6610b92de5ae0c4, region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., state=M_ZK_REGION_OFFLINE, servername=roman-rs-1,60020,1511033854754, createTime=1511034136563, payload.length=0
2017-11-18 19:42:18,578 WARN  [AM.ZK.Worker-pool2-t20] master.AssignmentManager: Attempted to handle region transition for server but it is not online: 6608080647cc1ca0c6610b92de5ae0c4, region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., state=M_ZK_REGION_OFFLINE, servername=roman-rs-1,60020,1511033854754, createTime=1511034136563, payload.length=0
2017-11-18 19:42:18,579 WARN  [AM.ZK.Worker-pool2-t21] master.AssignmentManager: Attempted to handle region transition for server but it is not online: 6608080647cc1ca0c6610b92de5ae0c4, region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., state=M_ZK_REGION_OFFLINE, servername=roman-rs-1,60020,1511033854754, createTime=1511034136563, payload.length=0
2017-11-18 19:42:18,579 WARN  [AM.ZK.Worker-pool2-t22] master.AssignmentManager: Attempted to handle region transition for server but it is not online: 6608080647cc1ca0c6610b92de5ae0c4, region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., state=M_ZK_REGION_OFFLINE, servername=roman-rs-1,60020,1511033854754, createTime=1511034136563, payload.length=0
2017-11-18 19:42:18,580 WARN  [AM.ZK.Worker-pool2-t23] master.AssignmentManager: Attempted to handle region transition for server but it is not online: 6608080647cc1ca0c6610b92de5ae0c4, region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., state=M_ZK_REGION_OFFLINE, servername=roman-rs-1,60020,1511033854754, createTime=1511034136563, payload.length=0
2017-11-18 19:42:18,586 WARN  [AM.ZK.Worker-pool2-t24] master.AssignmentManager: Attempted to handle region transition for server but it is not online: 6608080647cc1ca0c6610b92de5ae0c4, region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4., state=M_ZK_REGION_OFFLINE, servername=roman-rs-1,60020,1511033854754, createTime=1511034136563, payload.length=0
2017-11-18 19:42:21,819 INFO  [roman-master,60000,1511033853960.splitLogManagerTimeoutMonitor] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Froman-rs-3%2C60020%2C1511033854738-splitting%2Froman-rs-3%252C60020%252C1511033854738.1511033861810.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-18 19:42:26,820 INFO  [roman-master,60000,1511033853960.splitLogManagerTimeoutMonitor] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Froman-rs-3%2C60020%2C1511033854738-splitting%2Froman-rs-3%252C60020%252C1511033854738.1511033861810.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}
2017-11-18 19:42:31,821 INFO  [roman-master,60000,1511033853960.splitLogManagerTimeoutMonitor] master.SplitLogManager: total tasks = 1 unassigned = 1 tasks={/hbase/splitWAL/WALs%2Froman-rs-3%2C60020%2C1511033854738-splitting%2Froman-rs-3%252C60020%252C1511033854738.1511033861810.meta=last_update = -1 last_version = -1 cur_worker_name = null status = in_progress incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0}

[2]

ubuntu@roman-rs-1:/usr/local/hbase-0.98.6.1-hadoop1/logs$ cat hbase-ubuntu-regionserver-roman-rs-1.log
Sat Nov 18 19:37:33 UTC 2017 Starting regionserver on roman-rs-1
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514640
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2017-11-18 19:37:34,340 INFO  [main] util.VersionInfo: HBase 0.98.6.1-hadoop1
2017-11-18 19:37:34,340 INFO  [main] util.VersionInfo: Subversion git://apurtell-ltm1/Users/apurtell/src/hbase -r 96a1af660b33879f19a47e9113bf802ad59c7146
2017-11-18 19:37:34,340 INFO  [main] util.VersionInfo: Compiled by apurtell on Sun Sep 14 20:54:53 PDT 2014
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-ubuntu-regionserver-roman-rs-1.log
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LC_MEASUREMENT=de_DE.UTF-8
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LC_MONETARY=de_DE.UTF-8
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LC_TELEPHONE=de_DE.UTF-8
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LC_TIME=de_DE.UTF-8
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LANG=en_US.UTF-8
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:XDG_SESSION_ID=c16
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:MAIL=/var/mail/ubuntu
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LC_PAPER=de_DE.UTF-8
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LOGNAME=ubuntu
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:HBASE_REST_OPTS=
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:PWD=/usr/local/hbase-0.98.6.1-hadoop1
2017-11-18 19:37:34,434 INFO  [main] util.ServerCommandLine: env:LC_NAME=de_DE.UTF-8
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:SHELL=/bin/bash
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:LC_ADDRESS=de_DE.UTF-8
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_HEAPSIZE=4096
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=ubuntu
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/tmp/hbase-ubuntu-regionserver.znode
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:SSH_CLIENT=172.24.62.220 54448 22
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-ubuntu-regionserver-roman-rs-1
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/usr/local/hbase-0.98.6.1-hadoop1/bin/../logs
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:USER=ubuntu
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:CLASSPATH=/usr/local/hbase-0.98.6.1-hadoop1/bin/../conf:/usr/lib/jvm/java-1.8.0-openjdk-amd64//lib/tools.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/..:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/activation-1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/asm-3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-el-1.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-io-2.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-math-2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-net-1.4.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/guava-12.0.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hadoop-core-1.2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-client-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-common-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-common-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-examples-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-hadoop1-compat-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-it-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-it-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-protocol-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-server-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-server-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-shell-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-testing-util-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-thrift-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/junit-4.11.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/phoenix-4.13.0-HBase-0.98-server.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/zookeeper-3.4.6.jar:
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:LC_NUMERIC=de_DE.UTF-8
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:SSH_CONNECTION=172.24.62.220 54448 172.24.62.221 22
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_NICENESS=0
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_OPTS=-XX:+UseConcMarkSweepGC   -Dhbase.log.dir=/usr/local/hbase-0.98.6.1-hadoop1/bin/../logs -Dhbase.log.file=hbase-ubuntu-regionserver-roman-rs-1.log -Dhbase.home.dir=/usr/local/hbase-0.98.6.1-hadoop1/bin/.. -Dhbase.id.str=ubuntu -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_START_FILE=/tmp/hbase-ubuntu-regionserver.autorestart
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:XDG_RUNTIME_DIR=/run/user/1000
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS=
2017-11-18 19:37:34,435 INFO  [main] util.ServerCommandLine: env:HBASE_HOME=/usr/local/hbase-0.98.6.1-hadoop1/bin/..
2017-11-18 19:37:34,436 INFO  [main] util.ServerCommandLine: env:SHLVL=3
2017-11-18 19:37:34,436 INFO  [main] util.ServerCommandLine: env:HOME=/home/ubuntu
2017-11-18 19:37:34,436 INFO  [main] util.ServerCommandLine: env:LC_IDENTIFICATION=de_DE.UTF-8
2017-11-18 19:37:34,436 INFO  [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4
2017-11-18 19:37:34,436 INFO  [main] util.ServerCommandLine: vmName=OpenJDK 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.151-b12
2017-11-18 19:37:34,437 INFO  [main] util.ServerCommandLine: vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p, -Xmx4096m, -XX:+UseConcMarkSweepGC, -Dhbase.log.dir=/usr/local/hbase-0.98.6.1-hadoop1/bin/../logs, -Dhbase.log.file=hbase-ubuntu-regionserver-roman-rs-1.log, -Dhbase.home.dir=/usr/local/hbase-0.98.6.1-hadoop1/bin/.., -Dhbase.id.str=ubuntu, -Dhbase.root.logger=INFO,RFA, -Dhbase.security.logger=INFO,RFAS]
2017-11-18 19:37:34,513 DEBUG [main] regionserver.HRegionServer: regionserver/roman-rs-1/172.24.62.221:60020 HConnection server-to-server retries=350
2017-11-18 19:37:34,641 INFO  [main] ipc.SimpleRpcScheduler: Using default user call queue, count=3
2017-11-18 19:37:34,656 INFO  [main] ipc.RpcServer: regionserver/roman-rs-1/172.24.62.221:60020: started 10 reader(s).
2017-11-18 19:37:34,697 INFO  [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties
2017-11-18 19:37:34,736 INFO  [main] impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2017-11-18 19:37:34,740 INFO  [main] impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2017-11-18 19:37:34,740 INFO  [main] impl.MetricsSystemImpl: HBase metrics system started
2017-11-18 19:37:34,745 INFO  [main] impl.MetricsSourceAdapter: MBean for source jvm registered.
2017-11-18 19:37:34,747 INFO  [main] impl.MetricsSourceAdapter: MBean for source IPC,sub=IPC registered.
2017-11-18 19:37:34,783 INFO  [main] impl.MetricsSourceAdapter: MBean for source ugi registered.
2017-11-18 19:37:34,783 WARN  [main] impl.MetricsSystemImpl: Source name ugi already exists!
2017-11-18 19:37:34,786 INFO  [main] hfile.CacheConfig: Allocating LruBlockCache with maximum size 1.6g
2017-11-18 19:37:34,817 INFO  [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2017-11-18 19:37:34,844 INFO  [main] http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2017-11-18 19:37:34,849 INFO  [main] http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 60030
2017-11-18 19:37:34,850 INFO  [main] http.HttpServer: listener.getLocalPort() returned 60030 webServer.getConnectors()[0].getLocalPort() returned 60030
2017-11-18 19:37:34,850 INFO  [main] http.HttpServer: Jetty bound to port 60030
2017-11-18 19:37:34,850 INFO  [main] mortbay.log: jetty-6.1.26
2017-11-18 19:37:35,057 INFO  [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:60030
2017-11-18 19:37:35,073 INFO  [main] regionserver.ShutdownHook: Installed shutdown hook thread: Shutdownhook:regionserver60020
2017-11-18 19:37:35,079 INFO  [regionserver60020] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:60020 connecting to ZooKeeper ensemble=roman-master:2181
2017-11-18 19:37:35,086 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-11-18 19:37:35,086 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:host.name=roman-rs-1
2017-11-18 19:37:35,086 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_151
2017-11-18 19:37:35,086 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2017-11-18 19:37:35,086 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:java.class.path=/usr/local/hbase-0.98.6.1-hadoop1/bin/../conf:/usr/lib/jvm/java-1.8.0-openjdk-amd64//lib/tools.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/..:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/activation-1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/asm-3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-el-1.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-io-2.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-math-2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-net-1.4.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/guava-12.0.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hadoop-core-1.2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-client-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-common-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-common-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-examples-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-hadoop1-compat-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-it-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-it-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-protocol-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-server-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-server-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-shell-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-testing-util-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-thrift-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/junit-4.11.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/phoenix-4.13.0-HBase-0.98-server.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/zookeeper-3.4.6.jar:
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:os.name=Linux
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:os.version=4.4.0-96-generic
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:user.name=ubuntu
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:user.home=/home/ubuntu
2017-11-18 19:37:35,087 INFO  [regionserver60020] zookeeper.ZooKeeper: Client environment:user.dir=/usr/local/hbase-0.98.6.1-hadoop1
2017-11-18 19:37:35,088 INFO  [regionserver60020] zookeeper.ZooKeeper: Initiating client connection, connectString=roman-master:2181 sessionTimeout=90000 watcher=regionserver:60020, quorum=roman-master:2181, baseZNode=/hbase
2017-11-18 19:37:35,117 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Opening socket connection to server roman-master/172.24.62.220:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-18 19:37:35,191 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Socket connection established to roman-master/172.24.62.220:2181, initiating session
2017-11-18 19:37:35,202 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Session establishment complete on server roman-master/172.24.62.220:2181, sessionid = 0x15fd0a2d0c40001, negotiated timeout = 90000
2017-11-18 19:37:35,864 INFO  [regionserver60020] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x4fe5a664 connecting to ZooKeeper ensemble=roman-master:2181
2017-11-18 19:37:35,866 INFO  [regionserver60020] zookeeper.ZooKeeper: Initiating client connection, connectString=roman-master:2181 sessionTimeout=90000 watcher=hconnection-0x4fe5a664, quorum=roman-master:2181, baseZNode=/hbase
2017-11-18 19:37:35,868 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Opening socket connection to server roman-master/172.24.62.220:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-18 19:37:35,869 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Socket connection established to roman-master/172.24.62.220:2181, initiating session
2017-11-18 19:37:35,882 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Session establishment complete on server roman-master/172.24.62.220:2181, sessionid = 0x15fd0a2d0c40007, negotiated timeout = 90000
2017-11-18 19:37:36,073 DEBUG [regionserver60020] catalog.CatalogTracker: Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3b1687f6
2017-11-18 19:37:36,076 INFO  [regionserver60020] regionserver.HRegionServer: ClusterId : e788b872-5711-470f-970a-c9cfd49ba76d
2017-11-18 19:37:36,078 INFO  [regionserver60020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initializing
2017-11-18 19:37:36,100 INFO  [regionserver60020] zookeeper.RecoverableZooKeeper: Node /hbase/online-snapshot/acquired already exists and this is not a retry
2017-11-18 19:37:36,115 INFO  [regionserver60020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is initialized
2017-11-18 19:37:36,119 INFO  [regionserver60020] regionserver.MemStoreFlusher: globalMemStoreLimit=1.6g, globalMemStoreLimitLowMark=1.5g, maxHeap=4.0g
2017-11-18 19:37:36,121 INFO  [regionserver60020] regionserver.HRegionServer: CompactionChecker runs every 10sec
2017-11-18 19:37:36,127 INFO  [regionserver60020] regionserver.HRegionServer: reportForDuty to master=roman-master,60000,1511033853960 with port=60020, startcode=1511033854754
2017-11-18 19:37:36,360 DEBUG [regionserver60020] regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://roman-master/hbase
2017-11-18 19:37:36,361 DEBUG [regionserver60020] regionserver.HRegionServer: Config from master: fs.default.name=hdfs://roman-master
2017-11-18 19:37:36,361 DEBUG [regionserver60020] regionserver.HRegionServer: Config from master: hbase.master.info.port=60010
2017-11-18 19:37:36,398 INFO  [regionserver60020] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2017-11-18 19:37:36,407 DEBUG [regionserver60020] regionserver.HRegionServer: logdir=hdfs://roman-master/hbase/WALs/roman-rs-1,60020,1511033854754
2017-11-18 19:37:36,451 DEBUG [regionserver60020] regionserver.Replication: ReplicationStatisticsThread 300
2017-11-18 19:37:36,465 INFO  [regionserver60020] wal.FSHLog: WAL/HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true
2017-11-18 19:37:36,601 INFO  [regionserver60020] wal.FSHLog: New WAL /hbase/WALs/roman-rs-1,60020,1511033854754/roman-rs-1%2C60020%2C1511033854754.1511033856483
2017-11-18 19:37:36,608 INFO  [regionserver60020] impl.MetricsSourceAdapter: MBean for source RegionServer,sub=WAL registered.
2017-11-18 19:37:36,613 INFO  [regionserver60020] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds
2017-11-18 19:37:36,615 INFO  [regionserver60020] impl.MetricsSourceAdapter: MBean for source RegionServer,sub=Server registered.
2017-11-18 19:37:36,618 DEBUG [regionserver60020] executor.ExecutorService: Starting executor service name=RS_OPEN_REGION-roman-rs-1:60020, corePoolSize=3, maxPoolSize=3
2017-11-18 19:37:36,618 DEBUG [regionserver60020] executor.ExecutorService: Starting executor service name=RS_OPEN_META-roman-rs-1:60020, corePoolSize=1, maxPoolSize=1
2017-11-18 19:37:36,618 DEBUG [regionserver60020] executor.ExecutorService: Starting executor service name=RS_CLOSE_REGION-roman-rs-1:60020, corePoolSize=3, maxPoolSize=3
2017-11-18 19:37:36,618 DEBUG [regionserver60020] executor.ExecutorService: Starting executor service name=RS_CLOSE_META-roman-rs-1:60020, corePoolSize=1, maxPoolSize=1
2017-11-18 19:37:36,618 DEBUG [regionserver60020] executor.ExecutorService: Starting executor service name=RS_LOG_REPLAY_OPS-roman-rs-1:60020, corePoolSize=2, maxPoolSize=2
2017-11-18 19:37:36,625 INFO  [regionserver60020] regionserver.ReplicationSourceManager: Current list of replicators: [roman-rs-2,60020,1511033854813, roman-rs-1,60020,1511033854754, roman-rs-3,60020,1511033854738] other RSs: [roman-rs-2,60020,1511033854813, roman-rs-1,60020,1511033854754, roman-rs-3,60020,1511033854738]
2017-11-18 19:37:36,639 INFO  [regionserver60020] impl.MetricsSourceAdapter: MBean for source RegionServer,sub=Replication registered.
2017-11-18 19:37:36,640 INFO  [regionserver60020] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2ccd7fc7 connecting to ZooKeeper ensemble=roman-master:2181
2017-11-18 19:37:36,640 INFO  [regionserver60020] zookeeper.ZooKeeper: Initiating client connection, connectString=roman-master:2181 sessionTimeout=90000 watcher=hconnection-0x2ccd7fc7, quorum=roman-master:2181, baseZNode=/hbase
2017-11-18 19:37:36,642 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Opening socket connection to server roman-master/172.24.62.220:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-18 19:37:36,642 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Socket connection established to roman-master/172.24.62.220:2181, initiating session
2017-11-18 19:37:36,647 INFO  [regionserver60020-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Session establishment complete on server roman-master/172.24.62.220:2181, sessionid = 0x15fd0a2d0c4000a, negotiated timeout = 90000
2017-11-18 19:37:36,656 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting
2017-11-18 19:37:36,657 INFO  [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: starting
2017-11-18 19:37:36,657 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=0 queue=0
2017-11-18 19:37:36,657 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=1 queue=1
2017-11-18 19:37:36,657 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=2 queue=2
2017-11-18 19:37:36,657 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=3 queue=0
2017-11-18 19:37:36,658 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=4 queue=1
2017-11-18 19:37:36,658 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=5 queue=2
2017-11-18 19:37:36,658 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=6 queue=0
2017-11-18 19:37:36,658 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=7 queue=1
2017-11-18 19:37:36,659 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=8 queue=2
2017-11-18 19:37:36,659 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=9 queue=0
2017-11-18 19:37:36,659 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=10 queue=1
2017-11-18 19:37:36,659 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=11 queue=2
2017-11-18 19:37:36,660 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=12 queue=0
2017-11-18 19:37:36,660 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=13 queue=1
2017-11-18 19:37:36,660 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=14 queue=2
2017-11-18 19:37:36,660 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=15 queue=0
2017-11-18 19:37:36,665 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=16 queue=1
2017-11-18 19:37:36,665 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=17 queue=2
2017-11-18 19:37:36,665 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=18 queue=0
2017-11-18 19:37:36,665 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=19 queue=1
2017-11-18 19:37:36,665 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=20 queue=2
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=21 queue=0
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=22 queue=1
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=23 queue=2
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=24 queue=0
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=25 queue=1
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=26 queue=2
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=27 queue=0
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=28 queue=1
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: B.Default Start Handler index=29 queue=2
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=0 queue=0
2017-11-18 19:37:36,666 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=1 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=2 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=3 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=4 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=5 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=6 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=7 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=8 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Priority Start Handler index=9 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Replication Start Handler index=0 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Replication Start Handler index=1 queue=0
2017-11-18 19:37:36,667 DEBUG [regionserver60020] ipc.RpcExecutor: Replication Start Handler index=2 queue=0
2017-11-18 19:37:36,683 INFO  [regionserver60020] regionserver.HRegionServer: Serving as roman-rs-1,60020,1511033854754, RpcServer on roman-rs-1/172.24.62.221:60020, sessionid=0x15fd0a2d0c40001
2017-11-18 19:37:36,684 INFO  [regionserver60020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is starting
2017-11-18 19:37:36,684 DEBUG [regionserver60020] snapshot.RegionServerSnapshotManager: Start Snapshot Manager roman-rs-1,60020,1511033854754
2017-11-18 19:37:36,685 DEBUG [regionserver60020] procedure.ZKProcedureMemberRpcs: Starting procedure member 'roman-rs-1,60020,1511033854754'
2017-11-18 19:37:36,685 DEBUG [regionserver60020] procedure.ZKProcedureMemberRpcs: Checking for aborted procedures on node: '/hbase/online-snapshot/abort'
2017-11-18 19:37:36,685 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754] regionserver.SplitLogWorker: SplitLogWorker roman-rs-1,60020,1511033854754 starting
2017-11-18 19:37:36,686 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x61594243 connecting to ZooKeeper ensemble=roman-master:2181
2017-11-18 19:37:36,686 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754] zookeeper.ZooKeeper: Initiating client connection, connectString=roman-master:2181 sessionTimeout=90000 watcher=hconnection-0x61594243, quorum=roman-master:2181, baseZNode=/hbase
2017-11-18 19:37:36,687 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Opening socket connection to server roman-master/172.24.62.220:2181. Will not attempt to authenticate using SASL (unknown error)
2017-11-18 19:37:36,687 DEBUG [regionserver60020] procedure.ZKProcedureMemberRpcs: Looking for new procedures under znode:'/hbase/online-snapshot/acquired'
2017-11-18 19:37:36,687 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Socket connection established to roman-master/172.24.62.220:2181, initiating session
2017-11-18 19:37:36,688 INFO  [regionserver60020] procedure.RegionServerProcedureManagerHost: Procedure online-snapshot is started
2017-11-18 19:37:36,695 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754-SendThread(roman-master:2181)] zookeeper.ClientCnxn: Session establishment complete on server roman-master/172.24.62.220:2181, sessionid = 0x15fd0a2d0c4000d, negotiated timeout = 90000
2017-11-18 19:42:13,940 INFO  [PriorityRpcServer.handler=0,queue=0,port=60020] regionserver.HRegionServer: Open SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:14,056 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] zookeeper.ZKAssign: regionserver:60020-0x15fd0a2d0c40001, quorum=roman-master:2181, baseZNode=/hbase Transitioning 6608080647cc1ca0c6610b92de5ae0c4 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2017-11-18 19:42:14,072 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] zookeeper.ZKAssign: regionserver:60020-0x15fd0a2d0c40001, quorum=roman-master:2181, baseZNode=/hbase Transitioned node 6608080647cc1ca0c6610b92de5ae0c4 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING
2017-11-18 19:42:14,085 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Opening region: {ENCODED => 6608080647cc1ca0c6610b92de5ae0c4, NAME => 'SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.', STARTKEY => '', ENDKEY => ''}
2017-11-18 19:42:14,105 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Loading coprocessor class org.apache.phoenix.coprocessor.MetaDataEndpointImpl with path null and priority 805306366
2017-11-18 19:42:14,162 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Registered coprocessor service: region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. service=MetaDataService
2017-11-18 19:42:14,164 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.MetaDataEndpointImpl: Starting Tracing-Metrics Systems
2017-11-18 19:42:14,166 ERROR [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more
2017-11-18 19:42:14,167 FATAL [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegionServer: ABORTING region server roman-rs-1,60020,1511033854754: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
                at org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
                at org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
                at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
                at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
                at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
                at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
                at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
                at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
                at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.metrics2.MetricsInfo
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 24 more
2017-11-18 19:42:14,168 FATAL [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2017-11-18 19:42:14,172 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegionServer: STOPPED: The coprocessor org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
2017-11-18 19:42:14,172 INFO  [regionserver60020] ipc.RpcServer: Stopping server on 60020
2017-11-18 19:42:14,172 INFO  [RpcServer.listener,port=60020] ipc.RpcServer: RpcServer.listener,port=60020: stopping
2017-11-18 19:42:14,173 INFO  [regionserver60020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2017-11-18 19:42:14,173 INFO  [regionserver60020] regionserver.HRegionServer: Stopping infoServer
2017-11-18 19:42:14,173 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754] regionserver.SplitLogWorker: SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException
2017-11-18 19:42:14,173 INFO  [SplitLogWorker-roman-rs-1,60020,1511033854754] regionserver.SplitLogWorker: SplitLogWorker roman-rs-1,60020,1511033854754 exiting
2017-11-18 19:42:14,180 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Loading coprocessor class org.apache.phoenix.coprocessor.MetaDataRegionObserver with path null and priority 805306367
2017-11-18 19:42:14,186 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-11-18 19:42:14,186 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-11-18 19:42:14,200 INFO  [regionserver60020] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2017-11-18 19:42:14,208 INFO  [regionserver60020.logRoller] regionserver.LogRoller: LogRoller exiting.
2017-11-18 19:42:14,208 INFO  [regionserver60020.compactionChecker] regionserver.HRegionServer$CompactionChecker: regionserver60020.compactionChecker exiting
2017-11-18 19:42:14,208 INFO  [regionserver60020.nonceCleaner] regionserver.ServerNonceManager$1: regionserver60020.nonceCleaner exiting
2017-11-18 19:42:14,208 INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2017-11-18 19:42:14,208 INFO  [regionserver60020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2017-11-18 19:42:14,209 INFO  [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2017-11-18 19:42:14,209 INFO  [regionserver60020] regionserver.HRegionServer: aborting server roman-rs-1,60020,1511033854754
2017-11-18 19:42:14,209 DEBUG [regionserver60020] catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3b1687f6
2017-11-18 19:42:14,209 INFO  [regionserver60020] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15fd0a2d0c40007
2017-11-18 19:42:14,215 INFO  [regionserver60020] zookeeper.ZooKeeper: Session: 0x15fd0a2d0c40007 closed
2017-11-18 19:42:14,215 INFO  [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-11-18 19:42:14,216 INFO  [regionserver60020] regionserver.HRegionServer: stopping server roman-rs-1,60020,1511033854754; all regions closed.
2017-11-18 19:42:14,216 DEBUG [regionserver60020-WAL.AsyncNotifier] wal.FSHLog: regionserver60020-WAL.AsyncNotifier interrupted while waiting for  notification from AsyncSyncer thread
2017-11-18 19:42:14,216 INFO  [regionserver60020-WAL.AsyncNotifier] wal.FSHLog: regionserver60020-WAL.AsyncNotifier exiting
2017-11-18 19:42:14,216 DEBUG [regionserver60020-WAL.AsyncSyncer0] wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 interrupted while waiting for notification from AsyncWriter thread
2017-11-18 19:42:14,216 INFO  [regionserver60020-WAL.AsyncSyncer0] wal.FSHLog: regionserver60020-WAL.AsyncSyncer0 exiting
2017-11-18 19:42:14,216 DEBUG [regionserver60020-WAL.AsyncSyncer1] wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 interrupted while waiting for notification from AsyncWriter thread
2017-11-18 19:42:14,217 INFO  [regionserver60020-WAL.AsyncSyncer1] wal.FSHLog: regionserver60020-WAL.AsyncSyncer1 exiting
2017-11-18 19:42:14,217 DEBUG [regionserver60020-WAL.AsyncSyncer2] wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 interrupted while waiting for notification from AsyncWriter thread
2017-11-18 19:42:14,217 INFO  [regionserver60020-WAL.AsyncSyncer2] wal.FSHLog: regionserver60020-WAL.AsyncSyncer2 exiting
2017-11-18 19:42:14,217 DEBUG [regionserver60020-WAL.AsyncSyncer3] wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 interrupted while waiting for notification from AsyncWriter thread
2017-11-18 19:42:14,217 INFO  [regionserver60020-WAL.AsyncSyncer3] wal.FSHLog: regionserver60020-WAL.AsyncSyncer3 exiting
2017-11-18 19:42:14,217 DEBUG [regionserver60020-WAL.AsyncSyncer4] wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 interrupted while waiting for notification from AsyncWriter thread
2017-11-18 19:42:14,217 INFO  [regionserver60020-WAL.AsyncSyncer4] wal.FSHLog: regionserver60020-WAL.AsyncSyncer4 exiting
2017-11-18 19:42:14,217 DEBUG [regionserver60020-WAL.AsyncWriter] wal.FSHLog: regionserver60020-WAL.AsyncWriter interrupted while waiting for newer writes added to local buffer
2017-11-18 19:42:14,217 INFO  [regionserver60020-WAL.AsyncWriter] wal.FSHLog: regionserver60020-WAL.AsyncWriter exiting
2017-11-18 19:42:14,217 DEBUG [regionserver60020] wal.FSHLog: Closing WAL writer in hdfs://roman-master/hbase/WALs/roman-rs-1,60020,1511033854754
2017-11-18 19:42:14,344 INFO  [regionserver60020] regionserver.Leases: regionserver60020 closing leases
2017-11-18 19:42:14,344 INFO  [regionserver60020] regionserver.Leases: regionserver60020 closed leases
2017-11-18 19:42:16,195 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.apache.phoenix.coprocessor.MetaDataRegionObserver from HTD of SYSTEM.CATALOG successfully.
2017-11-18 19:42:16,196 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Loading coprocessor class org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver with path null and priority 805306366
2017-11-18 19:42:16,259 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver from HTD of SYSTEM.CATALOG successfully.
2017-11-18 19:42:16,269 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Loading coprocessor class org.apache.phoenix.coprocessor.ServerCachingEndpointImpl with path null and priority 805306366
2017-11-18 19:42:16,273 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Registered coprocessor service: region=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. service=ServerCachingService
2017-11-18 19:42:16,273 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.apache.phoenix.coprocessor.ServerCachingEndpointImpl from HTD of SYSTEM.CATALOG successfully.
2017-11-18 19:42:16,274 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Loading coprocessor class org.apache.phoenix.coprocessor.ScanRegionObserver with path null and priority 805306366
2017-11-18 19:42:16,275 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.apache.phoenix.coprocessor.ScanRegionObserver from HTD of SYSTEM.CATALOG successfully.
2017-11-18 19:42:16,276 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Loading coprocessor class org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver with path null and priority 805306366
2017-11-18 19:42:16,300 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver from HTD of SYSTEM.CATALOG successfully.
2017-11-18 19:42:16,304 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] impl.MetricsSourceAdapter: MBean for source RegionServer,sub=Regions registered.
2017-11-18 19:42:16,306 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.MetricsRegionSourceImpl: Creating new MetricsRegionSourceImpl for table SYSTEM.CATALOG 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:16,306 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Instantiated SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:16,355 INFO  [StoreOpener-6608080647cc1ca0c6610b92de5ae0c4-1] compactions.CompactionConfiguration: size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 604800000, major jitter 0.500000
2017-11-18 19:42:16,361 DEBUG [StoreOpener-6608080647cc1ca0c6610b92de5ae0c4-1] regionserver.HRegionFileSystem: No StoreFiles for: hdfs://roman-master/hbase/data/default/SYSTEM.CATALOG/6608080647cc1ca0c6610b92de5ae0c4/0
2017-11-18 19:42:16,367 INFO  [StoreOpener-6608080647cc1ca0c6610b92de5ae0c4-1] util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32
2017-11-18 19:42:16,374 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Found 0 recovered edits file(s) under hdfs://roman-master/hbase/data/default/SYSTEM.CATALOG/6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:16,378 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Onlined 6608080647cc1ca0c6610b92de5ae0c4; next sequenceid=1
2017-11-18 19:42:16,440 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] zookeeper.ZKAssign: regionserver:60020-0x15fd0a2d0c40001, quorum=roman-master:2181, baseZNode=/hbase Attempting to retransition opening state of node 6608080647cc1ca0c6610b92de5ae0c4
2017-11-18 19:42:16,444 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Closing SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.: disabling compactions & flushes
2017-11-18 19:42:16,444 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Updates disabled for region SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:16,446 INFO  [StoreCloserThread-SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.-1] regionserver.HStore: Closed 0
2017-11-18 19:42:16,446 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Stop coprocessor org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver
2017-11-18 19:42:16,446 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Stop coprocessor org.apache.phoenix.coprocessor.ServerCachingEndpointImpl
2017-11-18 19:42:16,446 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Stop coprocessor org.apache.phoenix.coprocessor.ScanRegionObserver
2017-11-18 19:42:16,446 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Stop coprocessor org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
2017-11-18 19:42:16,446 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] coprocessor.CoprocessorHost: Stop coprocessor org.apache.phoenix.coprocessor.MetaDataRegionObserver
2017-11-18 19:42:16,447 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] regionserver.HRegion: Closed SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
2017-11-18 19:42:16,447 INFO  [RS_OPEN_REGION-roman-rs-1:60020-0] handler.OpenRegionHandler: Opening of region {ENCODED => 6608080647cc1ca0c6610b92de5ae0c4, NAME => 'SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.', STARTKEY => '', ENDKEY => ''} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 7
2017-11-18 19:42:16,447 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] zookeeper.ZKAssign: regionserver:60020-0x15fd0a2d0c40001, quorum=roman-master:2181, baseZNode=/hbase Transitioning 6608080647cc1ca0c6610b92de5ae0c4 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
2017-11-18 19:42:16,455 DEBUG [RS_OPEN_REGION-roman-rs-1:60020-0] zookeeper.ZKAssign: regionserver:60020-0x15fd0a2d0c40001, quorum=roman-master:2181, baseZNode=/hbase Transitioned node 6608080647cc1ca0c6610b92de5ae0c4 from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN
2017-11-18 19:42:16,622 INFO  [regionserver60020.periodicFlusher] regionserver.HRegionServer$PeriodicMemstoreFlusher: regionserver60020.periodicFlusher exiting
2017-11-18 19:42:16,622 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2017-11-18 19:42:16,622 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2017-11-18 19:42:16,622 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2017-11-18 19:42:16,622 INFO  [regionserver60020] regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2017-11-18 19:42:16,623 INFO  [regionserver60020.leaseChecker] regionserver.Leases: regionserver60020.leaseChecker closing leases
2017-11-18 19:42:16,623 INFO  [regionserver60020.leaseChecker] regionserver.Leases: regionserver60020.leaseChecker closed leases
2017-11-18 19:42:16,638 INFO  [regionserver60020] client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x15fd0a2d0c4000a
2017-11-18 19:42:16,650 INFO  [regionserver60020] zookeeper.ZooKeeper: Session: 0x15fd0a2d0c4000a closed
2017-11-18 19:42:16,650 INFO  [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-11-18 19:42:16,661 INFO  [regionserver60020] zookeeper.ZooKeeper: Session: 0x15fd0a2d0c40001 closed
2017-11-18 19:42:16,661 INFO  [regionserver60020] regionserver.HRegionServer: stopping server roman-rs-1,60020,1511033854754; zookeeper connection closed.
2017-11-18 19:42:16,661 INFO  [regionserver60020] regionserver.HRegionServer: regionserver60020 exiting
2017-11-18 19:42:16,661 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
                at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66)
                at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
                at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
                at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2489)
2017-11-18 19:42:16,661 INFO  [regionserver60020-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-11-18 19:42:16,663 INFO  [Shutdownhook:regionserver60020] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-8,5,main]
2017-11-18 19:42:16,663 INFO  [Shutdownhook:regionserver60020] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2017-11-18 19:42:16,664 INFO  [Shutdownhook:regionserver60020] regionserver.ShutdownHook: Shutdown hook finished.

[3]

ubuntu@roman-master:/usr/local/hbase-0.98.6.1-hadoop1/logs$ cat hbase-ubuntu-zookeeper-roman-master.log
Sat Nov 18 19:37:31 UTC 2017 Starting zookeeper on roman-master
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 514627
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2017-11-18 19:37:32,054 ERROR [main] quorum.QuorumPeerConfig: Invalid configuration, only one server specified (ignoring)
2017-11-18 19:37:32,065 INFO  [main] server.ZooKeeperServerMain: Starting server
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:host.name=roman-master
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:java.version=1.8.0_151
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:java.class.path=/usr/local/hbase-0.98.6.1-hadoop1/bin/../conf:/usr/lib/jvm/java-1.8.0-openjdk-amd64//lib/tools.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/..:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/activation-1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/asm-3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-beanutils-1.7.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-cli-1.2.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-codec-1.7.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-collections-3.2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-configuration-1.6.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-digester-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-el-1.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-httpclient-3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-io-2.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-lang-2.6.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-math-2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/commons-net-1.4.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/guava-12.0.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hadoop-core-1.2.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hamcrest-core-1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-client-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-common-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-common-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-examples-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-hadoop1-compat-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-hadoop-compat-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-it-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-it-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-prefix-tree-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-protocol-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-server-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-server-0.98.6.1-hadoop1-tests.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-shell-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-testing-util-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/hbase-thrift-0.98.6.1-hadoop1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/high-scale-lib-1.1.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/htrace-core-2.04.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/httpclient-4.1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/httpcore-4.1.3.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-jaxrs-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jackson-xc-1.8.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jamon-runtime-2.3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jasper-compiler-5.5.23.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jasper-runtime-5.5.23.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jaxb-api-2.2.2.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-core-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-json-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jersey-server-1.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jettison-1.3.1.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-sslengine-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jetty-util-6.1.26.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jruby-complete-1.6.8.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/jsr305-1.3.9.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/junit-4.11.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/libthrift-0.9.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/log4j-1.2.17.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/metrics-core-2.2.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/netty-3.6.6.Final.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/phoenix-4.13.0-HBase-0.98-server.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/protobuf-java-2.5.0.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/slf4j-api-1.6.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/slf4j-log4j12-1.6.4.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/xmlenc-0.52.jar:/usr/local/hbase-0.98.6.1-hadoop1/bin/../lib/zookeeper-3.4.6.jar:
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:java.compiler=<NA>
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:os.name=Linux
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:os.arch=amd64
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:os.version=4.4.0-96-generic
2017-11-18 19:37:32,071 INFO  [main] server.ZooKeeperServer: Server environment:user.name=ubuntu
2017-11-18 19:37:32,072 INFO  [main] server.ZooKeeperServer: Server environment:user.home=/home/ubuntu
2017-11-18 19:37:32,072 INFO  [main] server.ZooKeeperServer: Server environment:user.dir=/usr/local/hbase-0.98.6.1-hadoop1
2017-11-18 19:37:32,077 INFO  [main] server.ZooKeeperServer: tickTime set to 3000
2017-11-18 19:37:32,077 INFO  [main] server.ZooKeeperServer: minSessionTimeout set to -1
2017-11-18 19:37:32,077 INFO  [main] server.ZooKeeperServer: maxSessionTimeout set to 90000
2017-11-18 19:37:32,086 INFO  [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:2181
2017-11-18 19:37:34,223 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.220:59490
2017-11-18 19:37:34,277 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.220:59490
2017-11-18 19:37:34,279 INFO  [SyncThread:0] persistence.FileTxnLog: Creating new log file: log.1
2017-11-18 19:37:34,326 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40000 with negotiated timeout 90000 for client /172.24.62.220:59490
2017-11-18 19:37:35,191 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.221:49142
2017-11-18 19:37:35,194 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.221:49142
2017-11-18 19:37:35,200 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40001 with negotiated timeout 90000 for client /172.24.62.221:49142
2017-11-18 19:37:35,240 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.222:37500
2017-11-18 19:37:35,242 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.222:37500
2017-11-18 19:37:35,246 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40002 with negotiated timeout 90000 for client /172.24.62.222:37500
2017-11-18 19:37:35,269 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.223:34540
2017-11-18 19:37:35,271 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.223:34540
2017-11-18 19:37:35,279 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40003 with negotiated timeout 90000 for client /172.24.62.223:34540
2017-11-18 19:37:35,750 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.220:59504
2017-11-18 19:37:35,751 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.220:59504
2017-11-18 19:37:35,755 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40004 with negotiated timeout 90000 for client /172.24.62.220:59504
2017-11-18 19:37:35,867 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.223:34546
2017-11-18 19:37:35,868 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40000 type:create cxid:0x27 zxid:0x12 txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot Error:KeeperErrorCode = NoNode for /hbase/online-snapshot
2017-11-18 19:37:35,868 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.222:37506
2017-11-18 19:37:35,868 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.221:49148
2017-11-18 19:37:35,869 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.222:37506
2017-11-18 19:37:35,869 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.223:34546
2017-11-18 19:37:35,869 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.221:49148
2017-11-18 19:37:35,880 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40005 with negotiated timeout 90000 for client /172.24.62.222:37506
2017-11-18 19:37:35,881 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40006 with negotiated timeout 90000 for client /172.24.62.223:34546
2017-11-18 19:37:35,881 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40007 with negotiated timeout 90000 for client /172.24.62.221:49148
2017-11-18 19:37:35,925 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.220:59506
2017-11-18 19:37:35,925 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.220:59506
2017-11-18 19:37:35,934 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40008 with negotiated timeout 90000 for client /172.24.62.220:59506
2017-11-18 19:37:35,934 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40008 type:create cxid:0x1 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/hbase/replication Error:KeeperErrorCode = NoNode for /hbase/replication
2017-11-18 19:37:36,082 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40001 type:create cxid:0x8 zxid:0x1e txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase/online-snapshot/acquired
2017-11-18 19:37:36,088 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40002 type:create cxid:0x8 zxid:0x1f txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase/online-snapshot/acquired
2017-11-18 19:37:36,127 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40003 type:create cxid:0x8 zxid:0x20 txntype:-1 reqpath:n/a Error Path:/hbase/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase/online-snapshot/acquired
2017-11-18 19:37:36,629 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.222:37516
2017-11-18 19:37:36,630 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.222:37516
2017-11-18 19:37:36,635 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40009 with negotiated timeout 90000 for client /172.24.62.222:37516
2017-11-18 19:37:36,642 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.221:49158
2017-11-18 19:37:36,643 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.221:49158
2017-11-18 19:37:36,644 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.223:34556
2017-11-18 19:37:36,644 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.223:34556
2017-11-18 19:37:36,646 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c4000a with negotiated timeout 90000 for client /172.24.62.221:49158
2017-11-18 19:37:36,652 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c4000b with negotiated timeout 90000 for client /172.24.62.223:34556
2017-11-18 19:37:36,665 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.222:37518
2017-11-18 19:37:36,681 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.222:37518
2017-11-18 19:37:36,686 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c4000c with negotiated timeout 90000 for client /172.24.62.222:37518
2017-11-18 19:37:36,687 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.221:49160
2017-11-18 19:37:36,687 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.221:49160
2017-11-18 19:37:36,692 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c4000d with negotiated timeout 90000 for client /172.24.62.221:49160
2017-11-18 19:37:36,698 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.223:34558
2017-11-18 19:37:36,705 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.223:34558
2017-11-18 19:37:36,713 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c4000e with negotiated timeout 90000 for client /172.24.62.223:34558
2017-11-18 19:37:41,517 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40000 type:delete cxid:0x50 zxid:0x2e txntype:-1 reqpath:n/a Error Path:/hbase/meta-region-server Error:KeeperErrorCode = NoNode for /hbase/meta-region-server
2017-11-18 19:37:42,303 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40000 type:create cxid:0x7a zxid:0x36 txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode = NoNode for /hbase/table-lock/hbase:namespace
2017-11-18 19:37:42,983 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40000 type:create cxid:0x9c zxid:0x46 txntype:-1 reqpath:n/a Error Path:/hbase/namespace/default Error:KeeperErrorCode = NodeExists for /hbase/namespace/default
2017-11-18 19:37:42,995 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40000 type:create cxid:0x9f zxid:0x48 txntype:-1 reqpath:n/a Error Path:/hbase/namespace/hbase Error:KeeperErrorCode = NodeExists for /hbase/namespace/hbase
2017-11-18 19:42:07,314 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.220:59528
2017-11-18 19:42:07,318 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.220:59528
2017-11-18 19:42:07,327 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c4000f with negotiated timeout 90000 for client /172.24.62.220:59528
2017-11-18 19:42:07,537 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.220:59530
2017-11-18 19:42:07,538 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.220:59530
2017-11-18 19:42:07,547 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40010 with negotiated timeout 90000 for client /172.24.62.220:59530
2017-11-18 19:42:07,554 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.220:59532
2017-11-18 19:42:07,554 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.220:59532
2017-11-18 19:42:07,558 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40011 with negotiated timeout 90000 for client /172.24.62.220:59532
2017-11-18 19:42:07,727 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40010
2017-11-18 19:42:07,734 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.220:59530 which had sessionid 0x15fd0a2d0c40010
2017-11-18 19:42:07,834 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40011
2017-11-18 19:42:07,838 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.220:59532 which had sessionid 0x15fd0a2d0c40011
2017-11-18 19:42:08,531 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x15fd0a2d0c40000 type:create cxid:0xa8 zxid:0x4f txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/SYSTEM.CATALOG Error:KeeperErrorCode = NoNode for /hbase/table-lock/SYSTEM.CATALOG
2017-11-18 19:42:09,158 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40006
2017-11-18 19:42:09,164 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.223:34546 which had sessionid 0x15fd0a2d0c40006
2017-11-18 19:42:11,708 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40005
2017-11-18 19:42:11,713 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.222:37506 which had sessionid 0x15fd0a2d0c40005
2017-11-18 19:42:12,214 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.223:34592
2017-11-18 19:42:12,215 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.223:34592
2017-11-18 19:42:12,225 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40012 with negotiated timeout 90000 for client /172.24.62.223:34592
2017-11-18 19:42:12,230 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.223:34594
2017-11-18 19:42:12,230 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.223:34594
2017-11-18 19:42:12,236 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40013 with negotiated timeout 90000 for client /172.24.62.223:34594
2017-11-18 19:42:14,209 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40007
2017-11-18 19:42:14,215 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.221:49148 which had sessionid 0x15fd0a2d0c40007
2017-11-18 19:42:14,813 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.222:37550
2017-11-18 19:42:14,814 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.222:37550
2017-11-18 19:42:14,825 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40014 with negotiated timeout 90000 for client /172.24.62.222:37550
2017-11-18 19:42:14,843 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxnFactory: Accepted socket connection from /172.24.62.222:37552
2017-11-18 19:42:14,844 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer: Client attempting to establish new session at /172.24.62.222:37552
2017-11-18 19:42:14,850 INFO  [SyncThread:0] server.ZooKeeperServer: Established session 0x15fd0a2d0c40015 with negotiated timeout 90000 for client /172.24.62.222:37552
2017-11-18 19:42:16,621 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40009
2017-11-18 19:42:16,627 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.222:37516 which had sessionid 0x15fd0a2d0c40009
2017-11-18 19:42:16,633 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40002
2017-11-18 19:42:16,638 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c4000a
2017-11-18 19:42:16,644 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.222:37500 which had sessionid 0x15fd0a2d0c40002
2017-11-18 19:42:16,649 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.221:49158 which had sessionid 0x15fd0a2d0c4000a
2017-11-18 19:42:16,650 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c4000b
2017-11-18 19:42:16,655 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.223:34556 which had sessionid 0x15fd0a2d0c4000b
2017-11-18 19:42:16,656 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40001
2017-11-18 19:42:16,661 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.221:49142 which had sessionid 0x15fd0a2d0c40001
2017-11-18 19:42:16,667 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40003
2017-11-18 19:42:16,676 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.223:34540 which had sessionid 0x15fd0a2d0c40003
2017-11-18 19:42:16,997 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15fd0a2d0c4000c, likely client has closed socket
                at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
                at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:17,000 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.222:37518 which had sessionid 0x15fd0a2d0c4000c
2017-11-18 19:42:17,000 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15fd0a2d0c40014, likely client has closed socket
                at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
                at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:17,001 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.222:37550 which had sessionid 0x15fd0a2d0c40014
2017-11-18 19:42:17,001 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15fd0a2d0c4000d, likely client has closed socket
                at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
                at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:17,001 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.221:49160 which had sessionid 0x15fd0a2d0c4000d
2017-11-18 19:42:17,001 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15fd0a2d0c40015, likely client has closed socket
                at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
                at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:17,002 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.222:37552 which had sessionid 0x15fd0a2d0c40015
2017-11-18 19:42:17,032 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15fd0a2d0c4000e, likely client has closed socket
                at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
                at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:17,033 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.223:34558 which had sessionid 0x15fd0a2d0c4000e
2017-11-18 19:42:17,033 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15fd0a2d0c40012, likely client has closed socket
                at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
                at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:17,035 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.223:34592 which had sessionid 0x15fd0a2d0c40012
2017-11-18 19:42:17,035 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x15fd0a2d0c40013, likely client has closed socket
                at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
                at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
                at java.lang.Thread.run(Thread.java:748)
2017-11-18 19:42:17,035 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.223:34594 which had sessionid 0x15fd0a2d0c40013
2017-11-18 19:43:39,000 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15fd0a2d0c4000c, timeout of 90000ms exceeded
2017-11-18 19:43:39,001 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15fd0a2d0c4000e, timeout of 90000ms exceeded
2017-11-18 19:43:39,001 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15fd0a2d0c4000d, timeout of 90000ms exceeded
2017-11-18 19:43:39,001 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c4000c
2017-11-18 19:43:39,001 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c4000e
2017-11-18 19:43:39,001 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c4000d
2017-11-18 19:43:45,000 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15fd0a2d0c40015, timeout of 90000ms exceeded
2017-11-18 19:43:45,000 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15fd0a2d0c40013, timeout of 90000ms exceeded
2017-11-18 19:43:45,000 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40015
2017-11-18 19:43:45,001 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40013
2017-11-18 19:43:48,001 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15fd0a2d0c40014, timeout of 90000ms exceeded
2017-11-18 19:43:48,001 INFO  [SessionTracker] server.ZooKeeperServer: Expiring session 0x15fd0a2d0c40012, timeout of 90000ms exceeded
2017-11-18 19:43:48,001 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40014
2017-11-18 19:43:48,001 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c40012
2017-11-18 19:50:39,808 INFO  [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x15fd0a2d0c4000f
2017-11-18 19:50:39,816 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /172.24.62.220:59528 which had sessionid 0x15fd0a2d0c4000f

[4]

ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin$ ./sqlline.py roman-master:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:roman-master:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:roman-master:2181
17/11/18 19:42:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error: Failed after attempts=35, exceptions:
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.io.IOException: Call to roman-rs-3/172.24.62.223:60020 failed on local exception: java.io.EOFException
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:10 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:11 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:13 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:27 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:47 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:57 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:57 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:18 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:38 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:18 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:38 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:50:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:50:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused (state=08000,code=101)
org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=35, exceptions:
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.io.IOException: Call to roman-rs-3/172.24.62.223:60020 failed on local exception: java.io.EOFException
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:10 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:11 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:13 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:27 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:47 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:57 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:57 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:18 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:38 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:18 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:38 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:50:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:50:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused

                at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:120)
                at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1125)
                at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1407)
                at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2640)
                at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1045)
                at org.apache.phoenix.compile.CreateTableCompiler$1.execute(CreateTableCompiler.java:192)
                at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:393)
                at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:376)
                at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
                at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:374)
                at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:363)
                at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1669)
                at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2442)
                at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2381)
                at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
                at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2381)
                at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
                at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
                at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
                at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)
                at sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)
                at sqlline.Commands.connect(Commands.java:1064)
                at sqlline.Commands.connect(Commands.java:996)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)
                at sqlline.SqlLine.dispatch(SqlLine.java:809)
                at sqlline.SqlLine.initArgs(SqlLine.java:588)
                at sqlline.SqlLine.begin(SqlLine.java:661)
                at sqlline.SqlLine.start(SqlLine.java:398)
                at sqlline.SqlLine.main(SqlLine.java:291)
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=35, exceptions:
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.io.IOException: Call to roman-rs-3/172.24.62.223:60020 failed on local exception: java.io.EOFException
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:09 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:10 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, org.apache.hadoop.hbase.ipc.RpcClient$FailedServerException: This server is in the failed servers list: roman-rs-3/172.24.62.223:60020
Sat Nov 18 19:42:11 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:13 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:27 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:47 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:42:57 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:43:57 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:17 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:37 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:44:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:18 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:38 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:45:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:18 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:38 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:46:58 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:47:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:48:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:49:59 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:50:19 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused
Sat Nov 18 19:50:39 UTC 2017, org.apache.hadoop.hbase.client.RpcRetryingCaller@5bd82fed, java.net.ConnectException: Connection refused

                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:141)
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:101)
                at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
                at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:190)
                at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:185)
                at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:111)
                at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:789)
                at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:178)
                at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:82)
                at org.apache.hadoop.hbase.client.HBaseAdmin.createTable(HBaseAdmin.java:590)
                at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1064)
                ... 31 more
Caused by: java.net.ConnectException: Connection refused
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
                at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
                at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:583)
                at org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:873)
                at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1578)
                at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1477)
                at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1694)
                at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1764)
                at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:32938)
                at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:324)
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:168)
                at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:58)
                at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:125)
                ... 41 more
sqlline version 1.2.0
0: jdbc:phoenix:roman-master:2181>


Mime
View raw message