phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: sqlline.py kills all regionservers
Date Sat, 18 Nov 2017 21:39:09 GMT
That’s quite an old version of 0.98 and we no longer support Hadoop 1.
Would it be possible for you to upgrade your cluster and use Hadoop 2
instead?

On Sat, Nov 18, 2017 at 1:16 PM Eisenhut, Roman <roman.eisenhut@tum.de>
wrote:

> Dear Phoenix community,
>
>
>
> I’m trying to implement apache-phoenix-4.13.0-HBase-0.98-bin on
> Hadoop-1.2.1 and HBase-0.98.6.1 on a 4 node cluster, running the following
> services:
>
>    - roman-master:
>       - NameNode
>       - SecondaryNamenode
>       - HMaster
>       - HQuorumPeer
>    - roman-rs-1:
>       - DataNode
>       - HRegionServer
>    - roman-rs-2:
>       - DataNode
>       - HRegionServer
>    - roman-rs-3:
>       - DataNode
>       - HRegionServer
>
>
>
> I have copied the corresponding server.jar into the lib directory of my
> HBase installation on every server.
>
>
>
> *ubuntu@roman-master:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep
> phoenix*
>
> *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15
> phoenix-4.13.0-HBase-0.98-server.jar*
>
>
>
> *ubuntu@roman-rs-1:~$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib | grep
> phoenix*
>
> *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:15
> phoenix-4.13.0-HBase-0.98-server.jar*
>
>
>
> *ubuntu@roman-rs-2:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib
> | grep phoenix*
>
> *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16
> phoenix-4.13.0-HBase-0.98-server.jar*
>
>
>
> *ubuntu@roman-rs-3:/usr/local$ ls -l /usr/local/hbase-0.98.6.1-hadoop1/lib
> | grep phoenix*
>
> *-rwxrwxrwx 1 ubuntu ubuntu 25706067 Nov 18 18:16
> phoenix-4.13.0-HBase-0.98-server.jar*
>
>
>
> I can:
>
>    - Start Hadoop
>    - Start HBase
>       - Create a table in HBase
>       - Put data
>       - Get data
>    - Browse
>       - http://roman-master:50070/dfshealth.jsp
>       - http://roman-master:60010/master-status
>
>
>
> jps shows all the processes are up and running on each machine and the
> http interfaces show everything is running properly. It seems like the
> system is in perfect health. However, as soon as I try to connect to
> phoenix via  */usr/local/
> apache-phoenix-4.13.0-HBase-0.98-bin/bin/sqlline.py roman-master:2181* on
> roman-master, where roman-maste is my zookeeper quorum, it kills every
> regionservers on the entire cluster. After a long time, I don’t know
> exactly how long, sqlline will return an output (can be found under [3]).
>
>
>
> *Connection call*
>
> ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin$
> ./sqlline.py roman-master:2181
>
> Setting property: [incremental, false]
>
> Setting property: [isolation, TRANSACTION_READ_COMMITTED]
>
> issuing: !connect jdbc:phoenix:roman-master:2181 none none
> org.apache.phoenix.jdbc.PhoenixDriver
>
> Connecting to jdbc:phoenix:roman-master:2181
>
> 17/11/18 19:24:33 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
>
>
> Underneath you’ll find the following files
>
>    - Conf files for
>       - Phoenix
>          - env.sh
>       - Hadoop
>          - hadoop-env.sh
>          - core-site.xml
>          - hdfs-site.xml
>       - HBase
>          - hbase-site.xml
>          - hbase-env.sh
>       - [1] master log on roman-master
>    - [2] regionserver log roman-rs-1
>    - [3] zookeeper log on roman-master
>    - [4] output of sqlline.py
>
>
>
> I’m pretty sure there’s a reasonable explanation for this behavior.
> Nevertheless, I’ve been trying to get phoenix up and running for days now
> and I can’t figure out what I’m doing wrong. I hope someone can point me
> into the right direction.
>
>
>
> Best regards,
>
> Roman
>
>
>
>
>
>
>
>
>
>
>
> *Phoenix conf:*
>
>
> ubuntu@roman-master:/usr/local/apache-phoenix-4.13.0-HBase-0.98-bin/bin/config$
> cat env.sh
>
> JAVA_HOME=/usr/lib/jvm/java-8-oracle/
>
> HBASE_PATH=/usr/local/hbase-0.98.6.1-hadoop1/
>
>
>
> *Hadoop conf:*
>
> *Hadoop-env.sh*
>
>
>
> export JAVA_HOME=/usr/lib/jvm/java-8-oracle
>
> # The maximum amount of heap to use, in MB. Default is 1000.
>
> # export HADOOP_HEAPSIZE=2000
>
> # Command specific options appended to HADOOP_OPTS when specified
>
> export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_NAMENODE_OPTS"
>
> export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_SECONDARYNAMENODE_OPTS"
>
> export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_DATANODE_OPTS"
>
> export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_BALANCER_OPTS"
>
> export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote
> $HADOOP_JOBTRACKER_OPTS"
>
> _EOF_
>
>
>
> <?xml version="1.0"?>
>
> <!-- core-site.xml -->
>
>
>
> <configuration>
>
>         <property>
>
>                 <name>fs.default.name</name>
>
>                 <value>hdfs://roman-master/</value>
>
>                 <final>true</final>
>
>         </property>
>
>                 <property>
>
>                                <name>hadoop.tmp.dir</name>
>
>                                <value>/usr/local/hadoop_tmp_dir</value>
>
>                 </property>
>
> </configuration>
>
>
>
> <?xml version="1.0"?>
>
> <!-- hdfs-site.xml -->
>
> <configuration>
>
>                 <property>
>
>                                <name>dfs.name.dir</name>
>
>                                <value>/usr/local/dfs/namenode</value>
>
>                                <final>true</final>
>
>                 </property>
>
>                 <property>
>
>                                <name>dfs.data.dir</name>
>
>                                <value>/usr/local/dfs/datanode</value>
>
>                                <final>true</final>
>
>                 </property>
>
>                 <property>
>
>                                <name>fs.checkpoint.dir</name>
>
>                                <value>/usr/local/dfs/checkpoint</value>
>
>                                <final>true</final>
>
>                 </property>
>
> </configuration>
>
>
>
> *HBase conf*
>
>
>
> *hbase.env.sh <http://hbase.env.sh>*
>
> export HBASE_HEAPSIZE=4096
>
> export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/
>
> export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
>
>
>
>
>
> *hbase-site.xml*
>
> <?xml version="1.0"?>
>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
>
>
> <configuration>
>
>                 <property>
>
>                                <name>hbase.zookeeper.quorum</name>
>
>                                <value>roman-master</value>
>
>                 </property>
>
> <!--
>
>         <property>
>
>                 <name>hbase.zookeeper.property.clientPort</name>
>
>                 <value>5000</value>
>
>         </property>
>
> -->
>
>                 <property>
>
>
> <name>hbase.zookeeper.property.dataDir</name>
>
>                                <value>/usr/local/zookeeper</value>
>
>                 </property>
>
>                 <property>
>
>                                <name>hbase.rootdir</name>
>
>                                <value>hdfs://roman-master/hbase</value>
>
>                 </property>
>
>                 <property>
>
>                                <name>hbase.cluster.distributed</name>
>
>                                <value>true</value>
>
>                 </property>
>
> </configuration>
>
>
>
>
>
> [1]
>
> 2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1]
> regionserver.HRegion: Instantiated
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
>
> 2017-11-18 19:42:08,789 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1]
> regionserver.HRegion: Closing
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.: disabling
> compactions & flushes
>
> 2017-11-18 19:42:08,790 DEBUG [RegionOpenAndInitThread-SYSTEM.CATALOG-1]
> regionserver.HRegion: Updates disabled for region
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
>
> 2017-11-18 19:42:08,790 INFO  [RegionOpenAndInitThread-SYSTEM.CATALOG-1]
> regionserver.HRegion: Closed
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.
>
> 2017-11-18 19:42:08,801 INFO
> [MASTER_TABLE_OPERATIONS-roman-master:60000-0] catalog.MetaEditor: Added 1
>
> 2017-11-18 19:42:08,801 INFO
> [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager:
> Bulk assigning 1 region(s) across 3 server(s), round-robin=true
>
> 2017-11-18 19:42:08,802 DEBUG
> [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner:
> Timeout-on-RIT=151000
>
> 2017-11-18 19:42:08,802 DEBUG
> [roman-master,60000,1511033853960-GeneralBulkAssigner-2]
> master.AssignmentManager: Assigning 1 region(s) to
> roman-rs-3,60020,1511033854738
>
> 2017-11-18 19:42:08,803 DEBUG
> [roman-master,60000,1511033853960-GeneralBulkAssigner-2]
> zookeeper.ZKAssign: master:60000-0x15fd0a2d0c40000,
> quorum=roman-master:2181, baseZNode=/hbase Async create of unassigned node
> 6608080647cc1ca0c6610b92de5ae0c4 with OFFLINE state
>
> 2017-11-18 19:42:08,809 DEBUG [main-EventThread] master.OfflineCallback:
> rs={6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034128801,
> server=null}, server=roman-rs-3,60020,1511033854738
>
> 2017-11-18 19:42:08,810 DEBUG [main-EventThread]
> master.OfflineCallback$ExistCallback: rs={6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034128801, server=null},
> server=roman-rs-3,60020,1511033854738
>
> 2017-11-18 19:42:08,813 INFO
> [roman-master,60000,1511033853960-GeneralBulkAssigner-2]
> master.AssignmentManager: roman-rs-3,60020,1511033854738 unassigned
> znodes=1 of total=1
>
> 2017-11-18 19:42:08,813 INFO
> [roman-master,60000,1511033853960-GeneralBulkAssigner-2]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034128803, server=null} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034128813,
> server=roman-rs-3,60020,1511033854738}
>
> 2017-11-18 19:42:08,826 DEBUG
> [roman-master,60000,1511033853960-GeneralBulkAssigner-2]
> master.AssignmentManager: Bulk assigning done for
> roman-rs-3,60020,1511033854738
>
> 2017-11-18 19:42:08,826 DEBUG
> [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.GeneralBulkAssigner:
> bulk assigning total 1 regions to 3 servers, took 24ms, with 1 regions
> still in transition
>
> 2017-11-18 19:42:08,826 INFO
> [MASTER_TABLE_OPERATIONS-roman-master:60000-0] master.AssignmentManager:
> Bulk assigning done
>
> 2017-11-18 19:42:08,841 DEBUG
> [MASTER_TABLE_OPERATIONS-roman-master:60000-0] lock.ZKInterProcessLockBase:
> Released /hbase/table-lock/SYSTEM.CATALOG/write-master:600000000000000
>
> 2017-11-18 19:42:08,841 INFO
> [MASTER_TABLE_OPERATIONS-roman-master:60000-0] handler.CreateTableHandler:
> failed. null
>
> 2017-11-18 19:42:08,848 DEBUG [AM.ZK.Worker-pool2-t10]
> master.AssignmentManager: Handling RS_ZK_REGION_OPENING,
> server=roman-rs-3,60020,1511033854738,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN,
> ts=1511034128813, server=roman-rs-3,60020,1511033854738}
>
> 2017-11-18 19:42:08,848 INFO  [AM.ZK.Worker-pool2-t10]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=PENDING_OPEN, ts=1511034128813,
> server=roman-rs-3,60020,1511033854738} to {6608080647cc1ca0c6610b92de5ae0c4
> state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738}
>
> 2017-11-18 19:42:08,946 ERROR [FifoRpcScheduler.handler1-thread-12]
> master.HMaster: Region server roman-rs-3,60020,1511033854738 reported a
> fatal error:
>
> ABORTING region server roman-rs-3,60020,1511033854738: The coprocessor
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw
> java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
>
> Cause:
>
> java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
>
>                 at
> org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
>
>                 at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
>
>                 at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
>
>                 at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
>
>                 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
>                 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
>                 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>                 at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
>
>                 at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
>
>                 at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
>
>                 at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>                 at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.metrics2.MetricsInfo
>
>                 at
> java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>                 at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>                 ... 24 more
>
>
>
> 2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11]
> master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN,
> server=roman-rs-3,60020,1511033854738,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING,
> ts=1511034128848, server=roman-rs-3,60020,1511033854738}
>
> 2017-11-18 19:42:11,245 INFO  [AM.ZK.Worker-pool2-t11]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OPENING, ts=1511034128848, server=roman-rs-3,60020,1511033854738} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034131245,
> server=roman-rs-3,60020,1511033854738}
>
> 2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-3,60020,1511033854738 accepted as a dest server = false
>
> 2017-11-18 19:42:11,245 DEBUG [AM.ZK.Worker-pool2-t11]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated
> random
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2)
> available servers, forceNewPlan=true
>
> 2017-11-18 19:42:11,246 DEBUG [AM.ZK.Worker-pool2-t11]
> handler.ClosedRegionHandler: Handling CLOSED event for
> 6608080647cc1ca0c6610b92de5ae0c4
>
> 2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
>
> 2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11]
> master.AssignmentManager: Using pre-existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-2,60020,1511033854813
>
> 2017-11-18 19:42:11,247 INFO  [AM.ZK.Worker-pool2-t11]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=CLOSED, ts=1511034131246, server=roman-rs-3,60020,1511033854738} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034131247,
> server=roman-rs-3,60020,1511033854738}
>
> 2017-11-18 19:42:11,247 DEBUG [AM.ZK.Worker-pool2-t11] zookeeper.ZKAssign:
> master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase
> Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
> with OFFLINE state
>
> 2017-11-18 19:42:11,256 INFO  [AM.ZK.Worker-pool2-t11]
> master.AssignmentManager: Assigning
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-2,60020,1511033854813
>
> 2017-11-18 19:42:11,257 INFO  [AM.ZK.Worker-pool2-t11]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034131247, server=roman-rs-3,60020,1511033854738} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034131256,
> server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:11,257 DEBUG [AM.ZK.Worker-pool2-t11]
> master.ServerManager: New admin connection to roman-rs-2,60020,1511033854813
>
> 2017-11-18 19:42:11,424 DEBUG [AM.ZK.Worker-pool2-t12]
> master.AssignmentManager: Handling M_ZK_REGION_OFFLINE,
> server=roman-rs-2,60020,1511033854813,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN,
> ts=1511034131256, server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:11,444 DEBUG [AM.ZK.Worker-pool2-t13]
> master.AssignmentManager: Handling RS_ZK_REGION_OPENING,
> server=roman-rs-2,60020,1511033854813,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN,
> ts=1511034131256, server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:11,444 INFO  [AM.ZK.Worker-pool2-t13]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=PENDING_OPEN, ts=1511034131256,
> server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
> state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:11,591 ERROR [FifoRpcScheduler.handler1-thread-14]
> master.HMaster: Region server roman-rs-2,60020,1511033854813 reported a
> fatal error:
>
> ABORTING region server roman-rs-2,60020,1511033854813: The coprocessor
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw
> java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
>
> Cause:
>
> java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
>
>                 at
> org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
>
>                 at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
>
>                 at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
>
>                 at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
>
>                 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
>                 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
>                 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>                 at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
>
>                 at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
>
>                 at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
>
>                 at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>                 at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.metrics2.MetricsInfo
>
>                 at
> java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>                 at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>                 ... 24 more
>
>
>
> 2017-11-18 19:42:13,874 DEBUG [AM.ZK.Worker-pool2-t14]
> master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN,
> server=roman-rs-2,60020,1511033854813,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING,
> ts=1511034131444, server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:13,874 INFO  [AM.ZK.Worker-pool2-t14]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OPENING, ts=1511034131444, server=roman-rs-2,60020,1511033854813} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034133874,
> server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-2,60020,1511033854813 accepted as a dest server = false
>
> 2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated
> random
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=2)
> available servers, forceNewPlan=true
>
> 2017-11-18 19:42:13,887 DEBUG [AM.ZK.Worker-pool2-t14]
> handler.ClosedRegionHandler: Handling CLOSED event for
> 6608080647cc1ca0c6610b92de5ae0c4
>
> 2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
>
> 2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14]
> master.AssignmentManager: Using pre-existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-1,60020,1511033854754
>
> 2017-11-18 19:42:13,888 INFO  [AM.ZK.Worker-pool2-t14]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=CLOSED, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034133888,
> server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:13,888 DEBUG [AM.ZK.Worker-pool2-t14] zookeeper.ZKAssign:
> master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase
> Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
> with OFFLINE state
>
> 2017-11-18 19:42:13,896 INFO  [AM.ZK.Worker-pool2-t14]
> master.AssignmentManager: Assigning
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-1,60020,1511033854754
>
> 2017-11-18 19:42:13,896 INFO  [AM.ZK.Worker-pool2-t14]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034133888, server=roman-rs-2,60020,1511033854813} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034133896,
> server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:13,896 DEBUG [AM.ZK.Worker-pool2-t14]
> master.ServerManager: New admin connection to roman-rs-1,60020,1511033854754
>
> 2017-11-18 19:42:14,055 DEBUG [AM.ZK.Worker-pool2-t15]
> master.AssignmentManager: Handling M_ZK_REGION_OFFLINE,
> server=roman-rs-1,60020,1511033854754,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN,
> ts=1511034133896, server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:14,073 DEBUG [AM.ZK.Worker-pool2-t16]
> master.AssignmentManager: Handling RS_ZK_REGION_OPENING,
> server=roman-rs-1,60020,1511033854754,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN,
> ts=1511034133896, server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:14,073 INFO  [AM.ZK.Worker-pool2-t16]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=PENDING_OPEN, ts=1511034133896,
> server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
> state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:14,171 ERROR [FifoRpcScheduler.handler1-thread-16]
> master.HMaster: Region server roman-rs-1,60020,1511033854754 reported a
> fatal error:
>
> ABORTING region server roman-rs-1,60020,1511033854754: The coprocessor
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl threw
> java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
>
> Cause:
>
> java.lang.NoClassDefFoundError: org/apache/hadoop/metrics2/MetricsInfo
>
>                 at
> org.apache.phoenix.trace.util.Tracing.addTraceMetricsSource(Tracing.java:252)
>
>                 at
> org.apache.phoenix.coprocessor.MetaDataEndpointImpl.start(MetaDataEndpointImpl.java:484)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:673)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:265)
>
>                 at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:218)
>
>                 at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:227)
>
>                 at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:183)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:623)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:530)
>
>                 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
>                 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
>                 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>                 at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4160)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4471)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4444)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4400)
>
>                 at
> org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4351)
>
>                 at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:482)
>
>                 at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:145)
>
>                 at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>                 at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.metrics2.MetricsInfo
>
>                 at
> java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
>                 at
> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>
>                 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
>                 ... 24 more
>
>
>
> 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Handling RS_ZK_REGION_FAILED_OPEN,
> server=roman-rs-1,60020,1511033854754,
> region=6608080647cc1ca0c6610b92de5ae0c4,
> current_state={6608080647cc1ca0c6610b92de5ae0c4 state=OPENING,
> ts=1511034134073, server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:16,457 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OPENING, ts=1511034134073, server=roman-rs-1,60020,1511033854754} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=CLOSED, ts=1511034136457,
> server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-1,60020,1511033854754 accepted as a dest server = false
>
> 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated
> random
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=2)
> available servers, forceNewPlan=true
>
> 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17]
> handler.ClosedRegionHandler: Handling CLOSED event for
> 6608080647cc1ca0c6610b92de5ae0c4
>
> 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
>
> 2017-11-18 19:42:16,457 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Using pre-existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.;
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-2,60020,1511033854813
>
> 2017-11-18 19:42:16,478 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=CLOSED, ts=1511034136457, server=roman-rs-1,60020,1511033854754} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=OFFLINE, ts=1511034136478,
> server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:16,478 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign:
> master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase
> Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
> with OFFLINE state
>
> 2017-11-18 19:42:16,488 INFO  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Assigning
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-2,60020,1511033854813
>
> 2017-11-18 19:42:16,488 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034136478, server=roman-rs-1,60020,1511033854754} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136488,
> server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:16,491 WARN  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Failed assignment of
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-2,60020,1511033854813, trying to assign elsewhere instead; try=1
> of 10
>
> java.io.IOException: Unexpected closed connection
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1450)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
>
>                 at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
>
>                 at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
>
>                 at
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
>
>                 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>                 at java.lang.Thread.run(Thread.java:748)
>
> 2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
>
> 2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated
> random
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3)
> available servers, forceNewPlan=true
>
> 2017-11-18 19:42:16,494 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=PENDING_OPEN, ts=1511034136488,
> server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:16,494 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign:
> master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase
> Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
> with OFFLINE state
>
> 2017-11-18 19:42:16,502 INFO  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Assigning
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-1,60020,1511033854754
>
> 2017-11-18 19:42:16,502 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034136494, server=roman-rs-2,60020,1511033854813} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136502,
> server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:16,503 WARN  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Failed assignment of
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=2
> of 10
>
> java.io.IOException: Call to roman-rs-1/172.24.62.221:60020 failed on
> local exception: java.io.EOFException
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.wrapException(RpcClient.java:1489)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1461)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
>
>                 at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
>
>                 at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
>
>                 at
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
>
>                 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>                 at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.io.EOFException
>
>                 at
> java.io.DataInputStream.readInt(DataInputStream.java:392)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.readResponse(RpcClient.java:1076)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:727)
>
> 2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-1,60020,1511033854754 accepted as a dest server = true
>
> 2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated
> random
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-2,60020,1511033854813; 3 (online=3, available=3)
> available servers, forceNewPlan=true
>
> 2017-11-18 19:42:16,504 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=PENDING_OPEN, ts=1511034136502,
> server=roman-rs-1,60020,1511033854754} to {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:16,504 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign:
> master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase
> Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
> with OFFLINE state
>
> 2017-11-18 19:42:16,513 INFO  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Assigning
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-2,60020,1511033854813
>
> 2017-11-18 19:42:16,513 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034136504, server=roman-rs-1,60020,1511033854754} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136513,
> server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:16,515 WARN  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Failed assignment of
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-2,60020,1511033854813, trying to assign elsewhere instead; try=3
> of 10
>
> java.net.ConnectException: Connection refused
>
>                 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
>                 at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>
>                 at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>
>                 at
> org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
>
>                 at
> org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:868)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1543)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1442)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1661)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1719)
>
>                 at
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:20964)
>
>                 at
> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:671)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2097)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1577)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1550)
>
>                 at
> org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:104)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:1032)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1447)
>
>                 at
> org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1260)
>
>                 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>                 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>                 at java.lang.Thread.run(Thread.java:748)
>
> 2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Found an existing plan for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. destination
> server is roman-rs-2,60020,1511033854813 accepted as a dest server = true
>
> 2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: No previous transition plan found (or ignoring an
> existing plan) for
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.; generated
> random
> plan=hri=SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4.,
> src=, dest=roman-rs-1,60020,1511033854754; 3 (online=3, available=3)
> available servers, forceNewPlan=true
>
> 2017-11-18 19:42:16,515 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=PENDING_OPEN, ts=1511034136513,
> server=roman-rs-2,60020,1511033854813} to {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813}
>
> 2017-11-18 19:42:16,515 DEBUG [AM.ZK.Worker-pool2-t17] zookeeper.ZKAssign:
> master:60000-0x15fd0a2d0c40000, quorum=roman-master:2181, baseZNode=/hbase
> Creating (or updating) unassigned node 6608080647cc1ca0c6610b92de5ae0c4
> with OFFLINE state
>
> 2017-11-18 19:42:16,538 INFO  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Assigning
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-1,60020,1511033854754
>
> 2017-11-18 19:42:16,538 INFO  [AM.ZK.Worker-pool2-t17]
> master.RegionStates: Transition {6608080647cc1ca0c6610b92de5ae0c4
> state=OFFLINE, ts=1511034136515, server=roman-rs-2,60020,1511033854813} to
> {6608080647cc1ca0c6610b92de5ae0c4 state=PENDING_OPEN, ts=1511034136538,
> server=roman-rs-1,60020,1511033854754}
>
> 2017-11-18 19:42:16,539 WARN  [AM.ZK.Worker-pool2-t17]
> master.AssignmentManager: Failed assignment of
> SYSTEM.CATALOG,,1511034128528.6608080647cc1ca0c6610b92de5ae0c4. to
> roman-rs-1,60020,1511033854754, trying to assign elsewhere instead; try=4
> of 10
>
> java.net.ConnectException: Connection refused
>
>                 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
>                 at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>
>                 at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>
>                 at
> org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
>
>                 at
> org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
>
>                 at
> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupConnection(RpcClient.java:578)
>
>                 at org.apache.hadoop.hba
>

Mime
View raw message