I think we discovered what the problem was. There was an issue with permissions on the local/jars directory.  This caused the other exception, looks like.

Caused by: java.lang.RuntimeException: Failed to create local dir /mnt/myco/temp/hbase-tomcat/local/jars, DynamicClassLoader failed to init

at org.apache.hadoop.hbase.util.DynamicClassLoader.<init>(DynamicClassLoader.java:94) ~[hbase-common-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.protobuf.ProtobufUtil.<clinit>(ProtobufUtil.java:201) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:897) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:694) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_60]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_60]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_60]

at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_60]

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:455) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:350) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:280) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]


On Tue, Sep 8, 2015 at 2:37 PM, Mujtaba Chohan <mujtaba@apache.org> wrote:
Can you try with phoenix-client jar instead of phoenix-client-minimal jar?

On Tue, Sep 8, 2015 at 10:42 AM, Dmitry Goldenberg <dgoldenberg123@gmail.com> wrote:
I'm getting this error while trying to connect to HBase in a clustered environment. The code seems to work fine in a single node environment.

The full set of stack traces is below.

I can see that ProtobufUtil class is in the hbase-client-0.98.9-hadoop2.jar on the classpath. Google's protobuf-java-2.5.0.jar is also on the classpath.

Does anyone have an idea as to what might be going wrong here? Thanks.

Caused by: java.sql.SQLException: ERROR 103 (08004): Unable to establish connection.

at org.apache.phoenix.exception.SQLExceptionCode$Factory$1.newException(SQLExceptionCode.java:362) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:133) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:283) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:166) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1831) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:126) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_60]

at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[?:1.8.0_60]

at com.myco.util.SqlUtils.getDbConnection(SqlUtils.java:208) ~[core-model-0.0.1-SNAPSHOT.jar:?]

... 35 more

Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:457) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:350) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:280) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:166) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1831) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:126) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_60]

at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[?:1.8.0_60]

at com.myco.util.SqlUtils.getDbConnection(SqlUtils.java:208) ~[core-model-0.0.1-SNAPSHOT.jar:?]

... 35 more

Caused by: java.lang.reflect.InvocationTargetException

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_60]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_60]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_60]

at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_60]

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:455) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:350) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:280) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:166) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1831) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:126) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_60]

at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[?:1.8.0_60]

at com.myco.util.SqlUtils.getDbConnection(SqlUtils.java:208) ~[core-model-0.0.1-SNAPSHOT.jar:?]

... 35 more

Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.protobuf.ProtobufUtil

at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:69) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:83) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.retrieveClusterId(HConnectionManager.java:897) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:694) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_60]

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_60]

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_60]

at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_60]

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:455) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.hadoop.hbase.client.HConnectionManager.createConnection(HConnectionManager.java:350) ~[hbase-client-0.98.9-hadoop2.jar:0.98.9-hadoop2]

at org.apache.phoenix.query.HConnectionFactory$HConnectionFactoryImpl.createConnection(HConnectionFactory.java:47) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.openConnection(ConnectionQueryServicesImpl.java:280) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.access$300(ConnectionQueryServicesImpl.java:166) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1831) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl$11.call(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1810) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:126) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133) ~[phoenix-4.3.1-client-minimal-hbase0.98.9-hadoop2.4.0.jar:?]

at java.sql.DriverManager.getConnection(DriverManager.java:664) ~[?:1.8.0_60]

at java.sql.DriverManager.getConnection(DriverManager.java:247) ~[?:1.8.0_60]