Hi John,

 

Yup, that is exactly the error I was having before I put in the aliases. It does mean that your app should work fine if you deploy and run it from inside Amazon, but is a bit of a pain when you are developing / running / testing anything locally. If you figure out a cleverer way to fix it than editing host files everywhere please do let me know :-)

 

As to why it is complaining about the last one in the list, I don’t know – maybe because when it tried the first two it didn’t throw an error because it still had others left to try, but once it tried the last one it knows it’s game over? But that is pure speculation.

 

Cheers,

Matt

 

 

From: Brady, John [mailto:john.brady@intel.com]
Sent: 25 February 2015 16:18
To: user@phoenix.apache.org
Subject: RE: JDBC connection zookeeper error

 

Hi Matthew,

 

I think you are you right, it is the same issue.

 

Below is the output from the app. Is this similar to what you got? Interestingly the unknown host 172.xxx in error below is the last zookeeper node on the list in the code.  Do you know why it didn’t give an error for the other two?

 

Thanks

John

 

About to open database connection

org.apache.phoenix.exception.PhoenixIOException: com.google.protobuf.ServiceException: java.net.UnknownHostException: unknown host: ip-172-xx-x-x.eu-west-1.compute.internal

     at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:832)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1126)

     at org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(DelegateConnectionQueryServices.java:110)

     at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1590)

     at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:568)

     at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:175)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:271)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:263)

     at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:261)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1043)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1551)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1520)

     at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1520)

     at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162)

     at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:126)

     at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at mysqljbdctest.PhoenixJDBC.main(PhoenixJDBC.java:22)

Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: java.net.UnknownHostException: unknown host: ip-172- xx-x-x.eu-west-1.compute.internal

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(HConnectionManager.java:1651)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(HConnectionManager.java:1677)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterService(HConnectionManager.java:1884)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:2671)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:397)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:402)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:764)

     ... 20 more

Caused by: com.google.protobuf.ServiceException: java.net.UnknownHostException: unknown host: ip-172- xx-x-x.eu-west-1.compute.internal

     at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1673)

     at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1714)

     at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:42561)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(HConnectionManager.java:1688)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(HConnectionManager.java:1597)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(HConnectionManager.java:1623)

     ... 26 more

Caused by: java.net.UnknownHostException: unknown host: ip-172- xx-x-x.eu-west-1.compute.internal

     at org.apache.hadoop.hbase.ipc.RpcClient$Connection.<init>(RpcClient.java:385)

     at org.apache.hadoop.hbase.ipc.RpcClient.createConnection(RpcClient.java:351)

     at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1525)

     at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1437)

     at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1656)

     ... 31 more

 

 

From: Matthew Johnson [mailto:matt.johnson@algomi.com]
Sent: Wednesday, February 25, 2015 3:01 PM
To: user@phoenix.apache.org
Subject: RE: JDBC connection zookeeper error

 

Hi John,

 

Glad that you are able to get data when running on the cluster, it means that at least your Phoenix server is working. Can you send the entire log output for your app, up to and including the stacktrace? I will compare it with the output from one of my apps that works.

 

To set the timeout and retries you should include a file called hbase-site.xml onto your classpath and set the following:

 

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

  <property>

    <name>hbase.client.retries.number</name>

    <value>3</value>

  </property>

  <property>

    <name>zookeeper.session.timeout</name>

    <value>60000</value>

  </property>

  <property>

    <name>zookeeper.recovery.retry</name>

    <value>3</value>

  </property>

</configuration>

 

 

Being able to telnet into zookeeper does not mean that your connection to HBase will work. The problem I was having is this:

 

-          App successfully connects to Zookeeper on 54.x.x.x

-          Zookeeper responds OK, and returns a 172.x.x.x IP address for connecting to HBase

-          App then uses the IP given by Zookeeper and tries to connect to HBase on 172.x.x.x

-          App connection fails with a weird error that looks a lot like the one you are getting (because 172.x.x.x is not visible from your local machine)

 

I have solved it with the following (repeat for each server in your cluster):

-          Add aliases on local machine to C:\Windows\System32\Drivers\etc\hosts

o   54.x.x.x   box1

-          Add alias on cluster machine to /etc/hosts

o   172.x.x.x   box1

-          Change hbase/conf/hbase-site.xml on each server in cluster to refer to everything by aliases:

o   <name>hbase.rootdir</name>  <value>hdfs://box1:8020/myhb</value>

o   <name>hbase.zookeeper.quorum</name> <value>box1 </value>

 

(I have also changed hadoop-2.6.0/etc/hadoop/core-site.xml and hadoop-2.6.0/etc/hadoop/yarn-site.xml to use these aliases as well, not sure if it’s necessary but I did it for consistency).

 

This way, App connects to ZK on ‘box1’, ZK returns ‘HBase is running on box1’, and app connects to HBase on ‘box1’. I am sure there is a neater way of doing this but this worked for me.

 

Cheers,

Matt

 

 

 

From: Brady, John [mailto:john.brady@intel.com]
Sent: 25 February 2015 14:19
To: user@phoenix.apache.org
Subject: RE: JDBC connection zookeeper error

 

Hi Matthew,

 

I was able to run the app successfully on the cluster and get data, thanks.

 

I am able to telnet from my local machine to one of the zookeeper nodes on port 2181 successfully also.

 

I think the code might just be timing out on my machine.

 

Is there a way to increase the timeout time? On the drivermanager object?

 

Thanks

 

 

From: Matthew Johnson [mailto:matt.johnson@algomi.com]
Sent: Tuesday, February 24, 2015 3:25 PM
To: user@phoenix.apache.org
Subject: RE: JDBC connection zookeeper error

 

Hi John,

 

If it didn’t return anything (as opposed to throwing an error) then I would assume it was actually able to connect but that the table has no data in it? Maybe try again, but try adding some logging before and after your select statement, for example printing out the size of the resultset?

 

If you did not see the error about Zookeeper being unable to connect when you run it on your cluster, then that suggests to me that you are having the same problem I had with Zookeeper on Amazon servers. Are you able to connect to HBase using native HBase client (instead of Phoenix) and running a simple query? Before I replaced all my IP addresses with internal/external aliases, I was not able to query HBase from my local PC at all let alone with Phoenix.

 

Cheers,

Matt

 

 

From: Brady, John [mailto:john.brady@intel.com]
Sent: 23 February 2015 21:43
To: user@phoenix.apache.org
Subject: RE: JDBC connection zookeeper error

 

Hi Matthew,

 

I tried using the full zookeeper quorum below. Do you need to specify the port 2181?

 

I tried running the app on the cluster. It didn’t return anything therefore I presume it didn’t connect. It is just the example US_POPULATION table.

 

This is my code:

 

import java.sql.*;

 

public class PhoenixJDBC {

    

     public static void main(String args[]) {

    

     try {

         //Register JDBC Driver

         Class.forName("org.apache.phoenix.jdbc.PhoenixDriver").newInstance();

      

         Connection conn = DriverManager.getConnection("jdbc:phoenix:54.154.34.128,54.154.161.140,54.154.181.8","","");

        

         //Create a Statement class to execute the SQL statement

         Statement stmt = conn.createStatement();

    

         //Execute the SQL statement and get the results in a Resultset

         ResultSet rs = stmt.executeQuery("select * from US_POPULATION");

    

         // Iterate through the ResultSet, displaying two values

         // for each row using the getString method

    

         while (rs.next())

             System.out.println("Name= " + rs.getString("host"));

     }

     catch (SQLException e) {

         e.printStackTrace();

     }

     catch (Exception e) {

         e.printStackTrace();

     }

     }

     }

 

Thanks

John

From: Matthew Johnson [mailto:matt.johnson@algomi.com]
Sent: Monday, February 23, 2015 5:31 PM
To: user@phoenix.apache.org
Subject: RE: JDBC connection zookeeper error

 

Hi John,

 

I cross-posted this on StackOverflow:

 

Have you tried entering the full zookeeper quorum in your URL? Something like this (you say you have 3 zookeepers in your cluster, obviously replace IPs below to match your zk quorum):

jdbc:phoenix:54.152.31.122,54.152.31.123,54.152.31.124

 

I had a problem on Amazon (is that where your cluster is?) where internal and external IPs were getting confused and zookeeper couldn't connect to HBase properly. I solved this (not the most scalable solution but it works!) by creating aliases in /etc/hosts on the machines in the cluster pointing at internal IP addresses, then on my local desktop using the same aliases but pointing to the external IPs. Then, altered my cluster setup to use aliases everywhere instead of IP addresses. Quick way to test if this is your problem - build your app, copy it onto one of the servers in your cluster, and see if it will run ok on there.

Hope that helps!

Cheers,

Matt

 

 

From: Nick Dimiduk [mailto:ndimiduk@gmail.com]
Sent: 23 February 2015 17:25
To: user@phoenix.apache.org
Subject: Re: JDBC connection zookeeper error

 

Can you double-check the Zookeeper quorum, port, and root node? The HBase Master web UI shows the first two, the third you can get to by looking at the output of zk_dump.jsp

 

On Mon, Feb 23, 2015 at 8:58 AM, Brady, John <john.brady@intel.com> wrote:

Hi Matthew,

 

Yes, I just doubled checked. The HBase master is running and I can access the HMaster status page – I can see the tables I created in Phoenix.

 

Would you have any other ideas?

 

Thanks

 

 

 

From: Matthew Johnson [mailto:matt.johnson@algomi.com]
Sent: Monday, February 23, 2015 3:13 PM
To: user@phoenix.apache.org
Subject: RE: JDBC connection zookeeper error

 

Hi John,

 

Is your HBase Master running? (It shows up in linux process list as HMaster). Are you able to access the HMaster status page? (On mine, it is http://<ip address>:60010).

 

Cheers,

Matt

 

 

From: Brady, John [mailto:john.brady@intel.com]
Sent: 23 February 2015 14:59
To: user@phoenix.apache.org
Subject: JDBC connection zookeeper error

 

Hello,

 

I have Phoenix installed on a 5 node Cloudera cluster with 3 zookeeper nodes.

 

I’m trying to run a basic SQL command from the JDBC program below but I’m getting zookeeper errors. Could anybody advise?

 

org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase

     at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:832)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1126)

     at org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(DelegateConnectionQueryServices.java:110)

     at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1590)

     at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:568)

     at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:175)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:271)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:263)

     at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:261)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1043)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1551)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1520)

     at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:77)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:1520)

     at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:162)

     at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(PhoenixEmbeddedDriver.java:126)

     at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:133)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at mysqljbdctest.PhoenixJDBC.main(PhoenixJDBC.java:15)

Caused by: org.apache.hadoop.hbase.MasterNotRunningException: org.apache.hadoop.hbase.MasterNotRunningException: Can't get connection to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(HConnectionManager.java:1651)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(HConnectionManager.java:1677)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterService(HConnectionManager.java:1884)

     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(HConnectionManager.java:2671)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:397)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:402)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:764)

     ... 20 more

 

 

 

package mysqljbdctest;

 

import java.sql.*;

 

public class PhoenixJDBC {

    

     public static void main(String args[]) {

    

     try {

         //Register JDBC Driver

         Class.forName("org.apache.phoenix.jdbc.PhoenixDriver").newInstance();

      

         Connection conn = DriverManager.getConnection("jdbc:phoenix:54.152.31.122","","");

        

         //Create a Statement class to execute the SQL statement

         Statement stmt = conn.createStatement();

    

         //Execute the SQL statement and get the results in a Resultset

         ResultSet rs = stmt.executeQuery("select * from US_POPULATION");

    

         // Iterate through the ResultSet, displaying two values

         // for each row using the getString method

    

         while (rs.next())

             System.out.println("Name= " + rs.getString("host"));

     }

     catch (SQLException e) {

         e.printStackTrace();

     }

     catch (Exception e) {

         e.printStackTrace();

     }

     }

     }

 

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

 

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.