phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Johnson <matt.john...@algomi.com>
Subject RE: JDBC connection zookeeper error
Date Thu, 26 Feb 2015 17:46:52 GMT
Hi John,



Afraid not, I’m using the basic Apache install – I don’t have any
experience of Cloudera unfortunately…



Cheers,

Matt





*From:* Brady, John [mailto:john.brady@intel.com]
*Sent:* 26 February 2015 17:30
*To:* user@phoenix.apache.org
*Cc:* Hugh McBride
*Subject:* RE: JDBC connection zookeeper error



Hi Matthew,



Can I clarify that you had Cloudera installed too?



When I modified the 3 config files directly on the servers to refer to
aliases, I was able to connect from my local machine but it killed
Cloudera.



Is there a way to do it through the Cloundera UI?



Thanks

John







*From:* Matthew Johnson [mailto:matt.johnson@algomi.com
<matt.johnson@algomi.com>]
*Sent:* Wednesday, February 25, 2015 4:33 PM
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi John,



Yup, that is exactly the error I was having before I put in the aliases. It
does mean that your app should work fine if you deploy and run it from
inside Amazon, but is a bit of a pain when you are developing / running /
testing anything locally. If you figure out a cleverer way to fix it than
editing host files everywhere please do let me know :-)



As to why it is complaining about the last one in the list, I don’t know –
maybe because when it tried the first two it didn’t throw an error because
it still had others left to try, but once it tried the last one it knows
it’s game over? But that is pure speculation.



Cheers,

Matt





*From:* Brady, John [mailto:john.brady@intel.com]
*Sent:* 25 February 2015 16:18
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi Matthew,



I think you are you right, it is the same issue.



Below is the output from the app. Is this similar to what you got?
Interestingly the unknown host 172.xxx in error below is the last zookeeper
node on the list in the code.  Do you know why it didn’t give an error for
the other two?



Thanks

John



About to open database connection

*org.apache.phoenix.exception.PhoenixIOException*:
*com.google.protobuf.ServiceException*: *java.net.UnknownHostException*:
unknown host: ip-172-xx-x-x.eu-west-1.compute.internal

     at org.apache.phoenix.util.ServerUtil.parseServerException(
*ServerUtil.java:108*)

     at
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(
*ConnectionQueryServicesImpl.java:832*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(
*ConnectionQueryServicesImpl.java:1126*)

     at
org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(
*DelegateConnectionQueryServices.java:110*)

     at org.apache.phoenix.schema.MetaDataClient.createTableInternal(
*MetaDataClient.java:1590*)

     at org.apache.phoenix.schema.MetaDataClient.createTable(
*MetaDataClient.java:568*)

     at org.apache.phoenix.compile.CreateTableCompiler$2.execute(
*CreateTableCompiler.java:175*)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
*PhoenixStatement.java:271*)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
*PhoenixStatement.java:263*)

     at org.apache.phoenix.call.CallRunner.run(*CallRunner.java:53*)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(
*PhoenixStatement.java:261*)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(
*PhoenixStatement.java:1043*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(
*ConnectionQueryServicesImpl.java:1551*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(
*ConnectionQueryServicesImpl.java:1520*)

     at org.apache.phoenix.util.PhoenixContextExecutor.call(
*PhoenixContextExecutor.java:77*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(
*ConnectionQueryServicesImpl.java:1520*)

     at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(
*PhoenixDriver.java:162*)

     at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(
*PhoenixEmbeddedDriver.java:126*)

     at org.apache.phoenix.jdbc.PhoenixDriver.connect(
*PhoenixDriver.java:133*)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at mysqljbdctest.PhoenixJDBC.main(*PhoenixJDBC.java:22*)

Caused by: *org.apache.hadoop.hbase.MasterNotRunningException*:
*com.google.protobuf.ServiceException*: *java.net.UnknownHostException*:
unknown host: ip-172- xx-x-x.eu-west-1.compute.internal

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(
*HConnectionManager.java:1651*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(
*HConnectionManager.java:1677*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterService(
*HConnectionManager.java:1884*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(
*HConnectionManager.java:2671*)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(
*HBaseAdmin.java:397*)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(
*HBaseAdmin.java:402*)

     at
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(
*ConnectionQueryServicesImpl.java:764*)

     ... 20 more

Caused by: *com.google.protobuf.ServiceException*:
*java.net.UnknownHostException*: unknown host: ip-172-
xx-x-x.eu-west-1.compute.internal

     at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(
*RpcClient.java:1673*)

     at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(
*RpcClient.java:1714*)

     at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(
*MasterProtos.java:42561*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(
*HConnectionManager.java:1688*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(
*HConnectionManager.java:1597*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(
*HConnectionManager.java:1623*)

     ... 26 more

Caused by: *java.net.UnknownHostException*: unknown host: ip-172-
xx-x-x.eu-west-1.compute.internal

     at org.apache.hadoop.hbase.ipc.RpcClient$Connection.<init>(
*RpcClient.java:385*)

     at org.apache.hadoop.hbase.ipc.RpcClient.createConnection(
*RpcClient.java:351*)

     at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(
*RpcClient.java:1525*)

     at org.apache.hadoop.hbase.ipc.RpcClient.call(*RpcClient.java:1437*)

     at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(
*RpcClient.java:1656*)

     ... 31 more





*From:* Matthew Johnson [mailto:matt.johnson@algomi.com
<matt.johnson@algomi.com>]
*Sent:* Wednesday, February 25, 2015 3:01 PM
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi John,



Glad that you are able to get data when running on the cluster, it means
that at least your Phoenix server is working. Can you send the entire log
output for your app, up to and including the stacktrace? I will compare it
with the output from one of my apps that works.



To set the timeout and retries you should include a file called
*hbase-site.xml* onto your classpath and set the following:



<?xml version=*"1.0"*?>

<?xml-stylesheet type=*"text/xsl"* href=*"configuration.xsl"*?>

<configuration>

  <property>

    <name>hbase.client.retries.number</name>

    <value>3</value>

  </property>

  <property>

    <name>zookeeper.session.timeout</name>

    <value>60000</value>

  </property>

  <property>

    <name>zookeeper.recovery.retry</name>

    <value>3</value>

  </property>

</configuration>





Being able to telnet into zookeeper does not mean that your connection to
HBase will work. The problem I was having is this:



-          App successfully connects to Zookeeper on 54.x.x.x

-          Zookeeper responds OK, and returns a 172.x.x.x IP address for
connecting to HBase

-          App then uses the IP given by Zookeeper and tries to connect to
HBase on 172.x.x.x

-          App connection fails with a weird error that looks a lot like
the one you are getting (because 172.x.x.x is not visible from your local
machine)



I have solved it with the following (repeat for each server in your
cluster):

-          Add aliases on local machine to
*C:\Windows\System32\Drivers\etc\hosts*

o   54.x.x.x   box1

-          Add alias on cluster machine to */etc/hosts*

o   172.x.x.x   box1

-          Change *hbase/conf/hbase-site.xml* on each server in cluster to
refer to everything by aliases:

o   <name>hbase.rootdir</name>  <value>hdfs://box1:8020/myhb</value>

o   <name>hbase.zookeeper.quorum</name> <value>box1 </value>



(I have also changed *hadoop-2.6.0/etc/hadoop/core-site.xml* and
*hadoop-2.6.0/etc/hadoop/yarn-site.xml* to use these aliases as well, not
sure if it’s necessary but I did it for consistency).



This way, App connects to ZK on ‘box1’, ZK returns ‘HBase is running on
box1’, and app connects to HBase on ‘box1’. I am sure there is a neater way
of doing this but this worked for me.



Cheers,

Matt







*From:* Brady, John [mailto:john.brady@intel.com]
*Sent:* 25 February 2015 14:19
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi Matthew,



I was able to run the app successfully on the cluster and get data, thanks.



I am able to telnet from my local machine to one of the zookeeper nodes on
port 2181 successfully also.



I think the code might just be timing out on my machine.



Is there a way to increase the timeout time? On the drivermanager object?



Thanks





*From:* Matthew Johnson [mailto:matt.johnson@algomi.com
<matt.johnson@algomi.com>]
*Sent:* Tuesday, February 24, 2015 3:25 PM
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi John,



If it didn’t return anything (as opposed to throwing an error) then I would
assume it was actually able to connect but that the table has no data in
it? Maybe try again, but try adding some logging before and after your
select statement, for example printing out the size of the resultset?



If you did not see the error about Zookeeper being unable to connect when
you run it on your cluster, then that suggests to me that you are having
the same problem I had with Zookeeper on Amazon servers. Are you able to
connect to HBase using native HBase client (instead of Phoenix) and running
a simple query? Before I replaced all my IP addresses with
internal/external aliases, I was not able to query HBase from my local PC
at all let alone with Phoenix.



Cheers,

Matt





*From:* Brady, John [mailto:john.brady@intel.com]
*Sent:* 23 February 2015 21:43
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi Matthew,



I tried using the full zookeeper quorum below. Do you need to specify the
port 2181?



I tried running the app on the cluster. It didn’t return anything therefore
I presume it didn’t connect. It is just the example US_POPULATION table.



This is my code:



*import* java.sql.*;



*public* *class* PhoenixJDBC {



     *public* *static* *void* main(String args[]) {



     *try* {

         //Register JDBC Driver

         Class.*forName*("org.apache.phoenix.jdbc.PhoenixDriver"
).newInstance();



         Connection conn = DriverManager.*getConnection*(
"jdbc:phoenix:54.154.34.128,54.154.161.140,54.154.181.8","","");



         //Create a Statement class to execute the SQL statement

         Statement stmt = conn.createStatement();



         //Execute the SQL statement and get the results in a *Resultset*

         ResultSet rs = stmt.executeQuery("select * from US_POPULATION");



         // Iterate through the ResultSet, displaying two values

         // for each row using the getString method



         *while* (rs.next())

             System.*out*.println("Name= " + rs.getString("host"));

     }

     *catch* (SQLException e) {

         e.printStackTrace();

     }

     *catch* (Exception e) {

         e.printStackTrace();

     }

     }

     }



Thanks

John

*From:* Matthew Johnson [mailto:matt.johnson@algomi.com
<matt.johnson@algomi.com>]
*Sent:* Monday, February 23, 2015 5:31 PM
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi John,



I cross-posted this on StackOverflow:



Have you tried entering the full zookeeper quorum in your URL? Something
like this (you say you have 3 zookeepers in your cluster, obviously replace
IPs below to match your zk quorum):

jdbc:phoenix:54.152.31.122,54.152.31.123,54.152.31.124



I had a problem on Amazon (is that where your cluster is?) where internal
and external IPs were getting confused and zookeeper couldn't connect to
HBase properly. I solved this (not the most scalable solution but it
works!) by creating aliases in /etc/hosts on the machines in the cluster
pointing at internal IP addresses, then on my local desktop using the same
aliases but pointing to the external IPs. Then, altered my cluster setup to
use aliases everywhere instead of IP addresses. Quick way to test if this
is your problem - build your app, copy it onto one of the servers in your
cluster, and see if it will run ok on there.

Hope that helps!

Cheers,

Matt





*From:* Nick Dimiduk [mailto:ndimiduk@gmail.com]
*Sent:* 23 February 2015 17:25
*To:* user@phoenix.apache.org
*Subject:* Re: JDBC connection zookeeper error



Can you double-check the Zookeeper quorum, port, and root node? The HBase
Master web UI shows the first two, the third you can get to by looking at
the output of zk_dump.jsp



On Mon, Feb 23, 2015 at 8:58 AM, Brady, John <john.brady@intel.com> wrote:

Hi Matthew,



Yes, I just doubled checked. The HBase master is running and I can access
the HMaster status page – I can see the tables I created in Phoenix.



Would you have any other ideas?



Thanks







*From:* Matthew Johnson [mailto:matt.johnson@algomi.com]
*Sent:* Monday, February 23, 2015 3:13 PM
*To:* user@phoenix.apache.org
*Subject:* RE: JDBC connection zookeeper error



Hi John,



Is your HBase Master running? (It shows up in linux process list as
HMaster). Are you able to access the HMaster status page? (On mine, it is
http://<ip address>:60010).



Cheers,

Matt





*From:* Brady, John [mailto:john.brady@intel.com]
*Sent:* 23 February 2015 14:59
*To:* user@phoenix.apache.org
*Subject:* JDBC connection zookeeper error



Hello,



I have Phoenix installed on a 5 node Cloudera cluster with 3 zookeeper
nodes.



I’m trying to run a basic SQL command from the JDBC program below but I’m
getting zookeeper errors. Could anybody advise?



*org.apache.phoenix.exception.PhoenixIOException*:
*org.apache.hadoop.hbase.MasterNotRunningException*: Can't get connection
to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase

     at org.apache.phoenix.util.ServerUtil.parseServerException(
*ServerUtil.java:108*)

     at
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(
*ConnectionQueryServicesImpl.java:832*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(
*ConnectionQueryServicesImpl.java:1126*)

     at
org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(
*DelegateConnectionQueryServices.java:110*)

     at org.apache.phoenix.schema.MetaDataClient.createTableInternal(
*MetaDataClient.java:1590*)

     at org.apache.phoenix.schema.MetaDataClient.createTable(
*MetaDataClient.java:568*)

     at org.apache.phoenix.compile.CreateTableCompiler$2.execute(
*CreateTableCompiler.java:175*)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
*PhoenixStatement.java:271*)

     at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
*PhoenixStatement.java:263*)

     at org.apache.phoenix.call.CallRunner.run(*CallRunner.java:53*)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(
*PhoenixStatement.java:261*)

     at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(
*PhoenixStatement.java:1043*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(
*ConnectionQueryServicesImpl.java:1551*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(
*ConnectionQueryServicesImpl.java:1520*)

     at org.apache.phoenix.util.PhoenixContextExecutor.call(
*PhoenixContextExecutor.java:77*)

     at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(
*ConnectionQueryServicesImpl.java:1520*)

     at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(
*PhoenixDriver.java:162*)

     at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.connect(
*PhoenixEmbeddedDriver.java:126*)

     at org.apache.phoenix.jdbc.PhoenixDriver.connect(
*PhoenixDriver.java:133*)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at java.sql.DriverManager.getConnection(Unknown Source)

     at mysqljbdctest.PhoenixJDBC.main(*PhoenixJDBC.java:15*)

Caused by: *org.apache.hadoop.hbase.MasterNotRunningException*:
*org.apache.hadoop.hbase.MasterNotRunningException*: Can't get connection
to ZooKeeper: KeeperErrorCode = ConnectionLoss for /hbase

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$StubMaker.makeStub(
*HConnectionManager.java:1651*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(
*HConnectionManager.java:1677*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getKeepAliveMasterService(
*HConnectionManager.java:1884*)

     at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHTableDescriptor(
*HConnectionManager.java:2671*)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(
*HBaseAdmin.java:397*)

     at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(
*HBaseAdmin.java:402*)

     at
org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(
*ConnectionQueryServicesImpl.java:764*)

     ... 20 more







*package* mysqljbdctest;



*import* java.sql.*;



*public* *class* PhoenixJDBC {



     *public* *static* *void* main(String args[]) {



     *try* {

         //Register JDBC Driver

         Class.*forName*("org.apache.phoenix.jdbc.PhoenixDriver"
).newInstance();



         Connection conn = DriverManager.*getConnection*(
"jdbc:phoenix:54.152.31.122","","");



         //Create a Statement class to execute the SQL statement

         Statement stmt = conn.createStatement();



         //Execute the SQL statement and get the results in a *Resultset*

         ResultSet rs = stmt.executeQuery("select * from US_POPULATION");



         // Iterate through the ResultSet, displaying two values

         // for each row using the getString method



         *while* (rs.next())

             System.*out*.println("Name= " + rs.getString("host"));

     }

     *catch* (SQLException e) {

         e.printStackTrace();

     }

     *catch* (Exception e) {

         e.printStackTrace();

     }

     }

     }



-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.



-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Mime
View raw message