phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rafa <raf...@gmail.com>
Subject Problem connecting JDBC client to a secure cluster
Date Tue, 11 Apr 2017 14:05:29 GMT
Hi everybody !,

We have a CDH 5.8 kerberized cluster in which we have installed Apache
Phoenix 4.7 (via CLABS parcel). Everything works as expected. The only
problem we are facing is when trying to connect a WeblogicServer to Apache
Phoenix via the fat client.

needed files are added in classpath: hbase-site.xml,core-site.xml and
hdfs-site.xml

Jaas.conf used:

Client {
com.sun.security.auth.module.Krb5LoginModule required principal="
phoenix@HADOOP.INT"
useKeyTab=true
keyTab=phoenix.keytab
storeKey=true
debug=true;
};

JDBC URL used: jdbc:phoenix:node-01u.xxxx.int:2181:hbase/phoenix@HADOOP.INT:
/wldoms/domcb1arqu/phoenix.keytab

The secured connection is made correctly in Zookeeper, but it never
succeeds when connecting to the HBase Master

17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client
environment:java.io.tmpdir=/tmp
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client
environment:java.compiler=<NA>
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client
environment:os.version=2.6.32-642.11.1.el6.x86_64
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client environment:user.name
=weblogic
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client
environment:user.home=/home/weblogic
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Client
environment:user.dir=/wldoms/dominios/domcb1arqu
17/04/10 12:38:23 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=node-01u.xxxx.int:2181 sessionTimeout=90000
watcher=hconnection-0x589619380x0, quorum=node-01u.xxxx.int:2181,
baseZNode=/hbase
17/04/10 12:38:23 INFO zookeeper.Login: successfully logged in.
17/04/10 12:38:23 INFO zookeeper.Login: TGT refresh thread started.
17/04/10 12:38:23 INFO zookeeper.Login: TGT valid starting at:        Mon
Apr 10 12:38:23 CEST 2017
17/04/10 12:38:23 INFO zookeeper.Login: TGT expires:                  Tue
Apr 11 12:38:23 CEST 2017
17/04/10 12:38:23 INFO zookeeper.Login: TGT refresh sleeping until: Tue Apr
11 08:20:46 CEST 2017
17/04/10 12:38:23 INFO client.ZooKeeperSaslClient: Client will use GSSAPI
as SASL mechanism.
17/04/10 12:38:23 INFO zookeeper.ClientCnxn: Opening socket connection to
server node-01u.xxxx.int/192.168.101.161:2181. Will attempt to
SASL-authenticate using Login Context section 'Client'
17/04/10 12:38:23 INFO zookeeper.ClientCnxn: Socket connection established,
initiating session, client: /192.168.60.6:49232, server:
node-01u.xxxx.int/192.168.101.161:2181
17/04/10 12:38:23 INFO zookeeper.ClientCnxn: Session establishment complete
on server node-01u.xxxx.int/192.168.101.161:2181, sessionid =
0x15afb9d0dee82a6, negotiated timeout = 60000
17/04/10 12:38:24 INFO metrics.Metrics: Initializing metrics system: phoenix
17/04/10 12:38:24 WARN impl.MetricsConfig: Cannot locate configuration:
tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
17/04/10 12:38:24 INFO impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
17/04/10 12:38:24 INFO impl.MetricsSystemImpl: phoenix metrics system
started
17/04/10 12:38:24 INFO Configuration.deprecation: hadoop.native.lib is
deprecated. Instead, use io.native.lib.available
17/04/10 12:39:13 INFO client.RpcRetryingCaller: Call exception, tries=10,
retries=35, started=48396 ms ago, cancelled=false, msg=
17/04/10 12:39:33 INFO client.RpcRetryingCaller: Call exception, tries=11,
retries=35, started=68572 ms ago, cancelled=false, msg=
17/04/10 12:39:53 INFO client.RpcRetryingCaller: Call exception, tries=12,
retries=35, started=88752 ms ago, cancelled=false, msg=
17/04/10 12:40:13 INFO client.RpcRetryingCaller: Call exception, tries=13,
retries=35, started=108791 ms ago, cancelled=false, msg=


keeps retrying until it finally fails.
....

We obtain:

Mon Apr 10 12:38:24 CEST 2017,
RpcRetryingCaller{globalStartTime=1491820704684, pause=100, retries=35},
org.apache.hadoop.hbase.MasterNotRunningException:
com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
node-05u.xxxx.int/192.168.101.167:60000 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection
to node-05u.xxxx.int/192.168.101.167:60000 is closing. Call id=0,
waitTime=32
Mon Apr 10 12:38:25 CEST 2017,
RpcRetryingCaller{globalStartTime=1491820704684, pause=100, retries=35},
org.apache.hadoop.hbase.MasterNotRunningException:
com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to
node-05u.xxxx.int/192.168.101.167:60000 failed on local exception:
org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection
to node-05u.xxxx.int/192.168.101.167:60000 is closing. Call id=1, waitTime=6


The connectivity to Hbase Master (port 60000) is ok from the WLS machine.

Looking at the HBase Master logs we see that the HBase Master is responding
everytime with "Authentication is required" error:


2017-04-10 12:47:15,849 DEBUG org.apache.hadoop.hbase.ipc.RpcServer:
RpcServer.listener,port=60000: connection from 192.168.60.6:34380; # active
connections: 5
2017-04-10 12:47:15,849 DEBUG org.apache.hadoop.hbase.ipc.RpcServer:
RpcServer.listener,port=60000: Caught exception while
reading:Authentication is required
2017-04-10 12:47:15,849 DEBUG org.apache.hadoop.hbase.ipc.RpcServer:
RpcServer.listener,port=60000: DISCONNECTING client 192.168.60.6:34380
because read count=-1. Number of active connections:

Executing a "hbase shell" manually inside the cluster after obtaining a
ticket with the same keytab we see:


2017-04-11 12:33:12,319 DEBUG org.apache.hadoop.hbase.ipc.RpcServer:
RpcServer.listener,port=60000: connection from 192.168.101.161:60370; #
active connections: 5
2017-04-11 12:33:12,330 DEBUG org.apache.hadoop.hbase.ipc.RpcServer:
Kerberos principal name is hbase/node-05u.xxxx.int@HADOOP.INT
2017-04-11 12:33:12,330 DEBUG
org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:hbase/
node-05u.xxxx.int@HADOOP.INT (auth:KERBEROS)
from:org.apache.hadoop.hbase.ipc.RpcServer$Connection.saslReadAndProcess(RpcServer.java:1354)
2017-04-11 12:33:12,331 DEBUG org.apache.hadoop.hbase.ipc.RpcServer:
Created SASL server with mechanism = GSSAPI
2017-04-11 12:33:12,331 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: Have
read input token of size 640 for processing by saslServer.evaluateResponse()
2017-04-11 12:33:12,333 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: Will
send token of size 108 from saslServer.
2017-04-11 12:33:12,335 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: Have
read input token of size 0 for processing by saslServer.evaluateResponse()
2017-04-11 12:33:12,335 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: Will
send token of size 32 from saslServer.
2017-04-11 12:33:12,336 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: Have
read input token of size 32 for processing by saslServer.evaluateResponse()
2017-04-11 12:33:12,336 DEBUG
org.apache.hadoop.hbase.security.HBaseSaslRpcServer: SASL server GSSAPI
callback: setting canonicalized client ID: phoenix@HADOOP.INT
2017-04-11 12:33:12,336 DEBUG org.apache.hadoop.hbase.ipc.RpcServer: SASL
server context established. Authenticated client: phoenix@HADOOP.INT
(auth:KERBEROS). Negotiated QoP is auth
2017-04-11 12:33:12,336 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Auth successful for phoenix@HADOOP.INT (auth:KERBEROS)
2017-04-11 12:33:12,338 INFO SecurityLogger.org.apache.hadoop.hbase.Server:
Connection from 192.168.101.161 port: 60370 with version info: version:
"1.2.0-cdh5.8.0" url:
"file:///data/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hbase-1.2.0-cdh5.8.0"
revision: "Unknown" user: "jenkins" date: "Tue Jul 12 16:09:11 PDT 2016"
src_checksum: "b910b34d6127cf42495e0a8bf37a0e9e"
2017-04-11 12:33:12,338 INFO
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization successful for phoenix@HADOOP.INT (auth:KERBEROS) for
protocol=interface
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingInterface

It seems that the JDBC driver is not trying to authenticate to Hbase.
Perhaps some of you have faced a similar situation or could point me to a
new direction.

Thank you very much for your help !
Best Regards,
rafa.

Mime
View raw message