phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas D'Silva" <tdsi...@salesforce.com>
Subject Re: Tephra errors when trying to create a transactional table in Phoenix 4.8.0
Date Wed, 31 Aug 2016 18:01:39 GMT
Can you check the Transaction Manager logs and see if there are any error?
Also can you do a jps and see confirm the Transaction Manager is running ?

On Wed, Aug 31, 2016 at 2:12 AM, F21 <f21.groups@gmail.com> wrote:

> Just another update. Even though the logs says that the transaction
> manager is not running, it is actually running.
>
> I confirmed this by checking the output of ps and connecting to the
> transaction manager:
>
> bash-4.3# ps
> PID   USER     TIME   COMMAND
>     1 root       0:01 bash /run-hbase-phoenix.sh
>   137 hadoop     0:19 /usr/lib/jvm/java-1.8-openjdk/bin/java
> -Dproc_master -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC
> -XX:PermSize=128m -XX:Ma
>   189 hadoop     0:08 /usr/lib/jvm/java-1.8-openjdk/bin/java
> -XX:+UseConcMarkSweepGC -cp /opt/hbase/bin/../lib/*:/opt/h
> base/bin/../conf/:/opt/hbase/phoenix-c
>   542 root       0:00 /bin/bash
>  9035 root       0:00 sleep 1
>  9036 root       0:00 ps
>
> bash-4.3# wget localhost:15165
> Connecting to localhost:15165 (127.0.0.1:15165)
> wget: error getting response: Connection reset by peer
>
>
> On 31/08/2016 3:25 PM, F21 wrote:
>
>> This only seems to be a problem when I have HBase running in fully
>> distributed mode (1 master, 1 regionserver and 1 zookeeper node in
>> different docker images).
>>
>> If I have HBase running in standalone mode with HBase and Phoenix and the
>> Query server in 1 docker image, it works correctly.
>>
>> On 31/08/2016 11:21 AM, F21 wrote:
>>
>>> I have HBase 1.2.2 and Phoenix 4.8.0 running on my HBase master running
>>> on alpine linux with OpenJDK JRE 8.
>>>
>>> This is my hbase-site.xml:
>>>
>>> <?xml version="1.0"?>
>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>> <configuration>
>>>   <property>
>>>     <name>hbase.rootdir</name>
>>>     <value>hdfs://mycluster/hbase</value>
>>>   </property>
>>>   <property>
>>>     <name>zookeeper.znode.parent</name>
>>>     <value>/hbase</value>
>>>   </property>
>>>   <property>
>>>     <name>hbase.cluster.distributed</name>
>>>     <value>true</value>
>>>   </property>
>>>   <property>
>>>     <name>hbase.zookeeper.quorum</name>
>>>     <value>m9edd51-zookeeper.m9edd51</value>
>>>   </property>
>>>   <property>
>>>     <name>data.tx.snapshot.dir</name>
>>>     <value>/tmp/tephra/snapshots</value>
>>>   </property>
>>>   <property>
>>>     <name>data.tx.timeout</name>
>>>     <value>60</value>
>>>   </property>
>>>   <property>
>>>     <name>phoenix.transactions.enabled</name>
>>>     <value>true</value>
>>>   </property>
>>> </configuration>
>>>
>>> I am able to start the master correctly. I am also able to create
>>> non-transactional table.
>>>
>>> However, if I create a transactional table, I get this error: ERROR
>>> [TTransactionServer-rpc-0] thrift.ProcessFunction: Internal error
>>> processing startShort
>>>
>>> This is what I see in the logs:
>>>
>>> 2016-08-31 01:08:33,560 WARN [main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
>>> zookeeper.ClientCnxn: Session 0x156de22abec0004 for server null, unexpected
>>> error, closing socket connection and attempting reconnect
>>> java.net.ConnectException: Connection refused
>>>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl
>>> .java:717)
>>>         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientC
>>> nxnSocketNIO.java:361)
>>>         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.
>>> java:1081)
>>> 2016-08-31 01:08:33,616 INFO  [DefaultMetricsCollector STOPPING]
>>> metrics.DefaultMetricsCollector: Stopped metrics reporter
>>> 2016-08-31 01:08:33,623 INFO  [ThriftRPCServer]
>>> tephra.TransactionManager: Took 170.7 ms to stop
>>> 2016-08-31 01:08:33,623 INFO  [ThriftRPCServer] rpc.ThriftRPCServer: RPC
>>> server for TTransactionServer stopped.
>>> 2016-08-31 01:08:34,776 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
>>> zookeeper.ClientCnxn: Opening socket connection to server
>>> m9edd51-zookeeper.m9edd51/172.18.0.2:2181. Will not attempt to
>>> authenticate using SASL (unknown error)
>>> 2016-08-31 01:08:34,777 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
>>> zookeeper.ClientCnxn: Socket connection established to
>>> m9edd51-zookeeper.m9edd51/172.18.0.2:2181, initiating session
>>> 2016-08-31 01:08:34,778 INFO [main-SendThread(m9edd51-zookeeper.m9edd51:2181)]
>>> zookeeper.ClientCnxn: Session establishment complete on server
>>> m9edd51-zookeeper.m9edd51/172.18.0.2:2181, sessionid =
>>> 0x156de22abec0004, negotiated timeout = 40000
>>> 2016-08-31 01:08:34,783 INFO [leader-election-tx.service-leader]
>>> zookeeper.LeaderElection: Connected to ZK, running election:
>>> m9edd51-zookeeper.m9edd51 for /tx.service/leader
>>> 2016-08-31 01:08:34,815 INFO  [ThriftRPCServer] rpc.ThriftRPCServer:
>>> Starting RPC server for TTransactionServer
>>> 2016-08-31 01:08:34,815 INFO  [ThriftRPCServer] rpc.ThriftRPCServer:
>>> Running RPC server for TTransactionServer
>>> 2016-08-31 01:08:34,816 INFO  [ThriftRPCServer]
>>> server.TThreadedSelectorServerWithFix: Starting
>>> TThreadedSelectorServerWithFix
>>> 2016-08-31 01:08:34,822 INFO [leader-election-tx.service-leader]
>>> distributed.TransactionService: Transaction Thrift Service started
>>> successfully on m9edd51-hmaster1.m9edd51/172.18.0.12:15165
>>> 2016-08-31 01:10:42,830 ERROR [TTransactionServer-rpc-0]
>>> thrift.ProcessFunction: Internal error processing startShort
>>> java.lang.IllegalStateException: Transaction Manager is not running.
>>>         at com.google.common.base.Preconditions.checkState(Precondition
>>> s.java:149)
>>>         at org.apache.tephra.TransactionManager.ensureAvailable(Transac
>>> tionManager.java:709)
>>>         at org.apache.tephra.TransactionManager.startTx(TransactionMana
>>> ger.java:768)
>>>         at org.apache.tephra.TransactionManager.startShort(TransactionM
>>> anager.java:728)
>>>         at org.apache.tephra.TransactionManager.startShort(TransactionM
>>> anager.java:716)
>>>         at org.apache.tephra.distributed.TransactionServiceThriftHandle
>>> r.startShort(TransactionServiceThriftHandler.java:71)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServ
>>> er.java:261)
>>>         at com.sun.proxy.$Proxy17.startShort(Unknown Source)
>>>         at org.apache.tephra.distributed.thrift.TTransactionServer$Proc
>>> essor$startShort.getResult(TTransactionServer.java:974)
>>>         at org.apache.tephra.distributed.thrift.TTransactionServer$Proc
>>> essor$startShort.getResult(TTransactionServer.java:959)
>>>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.
>>> java:39)
>>>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.
>>> java:39)
>>>         at org.apache.thrift.server.AbstractNonblockingServer$FrameBuff
>>> er.invoke(AbstractNonblockingServer.java:478)
>>>         at org.apache.thrift.server.Invocation.run(Invocation.java:18)
>>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1142)
>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:617)
>>>         at java.lang.Thread.run(Thread.java:745)
>>> 2016-08-31 01:10:42,936 ERROR [TTransactionServer-rpc-0]
>>> thrift.ProcessFunction: Internal error processing startShort
>>> java.lang.IllegalStateException: Transaction Manager is not running.
>>>         at com.google.common.base.Preconditions.checkState(Precondition
>>> s.java:149)
>>>         at org.apache.tephra.TransactionManager.ensureAvailable(Transac
>>> tionManager.java:709)
>>>         at org.apache.tephra.TransactionManager.startTx(TransactionMana
>>> ger.java:768)
>>>         at org.apache.tephra.TransactionManager.startShort(TransactionM
>>> anager.java:728)
>>>         at org.apache.tephra.TransactionManager.startShort(TransactionM
>>> anager.java:716)
>>>         at org.apache.tephra.distributed.TransactionServiceThriftHandle
>>> r.startShort(TransactionServiceThriftHandler.java:71)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>>         at java.lang.reflect.Method.invoke(Method.java:498)
>>>         at org.apache.tephra.rpc.ThriftRPCServer$1.invoke(ThriftRPCServ
>>> er.java:261)
>>>         at com.sun.proxy.$Proxy17.startShort(Unknown Source)
>>>         at org.apache.tephra.distributed.thrift.TTransactionServer$Proc
>>> essor$startShort.getResult(TTransactionServer.java:974)
>>>         at org.apache.tephra.distributed.thrift.TTransactionServer$Proc
>>> essor$startShort.getResult(TTransactionServer.java:959)
>>>         at org.apache.thrift.ProcessFunction.process(ProcessFunction.
>>> java:39)
>>>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.
>>> java:39)
>>>         at org.apache.thrift.server.AbstractNonblockingServer$FrameBuff
>>> er.invoke(AbstractNonblockingServer.java:478)
>>>         at org.apache.thrift.server.Invocation.run(Invocation.java:18)
>>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1142)
>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:617)
>>>         at java.lang.Thread.run(Thread.java:745)
>>>
>>> This seems very similar to PHOENIX-2504 (https://issues.apache.org/jir
>>> a/browse/PHOENIX-2504), but that appears to be fixed in 4.8.0?
>>>
>>> Any ideas why this might be happening?
>>>
>>> Cheers,
>>>
>>> Francis
>>>
>>>
>>>
>>
>

Mime
View raw message