phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Perko, Ralph J" <Ralph.Pe...@pnnl.gov>
Subject RE: hbase / phoenix errors
Date Tue, 07 Apr 2015 17:28:21 GMT
Thank you for the response

I am using Phoenix 4.3 as a separate installation.

Unfortunately I have no way to copy the actual log files so I will need to transcribe as much
as I can.

There are a lot of things going on – I’ll try to provide the highlights

Right now:
Using ambari – everything on the cluster is green – there are no apparent issues (but
there are many)

On the hbase master web site it shows a table split hung up (all red – “regions in transition”)
since yesterday evening.

All my phoenix tables are setup as follows:
Salted
100GB hregion max file size
Constant split size policy

If I attempt to connect to Phoenix using sqlline it get the exception:
NotServingRegionException:Region SYSTEM.CATALOG is not online

If I run hbase shell I can list the tables but cannot scan any of them

RS Log Messages:
Aside from the messages I provided earlier some errors and exceptions have come up as well
on the RS:

In order I believe:

ERROR StatsScanner failed to update stats table
ERROR largeCompaction Compaction Failed

ERROR largeCompaction Failed after attempt 350 – ConnectionRefused – this server is in
the failed servers list

Coprocessor.CoprocessorHost: the coprocessor …LocalIndexSplitter threw an exception
NoSuchMethodError hbase.regionserver.RegionServerService.getCatalogTracker

HRegion: compaction interrupted InterruptedOException
RuntimeException: HRegionServer aborted

Restart

ERROR RS_LOG_REPLAY wal.HLogSplitter  OutOFMemory

Restart

Many of these: RemoteException (LeaseExpiredException) Holder: DFSCLient…recovered.edits…:
File does not exist

Many java.net.ConnectionException: Connection refused

Java.net.ConnectionException SocketTimeoutException … row ‘’ on table ‘hbase.meta’

This is where we are today

I will provide whatever info you need

Thanks!
Ralph



From: Nick Dimiduk [mailto:ndimiduk@gmail.com]
Sent: Tuesday, April 07, 2015 9:05 AM
To: user@phoenix.apache.org
Subject: Re: hbase / phoenix errors

Also, beside each region server log file (.log) there's also the output file (.out). Check
the output files as well, as some serious crashes scenarios bypass the logs and go directly
to the out files.

-n

On Tuesday, April 7, 2015, Devaraj Das <ddas@hortonworks.com<mailto:ddas@hortonworks.com>>
wrote:
Hi Ralph, were you using the Phoenix bundled with HDP-2.2 or was that a separate installation?
Could you please copy/paste some log lines around the time of a regionserver's crash (look
for exceptions etc around that time in the regionserver logs).
Thanks
Devaraj

On Apr 6, 2015, at 3:00 PM, Perko, Ralph J <Ralph.Perko@pnnl.gov<javascript:_e(%7B%7D,'cvml','Ralph.Perko@pnnl.gov');>>
wrote:
Hi, we recently upgraded to Phoenix 4.3 and Hortonworks 2.2 (HBase .98.4) and we are running
into some issues.  I am wondering if I am missing something easy and hoping you can help.
 I have 34 regions servers and many keep crashing but without much in the way of error messages.

Here are the things that stand out:

ClientAsync.Process – waiting for some tasks to finish
smallCompaction RPCRetryingCaller: Call exception …. ‘msg row ‘SOME_PHOENIX_TABLE_NAME_IDX:<some
long key>’ on table: SYSTEM.STATS attempt 225/350

Similar ones for largeCompaction as well.

The other issue is the Pig loader hangs with these messages in the mapper logs:
[phoenix-1-thread-0] RPCRetryingCaller: Call exception msg row ‘’ on table ‘SYSTEM.CATALOG’

Eventually the mappers time out – no errors

Regions servers come up and down.  There are lots of connection refused errors as well.

Restarting hbase does not help.  The region servers will come up then go down again.

Zookeeper is up.  I’ve restarted just in case but it did not help

I cannot connect to Phoenix from the command line

Any help is appreciated.

Thanks!
Ralph

Mime
View raw message