phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samarth Jain <samarth.j...@gmail.com>
Subject Re: hbase / phoenix errors
Date Tue, 07 Apr 2015 20:32:06 GMT
I think that page needs to be updated. Sorry about that, Ralph. We ran into
problems with HBase 0.98.4 and local indexes where a similar (but not the
same) error was thrown:

Coprocessor.CoprocessorHost: the coprocessor …LocalIndexSplitter threw an
exception
NoSuchMethodError hbase.regionserver.RegionServerService.getCatalogTracker

See https://issues.apache.org/jira/browse/PHOENIX-1634.

Rajeshbabu - would be interesting to get your opinion on this too.

On Tue, Apr 7, 2015 at 1:19 PM, Perko, Ralph J <Ralph.Perko@pnnl.gov> wrote:

>  Based on the Phoenix compatibility chart at the download page I did not
> expect there to be issues with Phoenix 4.3 and Hbase 0.98.4.
>
> http://phoenix.apache.org/download.html
>
>
>
>
>
> *From:* Devaraj Das [mailto:ddas@hortonworks.com]
> *Sent:* Tuesday, April 07, 2015 12:58 PM
>
> *To:* user@phoenix.apache.org
> *Subject:* Re: hbase / phoenix errors
>
>
>
> What is the major driver to not use the HDP bundled Phoenix?
>
> It seems to me that the Phoenix version you have is not compatible with
> the underlying HBase version, leading to all these issues. In particular,
> the method getCatalogTracker in HDP-2.2 works only with 1 argument, but in
> Phoenix versions from the open source, it works with 0 arguments. This has
> been taken care of in the-yet-to-be-released HDP-2.3 (the HBase/Phoenix
> code both supports/uses the 0 argument getCatalogTracker).
>  ------------------------------
>
> *From:* Perko, Ralph J <Ralph.Perko@pnnl.gov>
> *Sent:* Tuesday, April 07, 2015 10:28 AM
> *To:* user@phoenix.apache.org
> *Subject:* RE: hbase / phoenix errors
>
>
>
> Thank you for the response
>
>
>
> I am using Phoenix 4.3 as a separate installation.
>
>
>
> Unfortunately I have no way to copy the actual log files so I will need to
> transcribe as much as I can.
>
>
>
> There are a lot of things going on – I’ll try to provide the highlights
>
>
>
> Right now:
>
> Using ambari – everything on the cluster is green – there are no apparent
> issues (but there are many)
>
>
>
> On the hbase master web site it shows a table split hung up (all red –
> “regions in transition”) since yesterday evening.
>
>
>
> All my phoenix tables are setup as follows:
>
> Salted
>
> 100GB hregion max file size
>
> Constant split size policy
>
>
>
> If I attempt to connect to Phoenix using sqlline it get the exception:
>
> NotServingRegionException:Region SYSTEM.CATALOG is not online
>
>
>
> If I run hbase shell I can list the tables but cannot scan any of them
>
>
>
> RS Log Messages:
>
> Aside from the messages I provided earlier some errors and exceptions have
> come up as well on the RS:
>
>
>
> In order I believe:
>
>
>
> ERROR StatsScanner failed to update stats table
>
> ERROR largeCompaction Compaction Failed
>
>
>
> ERROR largeCompaction Failed after attempt 350 – ConnectionRefused – this
> server is in the failed servers list
>
>
>
> Coprocessor.CoprocessorHost: the coprocessor …LocalIndexSplitter threw an
> exception
>
> NoSuchMethodError hbase.regionserver.RegionServerService.getCatalogTracker
>
>
>
> HRegion: compaction interrupted InterruptedOException
>
> RuntimeException: HRegionServer aborted
>
>
>
> Restart
>
>
>
> ERROR RS_LOG_REPLAY wal.HLogSplitter  OutOFMemory
>
>
>
> Restart
>
>
>
> Many of these: RemoteException (LeaseExpiredException) Holder:
> DFSCLient…recovered.edits…: File does not exist
>
>
>
> Many java.net.ConnectionException: Connection refused
>
>
>
> Java.net.ConnectionException SocketTimeoutException … row ‘’ on table
> ‘hbase.meta’
>
>
>
> This is where we are today
>
>
>
> I will provide whatever info you need
>
>
>
> Thanks!
>
> Ralph
>
>
>
>
>
>
>
> *From:* Nick Dimiduk [mailto:ndimiduk@gmail.com <ndimiduk@gmail.com>]
> *Sent:* Tuesday, April 07, 2015 9:05 AM
> *To:* user@phoenix.apache.org
> *Subject:* Re: hbase / phoenix errors
>
>
>
> Also, beside each region server log file (.log) there's also the output
> file (.out). Check the output files as well, as some serious crashes
> scenarios bypass the logs and go directly to the out files.
>
>
>
> -n
>
> On Tuesday, April 7, 2015, Devaraj Das <ddas@hortonworks.com> wrote:
>
> Hi Ralph, were you using the Phoenix bundled with HDP-2.2 or was that a
> separate installation? Could you please copy/paste some log lines around
> the time of a regionserver's crash (look for exceptions etc around that
> time in the regionserver logs).
>
> Thanks
>
> Devaraj
>
>
> On Apr 6, 2015, at 3:00 PM, Perko, Ralph J <Ralph.Perko@pnnl.gov> wrote:
>
>  Hi, we recently upgraded to Phoenix 4.3 and Hortonworks 2.2 (HBase
> .98.4) and we are running into some issues.  I am wondering if I am missing
> something easy and hoping you can help.  I have 34 regions servers and many
> keep crashing but without much in the way of error messages.
>
>
>
> Here are the things that stand out:
>
>
>
> ClientAsync.Process – waiting for some tasks to finish
>
> smallCompaction RPCRetryingCaller: Call exception …. ‘msg row
> ‘SOME_PHOENIX_TABLE_NAME_IDX:<some long key>’ on table: SYSTEM.STATS
> attempt 225/350
>
>
>
> Similar ones for largeCompaction as well.
>
>
>
> The other issue is the Pig loader hangs with these messages in the mapper
> logs:
>
> [phoenix-1-thread-0] RPCRetryingCaller: Call exception msg row ‘’ on table
> ‘SYSTEM.CATALOG’
>
>
>
> Eventually the mappers time out – no errors
>
>
>
> Regions servers come up and down.  There are lots of connection refused
> errors as well.
>
>
>
> Restarting hbase does not help.  The region servers will come up then go
> down again.
>
>
>
> Zookeeper is up.  I’ve restarted just in case but it did not help
>
>
>
> I cannot connect to Phoenix from the command line
>
>
>
> Any help is appreciated.
>
>
>
> Thanks!
>
> Ralph
>
>
>
>

Mime
View raw message