phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajeshbabu Chintaguntla <chrajeshbab...@gmail.com>
Subject Re: hbase / phoenix errors
Date Wed, 08 Apr 2015 17:06:13 GMT
The local indexing works fine  with 0.98.4.

Coprocessor.CoprocessorHost: the coprocessor …LocalIndexSplitter threw an
exception
NoSuchMethodError hbase.regionserver.RegionServerService.getCatalogTracker

This is specific to HDP I think. Any way I will try to remove
getCatalogTracker in local indexing.

Thanks,
Rajeshbabu.

On Wed, Apr 8, 2015 at 10:30 PM, Perko, Ralph J <Ralph.Perko@pnnl.gov>
wrote:

>  Devaraj,
>
>  The major driver is better subquery and join support.  I have been using
> 4.2.3 (pulled from source) to take advantage of a fix made to #1533 (see
> user group email dated on 12/18/14 between Maryann and myself).  The
> performance increase was significant.
>
>  Ralph
>
>  __________________________________________________
> *Ralph Perko*
> Pacific Northwest National Laboratory
> (509) 375-2272
> ralph.perko@pnnl.gov
>
>   From: Devaraj Das <ddas@hortonworks.com>
> Reply-To: "user@phoenix.apache.org" <user@phoenix.apache.org>
> Date: Tuesday, April 7, 2015 at 12:57 PM
> To: "user@phoenix.apache.org" <user@phoenix.apache.org>
>
> Subject: Re: hbase / phoenix errors
>
>   What is the major driver to not use the HDP bundled Phoenix?
>
> It seems to me that the Phoenix version you have is not compatible with
> the underlying HBase version, leading to all these issues. In particular,
> the method getCatalogTracker in HDP-2.2 works only with 1 argument, but in
> Phoenix versions from the open source, it works with 0 arguments. This has
> been taken care of in the-yet-to-be-released HDP-2.3 (the HBase/Phoenix
> code both supports/uses the 0 argument getCatalogTracker).
>  ------------------------------
> *From:* Perko, Ralph J <Ralph.Perko@pnnl.gov>
> *Sent:* Tuesday, April 07, 2015 10:28 AM
> *To:* user@phoenix.apache.org
> *Subject:* RE: hbase / phoenix errors
>
>
> Thank you for the response
>
>
>
> I am using Phoenix 4.3 as a separate installation.
>
>
>
> Unfortunately I have no way to copy the actual log files so I will need to
> transcribe as much as I can.
>
>
>
> There are a lot of things going on – I’ll try to provide the highlights
>
>
>
> Right now:
>
> Using ambari – everything on the cluster is green – there are no apparent
> issues (but there are many)
>
>
>
> On the hbase master web site it shows a table split hung up (all red –
> “regions in transition”) since yesterday evening.
>
>
>
> All my phoenix tables are setup as follows:
>
> Salted
>
> 100GB hregion max file size
>
> Constant split size policy
>
>
>
> If I attempt to connect to Phoenix using sqlline it get the exception:
>
> NotServingRegionException:Region SYSTEM.CATALOG is not online
>
>
>
> If I run hbase shell I can list the tables but cannot scan any of them
>
>
>
> RS Log Messages:
>
> Aside from the messages I provided earlier some errors and exceptions have
> come up as well on the RS:
>
>
>
> In order I believe:
>
>
>
> ERROR StatsScanner failed to update stats table
>
> ERROR largeCompaction Compaction Failed
>
>
>
> ERROR largeCompaction Failed after attempt 350 – ConnectionRefused – this
> server is in the failed servers list
>
>
>
> Coprocessor.CoprocessorHost: the coprocessor …LocalIndexSplitter threw an
> exception
>
> NoSuchMethodError hbase.regionserver.RegionServerService.getCatalogTracker
>
>
>
> HRegion: compaction interrupted InterruptedOException
>
> RuntimeException: HRegionServer aborted
>
>
>
> Restart
>
>
>
> ERROR RS_LOG_REPLAY wal.HLogSplitter  OutOFMemory
>
>
>
> Restart
>
>
>
> Many of these: RemoteException (LeaseExpiredException) Holder:
> DFSCLient…recovered.edits…: File does not exist
>
>
>
> Many java.net.ConnectionException: Connection refused
>
>
>
> Java.net.ConnectionException SocketTimeoutException … row ‘’ on table
> ‘hbase.meta’
>
>
>
> This is where we are today
>
>
>
> I will provide whatever info you need
>
>
>
> Thanks!
>
> Ralph
>
>
>
>
>
>
>
> *From:* Nick Dimiduk [mailto:ndimiduk@gmail.com <ndimiduk@gmail.com>]
> *Sent:* Tuesday, April 07, 2015 9:05 AM
> *To:* user@phoenix.apache.org
> *Subject:* Re: hbase / phoenix errors
>
>
>
> Also, beside each region server log file (.log) there's also the output
> file (.out). Check the output files as well, as some serious crashes
> scenarios bypass the logs and go directly to the out files.
>
>
>
> -n
>
> On Tuesday, April 7, 2015, Devaraj Das <ddas@hortonworks.com> wrote:
>
> Hi Ralph, were you using the Phoenix bundled with HDP-2.2 or was that a
> separate installation? Could you please copy/paste some log lines around
> the time of a regionserver's crash (look for exceptions etc around that
> time in the regionserver logs).
>
> Thanks
>
> Devaraj
>
>
> On Apr 6, 2015, at 3:00 PM, Perko, Ralph J <Ralph.Perko@pnnl.gov> wrote:
>
>  Hi, we recently upgraded to Phoenix 4.3 and Hortonworks 2.2 (HBase
> .98.4) and we are running into some issues.  I am wondering if I am missing
> something easy and hoping you can help.  I have 34 regions servers and many
> keep crashing but without much in the way of error messages.
>
>
>
> Here are the things that stand out:
>
>
>
> ClientAsync.Process – waiting for some tasks to finish
>
> smallCompaction RPCRetryingCaller: Call exception …. ‘msg row
> ‘SOME_PHOENIX_TABLE_NAME_IDX:<some long key>’ on table: SYSTEM.STATS
> attempt 225/350
>
>
>
> Similar ones for largeCompaction as well.
>
>
>
> The other issue is the Pig loader hangs with these messages in the mapper
> logs:
>
> [phoenix-1-thread-0] RPCRetryingCaller: Call exception msg row ‘’ on table
> ‘SYSTEM.CATALOG’
>
>
>
> Eventually the mappers time out – no errors
>
>
>
> Regions servers come up and down.  There are lots of connection refused
> errors as well.
>
>
>
> Restarting hbase does not help.  The region servers will come up then go
> down again.
>
>
>
> Zookeeper is up.  I’ve restarted just in case but it did not help
>
>
>
> I cannot connect to Phoenix from the command line
>
>
>
> Any help is appreciated.
>
>
>
> Thanks!
>
> Ralph
>
>
>
>

Mime
View raw message