phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samarth Jain <sama...@apache.org>
Subject help diagnosing issue
Date Tue, 01 Sep 2015 22:29:25 GMT
Ralph,

Couple of questions.

Do you have phoenix stats enabled?

Can you send us a stacktrace of RegionTooBusy exception? Looking at HBase
code it is thrown in a few places. Would be good to check where the
resource crunch is occurring at.



On Tue, Sep 1, 2015 at 2:26 PM, Perko, Ralph J <Ralph.Perko@pnnl.gov> wrote:

> Hi I have run into an issue several times now and could really use some
> help diagnosing the problem.
>
> Environment:
> phoenix 4.4
> hbase 0.98
> 34 node cluster
> Tables are defined with 40 salt buckets
> We are continuously loading large, bz2, csv files into Phoenix via Pig.
> The data is in the hundred of TB’s per month
>
> The process runs well for a few weeks but as the regions split and the
> number of regions gets into the hundreds per table we begin to get
> “RegionTooBusy” exceptions around Phoenix write code when the Pig jobs run.
>
> Something else I have noticed is the number of requests on the regions
> becomes really unbalanced.  While the number of regions is around 40, 80,
> 120 the number of requests per region (via the hbase master site) is pretty
> well balanced.  But as the number gets into the 200’s many of the regions
> have 0 requests while the other regions have hundreds of millions of
> requests.
>
> If I drop the tables and start over the issue goes away.  But we are
> approaching a production deadline and this is no longer an option.
>
> The cluster is on a closed network so sending log files is not possible
> although I can send scanned images of logs and answer specific questions.
>
> Can you please help me diagnose this issue.
>
> Thanks!
> Ralph
>
>

Mime
View raw message