phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Samarth Jain <>
Subject help diagnosing issue
Date Tue, 01 Sep 2015 22:29:25 GMT

Couple of questions.

Do you have phoenix stats enabled?

Can you send us a stacktrace of RegionTooBusy exception? Looking at HBase
code it is thrown in a few places. Would be good to check where the
resource crunch is occurring at.

On Tue, Sep 1, 2015 at 2:26 PM, Perko, Ralph J <> wrote:

> Hi I have run into an issue several times now and could really use some
> help diagnosing the problem.
> Environment:
> phoenix 4.4
> hbase 0.98
> 34 node cluster
> Tables are defined with 40 salt buckets
> We are continuously loading large, bz2, csv files into Phoenix via Pig.
> The data is in the hundred of TB’s per month
> The process runs well for a few weeks but as the regions split and the
> number of regions gets into the hundreds per table we begin to get
> “RegionTooBusy” exceptions around Phoenix write code when the Pig jobs run.
> Something else I have noticed is the number of requests on the regions
> becomes really unbalanced.  While the number of regions is around 40, 80,
> 120 the number of requests per region (via the hbase master site) is pretty
> well balanced.  But as the number gets into the 200’s many of the regions
> have 0 requests while the other regions have hundreds of millions of
> requests.
> If I drop the tables and start over the issue goes away.  But we are
> approaching a production deadline and this is no longer an option.
> The cluster is on a closed network so sending log files is not possible
> although I can send scanned images of logs and answer specific questions.
> Can you please help me diagnose this issue.
> Thanks!
> Ralph

View raw message