phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Perko, Ralph J" <Ralph.Pe...@pnnl.gov>
Subject help diagnosing issue
Date Tue, 01 Sep 2015 21:26:53 GMT
Hi I have run into an issue several times now and could really use some help diagnosing the
problem.

Environment:
phoenix 4.4
hbase 0.98
34 node cluster
Tables are defined with 40 salt buckets
We are continuously loading large, bz2, csv files into Phoenix via Pig.
The data is in the hundred of TB’s per month

The process runs well for a few weeks but as the regions split and the number of regions gets
into the hundreds per table we begin to get “RegionTooBusy” exceptions around Phoenix
write code when the Pig jobs run.

Something else I have noticed is the number of requests on the regions becomes really unbalanced.
 While the number of regions is around 40, 80, 120 the number of requests per region (via
the hbase master site) is pretty well balanced.  But as the number gets into the 200’s many
of the regions have 0 requests while the other regions have hundreds of millions of requests.

If I drop the tables and start over the issue goes away.  But we are approaching a production
deadline and this is no longer an option.

The cluster is on a closed network so sending log files is not possible although I can send
scanned images of logs and answer specific questions.

Can you please help me diagnose this issue.

Thanks!
Ralph


Mime
View raw message