phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>
Subject Re: Re: PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Date Fri, 30 Jan 2015 02:02:36 GMT
Hi, James & Jan
Thanks for your detaily illustration for this. Now I can see the balance of tuning both client
and server configs. Actually the table being count queried is so special that stores far more

data than other tables. I did try to increase both configs of "phoenix.query.threadPoolSize"
and "phoenix.query.queueSize" following 2x rate, but that did not work as expected. Then I
increasing phoenix.query.queueSize significantly from default to more larger like 50000, 
I can get the query results at last. Maybe that is the cause as the specific table is so special.


I would investigate more deep for what James suggests about chunk size and do UPDATE 
STATISTICS for the table. 

Thanks so much,
Sun.





CertusNet 

From: James Taylor
Date: 2015-01-30 03:38
To: user
Subject: Re: PhoenixIOException : Task rejected from org.apache.phoenix.job.JobManager
Just to add to what Jan said, if work is being rejected by the thread
pool, you can increase your phoenix.query.queueSize. However, you
don't want it to be so high that the client machine will never be able
to complete the queued tasks. By throwing a rejected exception, the
client is being told "you're giving me too much work to do and I may
never finish it". So there's a balance here.
 
The other side of the equation is how big you want the work to be
chunked up. This corresponds to how much data will be scanned by each
thread. The bigger the amount of work, the less number of chunks of
work will be queued in the client thread pool. The downside of this is
that you decrease the level of parallelism, thus increasing the
latency you'll see to get query results.
 
The "chunk size" is controlled through the
phoenix.stats.guidepost.per.region or phoenix.stats.guidepost.width
config properties. You'd set one or the other. Setting
phoenix.stats.guidepost.per.region would give you a "relative" chunk
size depending on how the HBase table is configured. If you set it to
1, then you'd only get a single thread scanning to entire region of a
table. If you set it to 2, then you'd get at most two threads scanning
a region. Setting phoenix.stats.guidepost.width on the other hand
gives you an absolute number of bytes that each scan will perform.
Both these config parameters are server-side settings, so you'd need
to set them in the hbase-sites.xml on each region server, bounce your
cluster, and run UPDATE STATISTICS <table name> to have them take
affect.
 
Hope this helps. These parameters are documented here:
http://phoenix.apache.org/tuning.html and there's some more
information here: http://phoenix.apache.org/update_statistics.html
 
Thanks,
James
 
On Thu, Jan 29, 2015 at 9:01 AM, Jan Fernando <jfernando@salesforce.com> wrote:
> Hi Sun,
>
> I believe this means you have exhausted the thread pool and the queue that
> buffers the tasks to service Phoenix queries. Each query will have different
> characteristics in terms of the level of parallelism that Phoenix will use
> to service that query. For example some queries require few (or 1) tasks,
> such as a query with LIMIT 10,  other queries that hit many regions and
> guideposts require a higher number of tasks. As you use up the queue and get
> near the max some queries may run if they are enough slots left in the queue
> to service it. So queries requiring a low number of slots like a LIMIT 10
> query might run but others won't.
>
> Hope this helps.
>
> --Jan
>
> On Wed, Jan 28, 2015 at 6:44 PM, sunfl@certusnet.com.cn
> <sunfl@certusnet.com.cn> wrote:
>>
>> Hi, all
>>
>> I got strange exception hints when doing query like SELECT COUNT(*) FROM
>> SOME_TABLE; through sqlline and the info is
>>
>>  java.lang.RuntimeException:
>> org.apache.phoenix.exception.PhoenixIOException: Task
>> org.apache.phoenix.job.JobManager$JobFutureTask@f55dfff rejected from
>>
>>     org.apache.phoenix.job.JobManager$1@165e96e[Running, pool size = 64,
>> active threads = 64, queued tasks = 5000, completed tasks = 5162]
>>
>> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
>> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
>> at sqlline.SqlLine.print(SqlLine.java:1735)
>> at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
>> at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
>> at sqlline.SqlLine.dispatch(SqlLine.java:821)
>> at sqlline.SqlLine.begin(SqlLine.java:699)
>> at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
>> at sqlline.SqlLine.main(SqlLine.java:424)
>>
>>
>>   However, such query like SELECT * FROM SOME_TABLE LIMIT 10 works. Also,
>> other table count query works too. Don't know what causes this kind of
>> issue. Any
>> available hints are appreciated.
>>
>> Thanks,
>> Sun.
>>
>>
>>
>>
>> ________________________________
>> ________________________________
>>
>> CertusNet
>>
>>
>
 
 
Mime
View raw message