phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Batyrshin Alexander <0x62...@gmail.com>
Subject Re: Concurrent phoenix queries throw unable to create new native thread error
Date Fri, 12 Oct 2018 07:44:05 GMT
Yes - phoenix.query.threadPoolSize is client side property.
And yes - I suggest to set it at hbase-site.xml and provide (via CLASSPATH) this config for
client Phoenix JDBC driver

Yep, you shoud provide hbase-site.xml to client-side fat JDBC driver

> On 11 Oct 2018, at 16:37, Hemal Parekh <hemal@bitscopic.com> wrote:
> 
> I have already set phoenix.query.threadPoolSize in hbase-site.xml which resides on server
side. I think phoenix.query.threadPoolSize is a client side property and so should be set
from the client application making phoenix connection. Are you suggesting that there should
also be a copy of hbase-site.xml on the client machine? 
> 
> On Thu, Oct 11, 2018 at 5:20 AM Batyrshin Alexander <0x62ash@gmail.com <mailto:0x62ash@gmail.com>>
wrote:
> Problem comes from that "phoenix.query.threadPoolSize" can not be set via connection
settings. You must set this props via hbase-site.xml.
> Looks like thread pool initialized before any connections inside JDBC driver.
> 
> I think that this moment must be clarified in documentation http://phoenix.apache.org/tuning.html
<http://phoenix.apache.org/tuning.html>
>  
> 
>> On 10 Oct 2018, at 18:59, Hemal Parekh <hemal@bitscopic.com <mailto:hemal@bitscopic.com>>
wrote:
>> 
>> We have an analytical application running concurrent phoenix queries against Hortonworks
HDP 2.6 cluster. Application uses phoenix JDBC connection to run queries. Often times, concurrent
queries fail with "java.lang.OutOfMemoryError: unable to create new native thread" error.
JDBC connection sets following phoenix properties. 
>> 
>> connectionProps.setProperty("phoenix.query.threadPoolSize", "2000")
>> connectionProps.setProperty("phoenix.query.querySize", "40000")
>> 
>> Phoenix version is 4.7 and Hbase version is 1.1.2, The HDP cluster has six regionservers
on six data nodes. Concurrent queries run against different phoenix tables, some are small
having few million records and some are big having few billions records. Most of the queries
do not have joins,  where clause includes conditions on rowkey and few nonkey columns. Queries
with joins (which are on small tables) have used USE_SORT_MERGE_JOIN hint. 
>> 
>> Are there other phoenix properties which need to be set on JDBC connection? Are above
values for phoenix.query.threadPoolSize and phoenix.query.querySize enough to handle concurrent
query use case? We have changed these two properties couple of times to increase their values
but the error still remains the same.
>> 
>> 
>> Thanks,
>> 
>> Hemal Parekh
>> 
>> 
>> 
> 
> 
> 
> -- 
> 
> Hemal Parekh
> Senior Data Warehouse Architect
> m. 240.449.4396
>  <http://bitscopic.com/>


Mime
View raw message