phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karavany, Ido" <ido.karav...@intel.com>
Subject Queries against Hbase Region not stopping & Crashing HBase
Date Sun, 23 Nov 2014 03:05:03 GMT
Hi All,

We're using CDH 5.0.1 and have tried use both Phoenix 4.0.0 and 4.2.0.
Both are working OK normally.

We're encountering a major issue when executing large queries (should return ~15M records).
Our table is large (~600M) and narrow (1-2 columns).
The query I'm currently referring to contains WHERE & ORDER BY clauses (no grouping)

Smaller (returns ~100-500K) similar queries against this table. 100K query takes ~1 second
We're executing the query using java code and using similar code to what exists in phoenix
FAQ page.

Our main purpose is not killing the region servers - we're good with timeouts for now.

What happens when executing the large queries

*         Phoenix Timeout (60 seconds) occurs - we catch it and closing the connection (not
in a connection pool)

*         For some reason load against HBase doesn't stop until we kill our application container
(runs on Tomcat)

*         While it is running (after the timeout and connection close) - region servers CPU
is high & they write a lot of files to /tmp folder until disk is getting full (write ~15GB
of files) and region servers are getting killed eventually

o    The execution doesn't end until region server is down.

*         The above happens even for 1 query of that size

we've tried to add the below to hbase-site.xml to all servers (hbase servers and client)
  <property>
    <name>phoenix.query.timeoutMs</name>
    <value>65000</value>
  </property>
  <property>
    <name>phoenix.query.maxSpoolToDiskBytes</name>
    <value>10240001</value>
  </property>


Can you please advise?

Thanks,
Ido

---------------------------------------------------------------------
Intel Israel (74) Limited

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Mime
View raw message