phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Queries against Hbase Region not stopping & Crashing HBase
Date Tue, 25 Nov 2014 02:03:04 GMT
Hi Ido,
ORDER BY is an expensive operation. Phoenix essentially needs to
re-write the table (the portion that you're selecting) on the RS using
memory-mapped files (likely the cause of the /tmp files you're seeing)
over which a merge sort will be performed.

I assume that the ORDER BY is the exception, not the norm, as
hopefully your current primary key order is used for the bulk of your
queries? An alternative to doing an ORDER BY is to create a secondary
index with a PK that matches your ORDER BY expressions. If those
expressions are simple column references, then you can do that today.
If they're more complex, you'd need to wait for PHOENIX-514 (which is
being actively worked on). Note also that Phoenix will automatically
do a reverse scan if your query is traversing in reverse order which
will help those cases.

But regardless of this, running these queries should not kill the
region server. The 4.2.1 patch release would be a good one to try -
there are a number of important bug fixes there that aren't in the
4.2.0 release. The primary advantage that 4.2.1 will have over 4.1 or
4.0 releases is that it "chunks" up the work based on a configurable
number of bytes (phoenix.stats.guidepost.width server-side config
param - see http://phoenix.apache.org/update_statistics.html for more
information). This should help your ORDER BY case if a large query
times-out, as the individual chunks that have not been started yet
will be canceled if the query times out or the connection is closed.
In releases prior to 4.2, for bigger queries, an entire regions worth
of data would be reordered instead which could take a long time and is
not cancelable - once a scan is started in HBase it cannot be
canceled.

One other bug fix that may help cause your queries to timeout as
expected is PHOENIX-1463. This is committed in the 4.2 branch, but not
in the 4.2.1 release - it'll appear in our soon-to-be 4.2.2 release.

Please let us know how it goes - it'd be good to get a high level
overview of your use case if possible.

Thanks,
James

On Sat, Nov 22, 2014 at 7:05 PM, Karavany, Ido <ido.karavany@intel.com> wrote:
> Hi All,
>
>
>
> We're using CDH 5.0.1 and have tried use both Phoenix 4.0.0 and 4.2.0.
>
> Both are working OK normally.
>
>
>
> We're encountering a major issue when executing large queries (should return
> ~15M records).
>
> Our table is large (~600M) and narrow (1-2 columns).
>
> The query I'm currently referring to contains WHERE & ORDER BY clauses (no
> grouping)
>
>
>
> Smaller (returns ~100-500K) similar queries against this table. 100K query
> takes ~1 second
>
> We're executing the query using java code and using similar code to what
> exists in phoenix FAQ page.
>
>
>
> Our main purpose is not killing the region servers - we're good with
> timeouts for now.
>
>
>
> What happens when executing the large queries
>
> ·         Phoenix Timeout (60 seconds) occurs - we catch it and closing the
> connection (not in a connection pool)
>
> ·         For some reason load against HBase doesn't stop until we kill our
> application container (runs on Tomcat)
>
> ·         While it is running (after the timeout and connection close) -
> region servers CPU is high & they write a lot of files to /tmp folder until
> disk is getting full (write ~15GB of files) and region servers are getting
> killed eventually
>
> o    The execution doesn't end until region server is down.
>
> ·         The above happens even for 1 query of that size
>
>
>
> we've tried to add the below to hbase-site.xml to all servers (hbase servers
> and client)
>
>   <property>
>
>     <name>phoenix.query.timeoutMs</name>
>
>     <value>65000</value>
>
>   </property>
>
>   <property>
>
>     <name>phoenix.query.maxSpoolToDiskBytes</name>
>
>     <value>10240001</value>
>
>   </property>
>
>
>
>
>
> Can you please advise?
>
>
>
> Thanks,
>
> Ido
>
>
>
> ---------------------------------------------------------------------
> Intel Israel (74) Limited
>
> This e-mail and any attachments may contain confidential material for
> the sole use of the intended recipient(s). Any review or distribution
> by others is strictly prohibited. If you are not the intended
> recipient, please contact the sender and delete all copies.

Mime
View raw message