phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ankit Singhal <ankitsingha...@gmail.com>
Subject Re: Performance degradation on query analysis
Date Thu, 19 Sep 2019 20:50:57 GMT
Please schedule compaction on SYSTEM.STATS table to clear the old entries.

On Thu, Sep 19, 2019 at 1:48 PM Stepan Migunov <
Stepan.Migunov@firstlinesoftware.com> wrote:

> Thanks, Josh. The problem was really related to reading the SYSTEM.STATS
> table.
> There were only 8,000 rows in the table, but COUNT took more than 10
> minutes. I noticed that the storage files (34) had a total size of 10 GB.
>
> DELETE FROM SYSTEM.STATS did not help - the storage files are still 10 GB,
> and COUNT took a long time.
> Then I truncated the table from the hbase shell. And this fixed the
> problem - after UPDATE STATS for each table, everything works fine.
>
> Are there any known issues with SYSTEM.STATS table? Apache Phoenix 4.13.1
> with 15 Region Servers.
>
> -----Original Message-----
> From: Josh Elser [mailto:elserj@apache.org]
> Sent: Tuesday, September 17, 2019 5:16 PM
> To: user@phoenix.apache.org
> Subject: Re: Performance degradation on query analysis
>
> Can you share the output you see from the EXPLAIN? Does it differ between
> times it's "fast" and times it's "slow"?
>
> Sharing the table(s) DDL statements would also help, along with the shape
> and version of your cluster (e.g. Apache Phoenix 4.14.2 with 8
> RegionServers).
>
> Spit-balling ideas:
>
> Could be reads over the SYSTEM.CATALOG table or the SYSTEM.STATS table.
>
> Have you looked more coarsely at the RegionServer logs/metrics? Any obvious
> saturation issues (e.g. handlers consumed, JVM GC pauses, host CPU
> saturation)?
>
> Turn on DEBUG log4j client side (beware of chatty ZK logging) and see if
> there's something obvious from when the EXPLAIN is slow.
>
> On 9/17/19 3:58 AM, Stepan Migunov wrote:
> > Hi
> > We have an issue with our production environment - from time to time we
> > notice a significant performance degradation for some queries. The
> strange
> > thing is that the EXPLAIN operator for these queries takes the same time
> > as queries execution (5 minutes or more). So, I guess, the issue is
> > related to query's analysis but not data extraction. Is it possible that
> > issue is related to SYSTEM.STATS access problem? Any other ideas?
> >
>

Mime
View raw message