phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Phoenix Query Server query logging
Date Tue, 11 Apr 2017 23:05:53 GMT
FWIW, we use our metric framework[1] to collect that information,
outputting one log line per statement or query and then Splunk to look at
it.

[1] http://phoenix.apache.org/metrics.html

On Tue, Apr 11, 2017 at 3:24 PM Michael Young <yomaiquin@gmail.com> wrote:

> Yes, the tracing webapp is working, at least in our distro HDP 2.5 release.
>
> However, it seemed to negatively impact our performance and created a
> large volume of trace data which was somewhat overwhelming.
>
> We could not get simple SELECT query logging and  query time info from the
> trace logs it produces.  So it didn't seem appropriate to address the type
> of simple query logging we have in our use case.  I suppose it is more for
> detailed tracing use cases.
>
> Cheers,
> Michael
>
> On Mon, Apr 3, 2017 at 2:28 PM, Ryan Templeton <rtempleton@hortonworks.com
> > wrote:
>
> I see there’s a phoenix-tracing-webapp project in the build plus this on
> the website - https://phoenix.apache.org/tracing.html
>
> Is this project still working and usable? The project looks like it’s had
> updates as of a few months ago…
>
>
> Thanks,
> Ryan
>
> On 3/3/17, 10:33 AM, "Josh Elser" <elserj@apache.org> wrote:
>
> >
> https://github.com/apache/calcite/blob/master/avatica/server/src/main/java/org/apache/calcite/avatica/jdbc/JdbcMeta.java
> >
> >This is ultimately where the requests from the client using the thin
> >JDBC driver get executed inside of PQS. The API's aren't 100%, but it
> >should be obvious what is getting invoked with you're familiar with the
> >JDBC APIs.
> >
> >Michael Young wrote:
> >> Josh,
> >> I am interested in looking at adding this to Avatica myself, although
> >> I'm not familiar with that code base.
> >>
> >> Can you point me to where in the avatica code I should look at to add
> >> this logging?
> >>
> >> Cheers
> >>
> >>
> >> On Tue, Feb 28, 2017 at 4:15 AM, Josh Elser <elserj@apache.org
> >> <mailto:elserj@apache.org>> wrote:
> >>
> >>     No, I don't believe there is any log4j logging done in PQS that
> >>     would show queries being executed.
> >>
> >>     Ideally, we would have a "query log" in Phoenix which would present
> >>     an interface to this data and it wouldn't require anything special
> >>     in PQS. However, I wouldn't be opposed to some trivial additions to
> >>     PQS (Avatica, really) to add a simple logging as a stopgap.
> >>
> >>
> >>     On Feb 27, 2017 20:49, "Michael Young" <yomaiquin@gmail.com
> >>     <mailto:yomaiquin@gmail.com>> wrote:
> >>
> >>         I hadn't seen a reply to my earlier question.
> >>
> >>         We have business analysts running queries using BI tools (like
> >>         Tableau) which connect via the Phoenix Query Server.
> >>
> >>         How can we log all SELECT queries (raw query, start time, end
> >>         time, etc...)?
> >>
> >>         Any way to tweak log4j or other properties to get this?  The
> >>         TRACE logging I tried (mentioned in my post above) was way too
> >>         dense to be useful for reporting usage, and doesn't seem to show
> >>         the full SQL query params and query start/end times.  Also, it
> >>         logs every UPSERT during data load (which overwhelms the log
> >>         files).  We really just need SELECTS logged.
> >>
> >>
> >>
> >>         On Tue, Jan 31, 2017 at 5:10 PM, Michael Young
> >>         <yomaiquin@gmail.com <mailto:yomaiquin@gmail.com>> wrote:
> >>
> >>             Does the Phoenix Query Server have an option to log the SQL
> >>             statements which are executed?
> >>
> >>             We see there are ways to get various PQS trace logs
> >>             modifying the log4j settings used by the queryserver.py:
> >>
> >>             log4j.org.apache.phoenix.jdbc.PhoenixStatementFactory=TRACE
> >>             (or DEBUG)
> >>             log4j.org.apache.phoenix.jdbc.PhoenixStatement=TRACE
> >>             log4j.logger.org.apache.calcite.avatica=TRACE
> >>             log4j.logger.org.apache.phoenix.queryserver.server=TRACE
> >>             etc...
> >>
> >>             but the data in the trace logs (which show SQL statements)
> >>             are not particularly user friendly.  And it does not seem
> >>             straightforward to get to end-to-end query execution times.
> >>
> >>             Any suggestions how to get simple SQL logs (raw query,
> >>             execution time, ...)?  The idea is to monitor user activity
> >>             and take action if query times are slow, or timeout.
> >>
> >>             Thanks,
> >>             Michael
> >>
> >>
> >>
> >>
> >
>
>
>

Mime
View raw message