phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Soldatov <sergeysolda...@gmail.com>
Subject Re: Potential causes for very slow DELETEs?
Date Fri, 18 Aug 2017 20:15:26 GMT
Hi Pedro,

Usually that kind of behavior should be reflected in the region server
logs. Try to turn DEBUG level and check what exactly RS is doing during
that time. Also you may check the thread dump of RS during the execution
and see what are rpc handlers are doing. One thing that should be checked
first - the RPC handlers. If they are all busy you may consider to increase
the number of handlers. If you have RPC scheduler and controller
configured, double check that regular handlers are used, but not IndexRPC
(there was a bug that client is sending all rpc with index priority). If
you see it, remove controller factory property on client side.

Thanks,
Sergey

On Fri, Aug 18, 2017 at 4:46 AM, Pedro Boado <pedro.boado@gmail.com> wrote:

> Hi all,
>
> We have two HBase 1.0 clusters running the same process in parallel
> -effectively keeps the same data in both Phoenix tables-
>
> This process feeds data into Phoenix 4.5 via HFile and once the data is
> loaded a Spark process deletes a few thousand rows from the tables
> -secondary indexing is disabled in our installation- .
>
> After an HBase restart -no config changes involved-, one of the clusters
> have started running these deletes too slowly (the fast run is taking 5min
> and the slow one around 1h). And more worryingly while the process is
> running Phoenix queries are taking hundreds of seconds instead of being sub
> second (even opening sqlline is very slow).
>
> We've almost run out of ideas trying to find the cause of this behaviour.
> There are no evident GC pauses, CPU usage,  Hdfs IO is normal, Memory usage
> is normal, etc.
>
> As soon as the delete process finishes Phoenix goes back to normal
> behaviour.
>
> Does anybody have any ideas for potential causes of this behaviour?
>
> Many thanks!!
>
> Pedro.
>

Mime
View raw message