phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <>
Subject Re: Global Indexes and impact on availability
Date Mon, 05 Dec 2016 22:39:56 GMT
Have you tried local indexes?

On Mon, Dec 5, 2016 at 2:35 PM Neelesh <> wrote:

> Hello,
>   When a region server is under stress (hotspotting, or large replication,
> call queue sizes hitting the limit, other processes competing with HBase
> etc), we experience latency spikes for all regions hosted by that region
> server.  This is somewhat expected in the plain HBase world.
> However, with a phoenix global index, this service deterioration seems to
> propagate to a lot more region servers, since the affected RS hosts some
> index regions. The actual data regions are on another RS and latencies on
> that RS spike because it cannot complete the index update calls quickly.
> And that second RS now causes issues on yet another one and so on.
> We've seen this happen on our cluster, and how we deal with this is by
> "fixing" the original RS - split regions/restart/move around regions,
> depending on what the problem is.
> Has any one experienced this issue? It feels like antithetical behavior
> for a distributed system. Cluster breaking down for the the very reasons
> its supposed to protect against.
> Love to hear the thoughts of Phoenix community on this

View raw message