phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Wang <simon.w...@airbnb.com>
Subject Re: Index tables at scale
Date Sun, 10 Jul 2016 20:50:31 GMT
Hi James,

Thanks for the response.

In our use case, there is a 256 region table, and we want to build ~12 indexes on it. We have
15 region servers. If each index is in its own table, that would be a total of 221 regions
per region server of this single table. I think the extra write time cost is okay. But the
number of regions is too high for us.

Best,
Simon


> On Jul 9, 2016, at 1:18 AM, James Taylor <jamestaylor@apache.org> wrote:
> 
> Hi Simon,
> The reason we've taken this approach with views is that it's possible with multi-tenancy
that the number of views would grow unbounded since you might end up with a view per tenant
(100K or 1M views or more - clearly too many for HBase to handle as separate tables).
> 
> With secondary indexes directly on physical tables, you're somewhat bounded by the hit
you're willing to take on the write side, as the cost of maintaining the index is similar
to the cost of the write to the data table. So the extra number of physical tables for indexes
seems within the bounds of what HBase could handle. 
> 
> How many secondary indexes are you creating and are you ok with the extra write-time
cost?
> 
> From a code consistency standpoint, using the same approach across local, global, and
view indexes might simplify things, though. Please file a JIRA with a bit more detail on your
use case.
> 
> Thanks,
> James
> 
> 
> 
> On Fri, Jul 8, 2016 at 8:59 PM, Simon Wang <simon.wang@airbnb.com <mailto:simon.wang@airbnb.com>>
wrote:
> Hi all,
> 
> I am writing to ask if there is a way to let Phoenix store all indexes on a single table
in the same HBase table. If each index must be stored in a separate table, creating more than
a few indexes on table with a large number of regions will not scale well.
> 
> From what I have learned, when Phoenix builds indexes on a view, it stores all indexes
in a table associated with the underlying table of the view. e.g. if V1 is a view of T1, all
indexes on V1 will be stored in _IDX_T1. It would be great if this behavior can be optionally
turned on for indexes on tables.
> 
> Best,
> Simon
> 


Mime
View raw message