phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mujtaba Chohan <mujt...@apache.org>
Subject Re: Index tables at scale
Date Mon, 11 Jul 2016 21:17:15 GMT
12 index tables * 256 region per table = ~3K regions for index tables
assuming we are talking of covered index which implies 200+ regions/region
server on a 15 node cluster.

On Mon, Jul 11, 2016 at 1:58 PM, James Taylor <jamestaylor@apache.org>
wrote:

> Hi Simon,
>
> I might be missing something, but with 12 separate index tables or 1 index
> table, the amount of data will be the same. Won't there be the same number
> of regions either way?
>
> Thanks,
> James
>
> On Sun, Jul 10, 2016 at 10:50 PM, Simon Wang <simon.wang@airbnb.com>
> wrote:
>
>> Hi James,
>>
>> Thanks for the response.
>>
>> In our use case, there is a 256 region table, and we want to build ~12
>> indexes on it. We have 15 region servers. If each index is in its own
>> table, that would be a total of 221 regions per region server of this
>> single table. I think the extra write time cost is okay. But the number of
>> regions is too high for us.
>>
>> Best,
>> Simon
>>
>>
>> On Jul 9, 2016, at 1:18 AM, James Taylor <jamestaylor@apache.org> wrote:
>>
>> Hi Simon,
>> The reason we've taken this approach with views is that it's possible
>> with multi-tenancy that the number of views would grow unbounded since you
>> might end up with a view per tenant (100K or 1M views or more - clearly too
>> many for HBase to handle as separate tables).
>>
>> With secondary indexes directly on physical tables, you're somewhat
>> bounded by the hit you're willing to take on the write side, as the cost of
>> maintaining the index is similar to the cost of the write to the data
>> table. So the extra number of physical tables for indexes seems within the
>> bounds of what HBase could handle.
>>
>> How many secondary indexes are you creating and are you ok with the extra
>> write-time cost?
>>
>> From a code consistency standpoint, using the same approach across local,
>> global, and view indexes might simplify things, though. Please file a JIRA
>> with a bit more detail on your use case.
>>
>> Thanks,
>> James
>>
>>
>>
>> On Fri, Jul 8, 2016 at 8:59 PM, Simon Wang <simon.wang@airbnb.com> wrote:
>>
>>> Hi all,
>>>
>>> I am writing to ask if there is a way to let Phoenix store all indexes
>>> on a single table in the same HBase table. If each index must be stored in
>>> a separate table, creating more than a few indexes on table with a large
>>> number of regions will not scale well.
>>>
>>> From what I have learned, when Phoenix builds indexes on a view, it
>>> stores all indexes in a table associated with the underlying table of the
>>> view. e.g. if V1 is a view of T1, all indexes on V1 will be stored in
>>> _IDX_T1. It would be great if this behavior can be optionally turned on for
>>> indexes on tables.
>>>
>>> Best,
>>> Simon
>>
>>
>>
>>
>

Mime
View raw message