phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Index tables at scale
Date Mon, 11 Jul 2016 20:58:52 GMT
Hi Simon,

I might be missing something, but with 12 separate index tables or 1 index
table, the amount of data will be the same. Won't there be the same number
of regions either way?

Thanks,
James

On Sun, Jul 10, 2016 at 10:50 PM, Simon Wang <simon.wang@airbnb.com> wrote:

> Hi James,
>
> Thanks for the response.
>
> In our use case, there is a 256 region table, and we want to build ~12
> indexes on it. We have 15 region servers. If each index is in its own
> table, that would be a total of 221 regions per region server of this
> single table. I think the extra write time cost is okay. But the number of
> regions is too high for us.
>
> Best,
> Simon
>
>
> On Jul 9, 2016, at 1:18 AM, James Taylor <jamestaylor@apache.org> wrote:
>
> Hi Simon,
> The reason we've taken this approach with views is that it's possible with
> multi-tenancy that the number of views would grow unbounded since you might
> end up with a view per tenant (100K or 1M views or more - clearly too many
> for HBase to handle as separate tables).
>
> With secondary indexes directly on physical tables, you're somewhat
> bounded by the hit you're willing to take on the write side, as the cost of
> maintaining the index is similar to the cost of the write to the data
> table. So the extra number of physical tables for indexes seems within the
> bounds of what HBase could handle.
>
> How many secondary indexes are you creating and are you ok with the extra
> write-time cost?
>
> From a code consistency standpoint, using the same approach across local,
> global, and view indexes might simplify things, though. Please file a JIRA
> with a bit more detail on your use case.
>
> Thanks,
> James
>
>
>
> On Fri, Jul 8, 2016 at 8:59 PM, Simon Wang <simon.wang@airbnb.com> wrote:
>
>> Hi all,
>>
>> I am writing to ask if there is a way to let Phoenix store all indexes on
>> a single table in the same HBase table. If each index must be stored in a
>> separate table, creating more than a few indexes on table with a large
>> number of regions will not scale well.
>>
>> From what I have learned, when Phoenix builds indexes on a view, it
>> stores all indexes in a table associated with the underlying table of the
>> view. e.g. if V1 is a view of T1, all indexes on V1 will be stored in
>> _IDX_T1. It would be great if this behavior can be optionally turned on for
>> indexes on tables.
>>
>> Best,
>> Simon
>
>
>
>

Mime
View raw message