phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Taylor <jamestay...@apache.org>
Subject Re: Excessive region splitting of Global Index table(8 Megabyte regions)
Date Sat, 12 Mar 2016 20:26:54 GMT
Ok - before you reset the MAX_FILESIZE, it'd be help if you could open an
HBase shell and let us know what the current values are for your data table
and index table:

describe YOUR_DATA_TABLE;
describe YOUR_INDEX_TABLE;

If your data table is 8GB, I'd guess your index should be 4GB at the
smallest. I think 1GB would be too low.

Thanks,
James

On Sat, Mar 12, 2016 at 12:23 PM, anil gupta <anilgupta84@gmail.com> wrote:

> Thanks for the reply, James. We have 2 global secondary index in this
> table and both of them exhibit same behavior.  Going to give your
> suggestion a try. I also think that regionsize for secondary index should
> not be 8GB. Will try to set the regionsize=1GB for secondary index and see
> how it goes.
>
> On Sat, Mar 12, 2016 at 12:00 PM, James Taylor <jamestaylor@apache.org>
> wrote:
>
>> Hi Anil,
>> Phoenix estimates the ratio between the data table and index table as
>> shown below to attempt to get the same number of splits in your index table
>> as your data table.
>>
>> /*
>>  * Approximate ratio between index table size and data table size:
>>  * More or less equal to the ratio between the number of key value
>>  * columns in each. We add one to the key value column count to
>>  * take into account our empty key value. We add 1/4 for any key
>>  * value data table column that was moved into the index table row key.
>>  */
>>
>> Phoenix then multiples the MAX_FILESIZE of the data table to come up with
>> a reasonable default value for the index table. Can you check in the HBase
>> shell what the MAX_FILESIZE is for the data table versus the index table?
>> Maybe there's a bug in Phoenix in how it calculates this ration.
>>
>> You can override the MAX_FILESIZE for your index through an ALTER TABLE
>> statement:
>>
>> ALTER TABLE my_table_schema.my_index_name SET MAX_FILESIZE=8589934592
>>
>> You can ignore the warnings you get in sqlline and you can verify the
>> setting took affect through the HBase shell by running the following
>> command:
>>
>> describe 'MY_TABLE_SCHEMA.MY_INDEX_NAME'
>>
>> HTH,
>>
>>     James
>>
>>
>> On Sat, Mar 12, 2016 at 10:18 AM, anil gupta <anilgupta84@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We are using HDP2.3.4 and Phoenix4.4.
>>> Our global index table is doing excessive splitting. Our cluster region
>>> size setting is 8 Gigabytes but global index table has 18 regions and max
>>> size of region is 10.9 MB.
>>> This is definitely not a good behavior. I looked into tuning (
>>> https://phoenix.apache.org/tuning.html) and i could not find anything
>>> relevant. Is this region splitting intentionally done by Phoenix for
>>> secondary index tables?
>>>
>>> Here is the output of du command:
>>> [ag@hdpclient1 ~]$ hadoop fs -du -h
>>> /apps/hbase/data/data/default/SEC_INDEX
>>> 761     /apps/hbase/data/data/default/SEC_INDEX/.tabledesc
>>> 0       /apps/hbase/data/data/default/SEC_INDEX/.tmp
>>> 9.3 M
>>> /apps/hbase/data/data/default/SEC_INDEX/079db2c953c30a8270ecbd52582e81ff
>>> 2.9 M
>>> /apps/hbase/data/data/default/SEC_INDEX/0952c070234c05888bfc2a01645e9e88
>>> 10.9 M
>>> /apps/hbase/data/data/default/SEC_INDEX/0d69bbb8991b868f0437b624410e9bed
>>> 8.2 M
>>> /apps/hbase/data/data/default/SEC_INDEX/206562491fd1de9db48cf422dd8c2059
>>> 7.9 M
>>> /apps/hbase/data/data/default/SEC_INDEX/25318837ab8e1db6922f5081c840d2e7
>>> 9.5 M
>>> /apps/hbase/data/data/default/SEC_INDEX/5369e0d6526b3d2cdab9937cb320ccb3
>>> 9.6 M
>>> /apps/hbase/data/data/default/SEC_INDEX/62704ee3c9418f0cd48210a747e1f8ac
>>> 7.8 M
>>> /apps/hbase/data/data/default/SEC_INDEX/631376fc5515d7785b2bcfc8a1f64223
>>> 2.8 M
>>> /apps/hbase/data/data/default/SEC_INDEX/6648d5396ba7a3c3bf884e5e1300eb0e
>>> 9.4 M
>>> /apps/hbase/data/data/default/SEC_INDEX/6e6e133580aea9a19a6b3ea643735072
>>> 8.1 M
>>> /apps/hbase/data/data/default/SEC_INDEX/8535a5c8a0989dcdfad2b1e9e9f3e18c
>>> 7.8 M
>>> /apps/hbase/data/data/default/SEC_INDEX/8ffa32e0c6357c2a0b413f3896208439
>>> 9.3 M
>>> /apps/hbase/data/data/default/SEC_INDEX/c27e2809cd352e3b06c0f11d3e7278c6
>>> 8.0 M
>>> /apps/hbase/data/data/default/SEC_INDEX/c4f5a98ce6452a6b5d052964cc70595a
>>> 8.1 M
>>> /apps/hbase/data/data/default/SEC_INDEX/c578d3190363c32032b4d92c8d307215
>>> 7.9 M
>>> /apps/hbase/data/data/default/SEC_INDEX/d750860bac8aa372eb28aaf055ea63e7
>>> 9.6 M
>>> /apps/hbase/data/data/default/SEC_INDEX/e9756aa4c7c8b9bfcd0857b43ad5bfbe
>>> 8.0 M
>>> /apps/hbase/data/data/default/SEC_INDEX/ebaae6c152e82c9b74c473babaf644dd
>>>
>>> --
>>> Thanks & Regards,
>>> Anil Gupta
>>>
>>
>>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>

Mime
View raw message