I integrated one of the online services in my company with hbase using apache phoenix, after loading few millions of records I noticed that we have hotspot problem. All the records are going to one region as the keys are generated using sequence.
Usecase is: each user has 1000’s of records with combination of userid and second record id as rowkey (primary key uid, XXX). When user logs in we fetch all records by using userid and render the results to user. But updates will always be with combination (userid + XXX). Below are my questions.
1. If I salt the table using apache phoenix, is there any performance impact on reads as the reads has to query all regions?
2. If I have to salt the table, how many buckets should I use for 8 regional servers with 272 regions, roughly 33 regions for a regions server?
3. If I salt the table using phoenix, what is the effort to move away from pehonix and use the hbase client directly in later times ( not that I want to but just checking the options)
Thanks for your help.
Principal Software Engineer