phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marica Tan <marica....@gmail.com>
Subject Re: How to manually generate a salted row key?
Date Wed, 14 Sep 2016 07:50:13 GMT
Hi Josh,

It turns out the salt bucket in the create table script is different than
what we're using in the generation of row key :D

Anyway thanks for the help :)


Marica

On Wed, Sep 14, 2016 at 3:33 PM, Marica Tan <marica.tan@gmail.com> wrote:

> Hi Josh,
>
> It works without salting.  Tried the getSaltedKey but still the same
> problem.
>
> The generated row key when doing an upsert via phoenix sqlline, is
> different and I can query the one created via phoenix.
>
> via phoenix: \x09O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA
>
> via hbase api: \x15O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA
>
>
> Btw, I modified the code a little:
>
> def generateRowKey(nBuckets: Integer, compositeKeys: String*): Array[Byte] = {
>   val keys = compositeKeys.tail.foldLeft(ArrayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a,
b) => {
>     a += QueryConstants.SEPARATOR_BYTE_ARRAY
>     a += convertToByteArray(b)
>   })
>
>   updateSaltingByte(keys, nBuckets)
> }
>
> def convertToByteArray(key: String): Array[Byte] = key match {
>   case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)
>   case _ => ByteUtil.EMPTY_BYTE_ARRAY
> }
>
> def updateSaltingByte(rowKey: ArrayBuffer[Array[Byte]], nBuckets: Integer): Array[Byte]
= {
>   if (nBuckets > 0) {
>     SaltingUtil.getSaltedKey(new ImmutableBytesWritable(ByteUtil.concat(Array.fill[Byte](1)(0),
rowKey.toSeq: _*)), nBuckets);
>
>   } else {
>     ByteUtil.concat(rowKey.head, rowKey.tail.toSeq: _*)
>
>   }
> }
>
>
> From what I understand, phoenix will just combine those keys separated by
> a zero byte. Am I doing it wrong?
>
> I would greatly appreciate any help you can give me to solve this problem.
>
>
> Thanks!
> --
> Marica
>
> On Wed, Sep 14, 2016 at 8:15 AM, Marica Tan <marica.tan@gmail.com> wrote:
>
>> Hi Josh,
>>
>> Haven't tried without salting yet. The problem we encountered is only on
>> the firstKey, when we do select with a where condition on the secondKey or
>> thirdKey, it returns the correct result. So I'm guessing it has something
>> to do with the salted key.
>>
>> Will try the getSaltedKey first and hoping that it will fix the problem :)
>>
>> Thanks for the help!
>>
>> --
>> Marica
>>
>> On 2016-09-13 21:50 (+0800), Josh Mahonin <j...@gmail.com> wrote:
>> > Hi Marica,>
>> >
>> > Are you able to successfully write your rowkey without salting? If not,
>> it>
>> > could be that your 'generateRowKey' function is the culprit.>
>> >
>> > FWIW, we have some code that does something similar, though we use>
>> > 'getSaltedKey':>
>> >
>> > // If salting, we need to prepend an empty byte to 'rowKey', then fill
>> it>
>> > if (saltBuckets > 0) {>
>> > rowKey = ByteUtil.concat(new byte[1], rowKey);>
>> > rowKey = SaltingUtil.getSaltedKey(new ImmutableBytesWritable(rowKey),>
>> > saltBuckets);>
>> > }>
>> >
>> > Good luck,>
>> >
>> > Josh>
>>
>> >
>> > On Tue, Sep 13, 2016 at 1:59 AM, Marica Tan <marica.tan@gmail.com>
>> wrote:>
>> >
>> > > Hi,>
>> > >>
>> > > We have a table created via phoenix with salt bucket, but we're using>
>> > > HBase API to insert records since we need to manually set the HBase
>> version>
>> > > and I believe that it isn't possible via phoenix.>
>> > >>
>> > > Our table has a composite key (firstKey varchar, secondKey varchar,>
>> > > thirdKey varchar) and when we do a select query with a where
>> condition on>
>> > > the firstKey, not all records are retrieved.>
>> > >>
>> > > We checked the value of the firstKey and it should return 10 records,
>> but>
>> > > we're only getting 7.>
>> > >>
>> > > If we do a where firstKey = 'someValue' we get 7>
>> > > If we do a where firstKey like '%someValue' we get 10>
>> > >>
>> > > So we think the main culprit is the way we generate the row key.
>> Here's>
>> > > the code:>
>> > >>
>> > > def generateRowKey(nBuckets: Integer, compositeKeys: String*):>
>> > > Array[Byte] = {>
>> > >>
>> > > val keys = compositeKeys.tail.foldLeft(Ar
>> rayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a, b)
>> => {>
>> > > a += QueryConstants.SEPARATOR_BYTE_ARRAY>
>> > > a += convertToByteArray(b)>
>> > > })>
>> > >>
>> > > val rowKey = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
>> keys.toSeq: _*)>
>> > >>
>> > > updateSaltingByte(rowKey, nBuckets)>
>> > >>
>> > > rowKey>
>> > > }>
>> > >>
>> > > def convertToByteArray(key: String): Array[Byte] = key match {>
>> > > case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)>
>> > > case _ => ByteUtil.EMPTY_BYTE_ARRAY>
>> > > }>
>> > >>
>> > > def updateSaltingByte(rowKey: Array[Byte], nBuckets: Integer): Unit =
>> {>
>> > > if (nBuckets > 0) {>
>> > > rowKey(0) = SaltingUtil.getSaltingByte(rowKey,>
>> > > SaltingUtil.NUM_SALTING_BYTES, rowKey.length -
>> SaltingUtil.NUM_SALTING_BYTES, nBuckets)>
>> > > }>
>> > > }>
>> > >>
>> > >>
>> > > Btw, we're using phoenix 4.4>
>> > >>
>> > >>
>> > > Thanks,>
>> > > -->
>> > > Marica Tan>
>> > >>
>> >
>>
>
>

Mime
View raw message