phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marica Tan <marica....@gmail.com>
Subject Re: How to manually generate a salted row key?
Date Wed, 14 Sep 2016 07:33:55 GMT
Hi Josh,

It works without salting.  Tried the getSaltedKey but still the same
problem.

The generated row key when doing an upsert via phoenix sqlline, is
different and I can query the one created via phoenix.

via phoenix: \x09O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA

via hbase api: \x15O1IC\x00CUSTOMER\x0076s7DeAqt82_1LFY7brsTA


Btw, I modified the code a little:

def generateRowKey(nBuckets: Integer, compositeKeys: String*): Array[Byte] = {
  val keys = compositeKeys.tail.foldLeft(ArrayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a,
b) => {
    a += QueryConstants.SEPARATOR_BYTE_ARRAY
    a += convertToByteArray(b)
  })

  updateSaltingByte(keys, nBuckets)
}

def convertToByteArray(key: String): Array[Byte] = key match {
  case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)
  case _ => ByteUtil.EMPTY_BYTE_ARRAY
}

def updateSaltingByte(rowKey: ArrayBuffer[Array[Byte]], nBuckets:
Integer): Array[Byte] = {
  if (nBuckets > 0) {
    SaltingUtil.getSaltedKey(new
ImmutableBytesWritable(ByteUtil.concat(Array.fill[Byte](1)(0),
rowKey.toSeq: _*)), nBuckets);

  } else {
    ByteUtil.concat(rowKey.head, rowKey.tail.toSeq: _*)

  }
}


>From what I understand, phoenix will just combine those keys separated by a
zero byte. Am I doing it wrong?

I would greatly appreciate any help you can give me to solve this problem.


Thanks!
--
Marica

On Wed, Sep 14, 2016 at 8:15 AM, Marica Tan <marica.tan@gmail.com> wrote:

> Hi Josh,
>
> Haven't tried without salting yet. The problem we encountered is only on
> the firstKey, when we do select with a where condition on the secondKey or
> thirdKey, it returns the correct result. So I'm guessing it has something
> to do with the salted key.
>
> Will try the getSaltedKey first and hoping that it will fix the problem :)
>
> Thanks for the help!
>
> --
> Marica
>
> On 2016-09-13 21:50 (+0800), Josh Mahonin <j...@gmail.com> wrote:
> > Hi Marica,>
> >
> > Are you able to successfully write your rowkey without salting? If not,
> it>
> > could be that your 'generateRowKey' function is the culprit.>
> >
> > FWIW, we have some code that does something similar, though we use>
> > 'getSaltedKey':>
> >
> > // If salting, we need to prepend an empty byte to 'rowKey', then fill
> it>
> > if (saltBuckets > 0) {>
> > rowKey = ByteUtil.concat(new byte[1], rowKey);>
> > rowKey = SaltingUtil.getSaltedKey(new ImmutableBytesWritable(rowKey),>
> > saltBuckets);>
> > }>
> >
> > Good luck,>
> >
> > Josh>
>
> >
> > On Tue, Sep 13, 2016 at 1:59 AM, Marica Tan <marica.tan@gmail.com>
> wrote:>
> >
> > > Hi,>
> > >>
> > > We have a table created via phoenix with salt bucket, but we're using>
> > > HBase API to insert records since we need to manually set the HBase
> version>
> > > and I believe that it isn't possible via phoenix.>
> > >>
> > > Our table has a composite key (firstKey varchar, secondKey varchar,>
> > > thirdKey varchar) and when we do a select query with a where condition
> on>
> > > the firstKey, not all records are retrieved.>
> > >>
> > > We checked the value of the firstKey and it should return 10 records,
> but>
> > > we're only getting 7.>
> > >>
> > > If we do a where firstKey = 'someValue' we get 7>
> > > If we do a where firstKey like '%someValue' we get 10>
> > >>
> > > So we think the main culprit is the way we generate the row key.
> Here's>
> > > the code:>
> > >>
> > > def generateRowKey(nBuckets: Integer, compositeKeys: String*):>
> > > Array[Byte] = {>
> > >>
> > > val keys = compositeKeys.tail.foldLeft(Ar
> rayBuffer[Array[Byte]](convertToByteArray(compositeKeys.head)))((a, b) =>
> {>
> > > a += QueryConstants.SEPARATOR_BYTE_ARRAY>
> > > a += convertToByteArray(b)>
> > > })>
> > >>
> > > val rowKey = ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
> keys.toSeq: _*)>
> > >>
> > > updateSaltingByte(rowKey, nBuckets)>
> > >>
> > > rowKey>
> > > }>
> > >>
> > > def convertToByteArray(key: String): Array[Byte] = key match {>
> > > case x if StringUtils.isNotBlank(x) => Bytes.toBytes(x)>
> > > case _ => ByteUtil.EMPTY_BYTE_ARRAY>
> > > }>
> > >>
> > > def updateSaltingByte(rowKey: Array[Byte], nBuckets: Integer): Unit =
> {>
> > > if (nBuckets > 0) {>
> > > rowKey(0) = SaltingUtil.getSaltingByte(rowKey,>
> > > SaltingUtil.NUM_SALTING_BYTES, rowKey.length -
> SaltingUtil.NUM_SALTING_BYTES, nBuckets)>
> > > }>
> > > }>
> > >>
> > >>
> > > Btw, we're using phoenix 4.4>
> > >>
> > >>
> > > Thanks,>
> > > -->
> > > Marica Tan>
> > >>
> >
>

Mime
View raw message