From user-return-8496-apmail-phoenix-user-archive=phoenix.apache.org@phoenix.apache.org Tue Jan 8 06:29:30 2019 Return-Path: X-Original-To: apmail-phoenix-user-archive@minotaur.apache.org Delivered-To: apmail-phoenix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DD42818C95 for ; Tue, 8 Jan 2019 06:29:30 +0000 (UTC) Received: (qmail 7856 invoked by uid 500); 8 Jan 2019 06:29:30 -0000 Delivered-To: apmail-phoenix-user-archive@phoenix.apache.org Received: (qmail 7804 invoked by uid 500); 8 Jan 2019 06:29:29 -0000 Mailing-List: contact user-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@phoenix.apache.org Delivered-To: mailing list user@phoenix.apache.org Received: (qmail 7794 invoked by uid 99); 8 Jan 2019 06:29:29 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jan 2019 06:29:29 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 646F7C236B for ; Tue, 8 Jan 2019 06:29:29 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.655 X-Spam-Level: * X-Spam-Status: No, score=1.655 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.144, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Te2ZAofqfge8 for ; Tue, 8 Jan 2019 06:29:28 +0000 (UTC) Received: from mail-io1-f41.google.com (mail-io1-f41.google.com [209.85.166.41]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 9EFDD5F3CC for ; Tue, 8 Jan 2019 06:29:27 +0000 (UTC) Received: by mail-io1-f41.google.com with SMTP id s22so2310168ioc.8 for ; Mon, 07 Jan 2019 22:29:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=HoaI/RhITL4C+eUksi6lhgrLGIq8eyaUgPNkHeKR+Yg=; b=C/bWQW7mvUNt27+q0Pz9XtBFAkHJrmYqyBhJreBjDdV07zaw40Tv+YGKyqfzX73a42 4CmXP+2cpOuOKDNXL6NmOQ0QrF7NGQlunc0k3k/2LzHSdrsDxfH1RRU5ma/ArOqKjaPP k2EZEHCV+IPUQSO/LSvuXm6MELtXOYb4Ga4Zxy0B30XBGpLR9wRJvxFt3NJCW+h5go13 kqR7IHmeQne8J6aWePzlv3HbBL9dcD/Vow6TIopWsbL7zbGPNJVcmLRl/K41iTvLUVxq zjidtMjHH/S0nW/K5N8zsj12TZ/Y9pHezV02UFaCaVpGlQhDW1Nxw0K73fAzcqt0uab4 ewsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=HoaI/RhITL4C+eUksi6lhgrLGIq8eyaUgPNkHeKR+Yg=; b=R5BwOntc3gP4lczxudDxVxZkUCnlEpeJn6qFqCzFmvnDQ//WcadsMOY5ZY6Yb00htC H5wjVLFJ5HlrrBfp9bO4E+j6sgIZosOZUHDKiWDb80CqrR3uBpV4i6IKvYRRlrUWAy5q 8k9uONUAUmI+DlH39LGb3vrpIQW5zbOZjoFnKJPDfWr1dmEiTCxzYfqiRrWO3bjfqT+/ 8Msf95K8Ui3qpKEc/h3UBj7rnKGeMdg6o0MnBQYjjOrDTpqASOceS0xc2syfJA21Hzg6 onc41FKQzPmOSpujwIKrSucoGISfYFj992l2Z3yWu0kD21L2McoIw+fJWNmMdmQF8g+v JZ5w== X-Gm-Message-State: AJcUukd7PziM9/TKkxiurzat2n/rm83fjErjMVCUmN8wmx3Ad7m4H19i TL/9RYk5cbdRe/IUpM7B6YL1syVHuEskmlx8vMJ09HTWU5o0CA== X-Google-Smtp-Source: ALg8bN5MtA3grjWuS6t53T+T7PjCaU5bV1P8PfCYvyDRJKeRaxf6UR+uXQQjfmwsiPSGyBjpTAyVkezs68+Wn1r/Nec= X-Received: by 2002:a5d:9803:: with SMTP id a3mr308320iol.203.1546928961245; Mon, 07 Jan 2019 22:29:21 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Anil Date: Tue, 8 Jan 2019 11:58:45 +0530 Message-ID: Subject: Re: Hbase vs Phienix column names To: user@phoenix.apache.org Content-Type: multipart/alternative; boundary="0000000000003a55a3057eec77c8" --0000000000003a55a3057eec77c8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I mean can I use following methods ? TWO_BYTE_QUALIFIERS.decode() TWO_BYTE_QUALIFIERS.encode() are there any additional transformations (eg : storage formats) to be considered while reading from hbase ? Thanks. Regards On Tue, 8 Jan 2019 at 11:46, Anil wrote: > Hi Thomas, > > I have checked the hbase system.catalog table and COLUMN_QUALIFIER value > is not encoded. From the internal code, i understood default encoded > scheme used for column is QualifierEncodingScheme.TWO_BYTE_QUALIFIERS. > > Can i use this encoding to get the values from hbase ? Thanks. > > Regards, > Anil > > On Tue, 8 Jan 2019 at 11:24, Thomas D'Silva > wrote: > >> There isn't an existing utility that does that. You would have to look u= p >> the COLUMN_QUALIFIER for the columns you are interested in from >> SYSTEM.CATALOG >> and use then create a Scan. >> >> On Mon, Jan 7, 2019 at 9:22 PM Anil wrote: >> >>> Hi Team, >>> >>> Is there any utility to read hbase data using hbase apis which is >>> created with phoniex with column name encoding ? >>> >>> Idea is to use the all performance and disk usage improvements achieved >>> with phoenix column name encoding feature and use our existing hbase jo= bs >>> for our data analysis. >>> >>> Thanks, >>> Anil >>> >>> On Tue, 11 Dec 2018 at 14:02, Anil wrote: >>> >>>> Thanks. >>>> >>>> On Tue, 11 Dec 2018 at 11:51, Jaanai Zhang >>>> wrote: >>>> >>>>> The difference since used encode column names that support in 4.10 >>>>> version(Also see PHOENIX-1598 >>>>> ). >>>>> You can config COLUMN_ENCODED_BYTES property to keep the original >>>>> column names in the create table SQL, an example for: >>>>> >>>>> create table test( >>>>> >>>>> id varchar primary key, >>>>> >>>>> col varchar >>>>> >>>>> )COLUMN_ENCODED_BYTES =3D0 ; >>>>> >>>>> >>>>> >>>>> ---------------------------------------- >>>>> Jaanai Zhang >>>>> Best regards! >>>>> >>>>> >>>>> >>>>> Anil =E4=BA=8E2018=E5=B9=B412=E6=9C=8811=E6=97= =A5=E5=91=A8=E4=BA=8C =E4=B8=8B=E5=8D=881:24=E5=86=99=E9=81=93=EF=BC=9A >>>>> >>>>>> HI, >>>>>> >>>>>> We have upgraded phoenix to Phoenix-4.11.0-cdh5.11.2 from phoenix >>>>>> 4.7. >>>>>> >>>>>> Problem - When a table is created in phoenix, underlying hbase colum= n >>>>>> names and phoenix column names are different. Tables created in 4.7 = version >>>>>> looks good. Looks >>>>>> >>>>>> CREATE TABLE TST_TEMP (TID VARCHAR PRIMARY KEY ,PRI VARCHAR,SFLG >>>>>> VARCHAR,PFLG VARCHAR,SOLTO VARCHAR,BILTO VARCHAR) COMPRESSION =3D 'S= NAPPY'; >>>>>> >>>>>> 0: jdbc:phoenix:dq-13.labs.> select TID,PRI,SFLG from TST_TEMP limit >>>>>> 2; >>>>>> +-------------+------------+-----------+ >>>>>> | TID | PRI | SFLG | >>>>>> +-------------+------------+-----------+ >>>>>> | 0060189122 | 0.00 | | >>>>>> | 0060298478 | 13390.26 | | >>>>>> +-------------+------------+-----------+ >>>>>> >>>>>> >>>>>> hbase(main):011:0> scan 'TST_TEMP', {LIMIT =3D> 2} >>>>>> ROW COLUMN+CELL >>>>>> 0060189122 column=3D0:\x00\x00\x00\x00= , >>>>>> timestamp=3D1544296959236, value=3Dx >>>>>> 0060189122 column=3D0:\x80\x0B, >>>>>> timestamp=3D1544296959236, value=3D0.00 >>>>>> 0060298478 column=3D0:\x00\x00\x00\x00= , >>>>>> timestamp=3D1544296959236, value=3Dx >>>>>> 0060298478 column=3D0:\x80\x0B, >>>>>> timestamp=3D1544296959236, value=3D13390.26 >>>>>> >>>>>> >>>>>> hbase columns names are completely different than phoenix column >>>>>> names. This change observed only post up-gradation. all existing tab= les >>>>>> created in earlier versions looks good and alter statements to exist= ing >>>>>> tables also looks good. >>>>>> >>>>>> Is there any workaround to avoid this difference? we could not run >>>>>> hbase mapreduce jobs on hbase tables created by phoenix. Thanks. >>>>>> >>>>>> Thanks >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> --0000000000003a55a3057eec77c8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I mean can I use following methods ?
=
TWO_BYTE_QUALIFIERS.decode()
TWO_BYTE_QUALIFIE= RS.encode()

are there any additional transform= ations (eg : storage formats) to be considered while reading from hbase ?

Thanks. Regards

On Tue, 8 Jan 2019 at 11:46, A= nil <anilklce@gmail.com> wr= ote:
Hi Thomas,

I have checked the hba= se system.catalog table and COLUMN_QUALIFIER value is not encoded.=C2=A0 Fr= om the internal code, i understood default encoded scheme used for column i= s=C2=A0QualifierEncodingScheme.TWO_BYTE_QUALIFIERS.

Can i use this encoding to get the values from hbase ? Thanks.
=
Regards,
Anil

On Tue, 8 Jan 2019 at 11:24, Thomas D'Silv= a <tdsilva@s= alesforce.com> wrote:
There isn't an existing = utility that does that. You would have to look up the COLUMN_QUALIFIER for = the columns you are interested in from SYSTEM.CATALOG
and use the= n create a Scan.

On Mon, Jan 7, 2019 at 9:22 PM Anil <anilklce@gmail.com> wrote:
Hi Team,

<= /div>
Is there any utility to read hbase data using hbase apis which is= created with phoniex with column name encoding ?

= Idea is to use the all performance and disk usage improvements achieved wit= h phoenix column name encoding feature and use our existing hbase jobs for = our data analysis.

Thanks,
Anil

On Tue, 11 Dec 2018 at 1= 4:02, Anil <anil= klce@gmail.com> wrote:
Thanks.

=
On Tue, 11 Dec 2018 at 11:51, Jaanai Zhang <cloud.poster@gmail.com= > wrote:
The difference since used encode=C2=A0column = names that support in 4.10 version(Also see=C2=A0PHOENIX-1598).
You ca= n config COLUMN_ENCODED_BYTES property to keep the original column names in= the create table SQL, an example for:

create table test(

id varchar=C2=A0 primary key,

col varchar

)COLUMN_ENCODED_BYTES =3D0 ;



----------------------------------------
=C2= =A0 =C2=A0Jaanai Zhang
=C2=A0 =C2=A0Best regards!
<= br>

Anil <anilklce@gmail.com> =E4=BA=8E201= 8=E5=B9=B412=E6=9C=8811=E6=97=A5=E5=91=A8=E4=BA=8C =E4=B8=8B=E5=8D=881:24= =E5=86=99=E9=81=93=EF=BC=9A
HI,

We have upgraded phoenix to Ph= oenix-4.11.0-cdh5.11.2 from phoenix 4.7.=C2=A0

Pro= blem - When a table is created in phoenix, underlying hbase column names an= d phoenix column names are different. Tables created in 4.7 version looks g= ood. Looks

CREATE TABLE TST_TEMP (TID VARCHAR = PRIMARY KEY ,PRI VARCHAR,SFLG VARCHAR,PFLG VARCHAR,SOLTO VARCHAR,BILTO VARC= HAR) COMPRESSION =3D 'SNAPPY';

0:= jdbc:phoenix:dq-13.labs.> select TID,PRI,SFLG from TST_TEMP limit 2;
+-------------+------------+-----------+
|=C2=A0 =C2=A0TI= D=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 PRI=C2=A0 =C2=A0 =C2=A0|=C2=A0 = =C2=A0 SFLG=C2=A0 =C2=A0|
+-------------+------------+-----------= +
| 0060189122=C2=A0 | 0.00=C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
| 0060298478=C2=A0 | 13390.26=C2= =A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|
+----------= ---+------------+-----------+


hbase(main):011:0> scan 'TST_TEMP', {LIMIT =3D> 2}
ROW=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 COLUMN+C= ELL
=C2=A00060189122=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 column=3D0:\x00= \x00\x00\x00, timestamp=3D1544296959236, value=3Dx
=C2=A000601891= 22=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 column=3D0:\x80\x0B, timestamp=3D1544296959= 236, value=3D0.00
=C2=A00060298478=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 c= olumn=3D0:\x00\x00\x00\x00, timestamp=3D1544296959236, value=3Dx
= =C2=A00060298478=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 column=3D0:\x80\x0B, timestam= p=3D1544296959236, value=3D13390.26


hbase columns names are completely different than phoenix column nam= es. This change observed only post up-gradation. all existing tables create= d in earlier versions looks good and alter statements to existing tables al= so looks good.

Is there any workaround to avoid th= is difference? we could not run hbase mapreduce jobs on hbase tables create= d=C2=A0 by phoenix. Thanks.

Thanks

<= /div>





<= /div>
--0000000000003a55a3057eec77c8--