phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From puneet <puneet.ku...@pubmatic.com>
Subject Re: How to Use Compression in Phoenix 4.0.0
Date Wed, 02 Jul 2014 12:20:41 GMT
Hi James,

Thanks for you quick reply.
------------------------------------

I had created the table with COMPRESSION='SNAPPY' property but when I 
describe it through hbase shell then it shows COMPRESION='NONE'

I did a workaround and performed alter from Hbase shell as below :-

disable 'ADV_CAT_SNAPPY'
0 row(s) in 2.0320 seconds

hbase(main):027:0> alter 'ADV_CAT_SNAPPY',{NAME=>'0',COMPRESSION=>'SNAPPY'}
Updating all regions with the new schema...
1/1 regions updated.
Done.
0 row(s) in 1.1440 seconds

hbase(main):028:0> enable 'ADV_CAT_SNAPPY'
0 row(s) in 2.0970 seconds
-----------------------------
After the above steps the compression is modifed to SNAPPY.

But the compression seems to be happening only when I am Loading via 
PSQL utility.

When I run it from Loading via MaprReduce , then it is not complressing 
the data. I have Hbase env file modifed to use Snappy lib in Hadoop


    Need steps for enabling compression via mapreduce.



On Wednesday 02 July 2014 03:42 PM, James Taylor wrote:
> If the table already exists, then CREATE TABLE IF NOT EXISTS is a noop.
> Drop the table first and then create initially with the COMPRESSION='SNAPPY'
> property.
>
> Thanks,
> James
>
>
> On Wed, Jul 2, 2014 at 11:29 AM, puneet <puneet.kumar@pubmatic.com> wrote:
>
>>   Hi Team,
>>
>> I need snappy compression to be used for the Hbase table but it is not
>> creating the table as SNAPPY compressed when I look through the DDL's in
>> the Hbase shell.
>>
>> Also while loading the data the size of the data remains the same for
>> SNAPPY or NONE.
>>
>> Please help if I am missing anything.
>>
>> Details Below:-
>>
>> CREATE TABLE IF NOT EXISTS adv_cat (
>>       cat varchar ,
>>       domain VARCHAR
>>       CONSTRAINT PK PRIMARY KEY (cat,domain)
>> );
>>
>> DDL:- describe 'ADV_CAT_GZ'
>> DESCRIPTION ENABLED
>>   'ADV_CAT_GZ', {METHOD => 'table_att', coprocessor$1 =>
>> '|org.apache.phoenix.coprocessor.ScanRegionObser true
>>   ver|1|', coprocessor$2 =>
>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
>> coproc
>>   essor$3 =>
>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>> coprocessor$4 => '|org.
>>   apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|', coprocessor$5
>> => '|org.apache.phoenix.hbase.i
>>   ndex.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixIndexBuilder,org.apache.hadoop.hb
>>
>>   ase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'}, {NAME
>> => '0', DATA_BLOCK_ENCODING =
>>   > 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS
>> => '3', COMPRESSION => 'NONE',
>>    MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'true',
>> BLOCKSIZE => '65536', IN_MEMOR
>>   Y => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>>
>>
>> CREATE TABLE IF NOT EXISTS adv_cat_snappy (
>>       cat varchar ,
>>       domain VARCHAR
>>       CONSTRAINT PK PRIMARY KEY (cat,domain)
>> ) COMPRESSION='SNAPPY';
>>
>> describe 'ADV_CAT_SNAPPY'
>> DESCRIPTION ENABLED
>>   'ADV_CAT_SNAPPY', {METHOD => 'table_att', coprocessor$1 =>
>> '|org.apache.phoenix.coprocessor.ScanRegionO true
>>   bserver|1|', coprocessor$2 =>
>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|', co
>>   processor$3 =>
>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>> coprocessor$4 => '|
>>   org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
>> coprocessor$5 => '|org.apache.phoenix.hba
>>   se.index.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixIndexBuilder,org.apache.hadoo
>>
>>   p.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'},
>> {NAME => '0', DATA_BLOCK_ENCODI
>>   NG => 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
>> VERSIONS => '3', COMPRESSION => 'NO
>>   NE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS =>
>> 'true', BLOCKSIZE => '65536', IN_M
>>   EMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>> 1 row(s) in 0.0660 seconds
>>
>>   --
>> ------------------------------
>>
>> *Puneet Ojha* | Tech Lead: Data Analytics
>>   6th Floor, Amar Paradigm | Baner Road, Pune 411045
>> o: +91-20-67285700 | m: +91-8605359898 | Skype: puneetkr.ojha
>>
>> [image: PubMatic]
>> <http://www.pubmatic.com/index-eng.php?utm_source=Footer&utm_medium=Email&utm_content=PuneLogo&utm_campaign=Corporate>
>> Every ad. Every sales channel. Every screen. *One platform*
>>
>> On the Web: [image: Facebook]
>> <http://www.pubmatic.com/social/facebook.php?utm_source=Footer&utm_medium=Email&utm_content=PuneFacebook&utm_campaign=Corporate>
[image:
>> Twitter]
>> <http://www.pubmatic.com/social/twitter.php?utm_source=Footer&utm_medium=Email&utm_content=PuneTwitter&utm_campaign=Corporate>
[image:
>> LinkedIn]
>> <http://www.pubmatic.com/social/linkedin.php?utm_source=Footer&utm_medium=Email&utm_content=PuneLinkedIn&utm_campaign=Corporate>
>>
>>   Find out more about our services: PubConnect
>> <http://www.pubmatic.com/pubconnect.php?utm_source=Footer&utm_medium=Email&utm_content=PunePubConnect&utm_campaign=Corporate>
>> | PubDirect
>> <http://www.pubmatic.com/pubdirect.php?utm_source=Footer&utm_medium=Email&utm_content=PunePubDirect&utm_campaign=Corporate>
>> | Solutions for Media Buyers
>> <http://www.pubmatic.com/media-buyers-overview.php?utm_source=Footer&utm_medium=Email&utm_content=PuneMediaBuyers&utm_campaign=Corporate>
>> What's New: PubMatic Launches Mobile SDK for app developers.
>> <http://www.pubmatic.com/press/2013/PubMatic-Launches-Self-Service-Open-SDK.php?utm_source=Footer&utm_medium=Email&utm_content=PuneSDKLink&utm_campaign=Corporate>
>> Latest Research: Real-Time Bidding in the United States and Worldwide,
>> 2010-2017 (Q4 2013)
>> <http://www.pubmatic.com/reports-and-whitepapers.php?utm_source=Footer&utm_medium=Email&utm_content=PuneReportsLink&utm_campaign=Corporate>
>> Recent Videos: Ad Revenue 6
>> <http://www.pubmatic.com/ar6_videos.php?utm_source=Footer&utm_medium=Email&utm_content=NYCAR6Link&utm_campaign=Corporate>
>> | Ad Revenue Europe
>> <http://www.pubmatic.com/arE_videos.php?utm_source=Footer&utm_medium=Email&utm_content=NYCAR6Link&utm_campaign=Corporate>
>>
>>
>>
>> If the table already exists, then CREATE TABLE IF NOT EXISTS is a 
>> noop. Drop the table first and then create initially with the 
>> COMPRESSION='SNAPPY' property.
>>
>> Thanks,
>> James
>>
>>
>> On Wed, Jul 2, 2014 at 11:29 AM, puneet <puneet.kumar@pubmatic.com 
>> <mailto:puneet.kumar@pubmatic.com>> wrote:
>>
>>     Hi Team,
>>
>>     I need snappy compression to be used for the Hbase table but it
>>     is not creating the table as SNAPPY compressed when I look
>>     through the DDL's in the Hbase shell.
>>
>>     Also while loading the data the size of the data remains the same
>>     for SNAPPY or NONE.
>>
>>     Please help if I am missing anything.
>>
>>     Details Below:-
>>
>>     CREATE TABLE IF NOT EXISTS adv_cat (
>>          cat varchar ,
>>          domain VARCHAR
>>          CONSTRAINT PK PRIMARY KEY (cat,domain)
>>     );
>>
>>     DDL:- describe 'ADV_CAT_GZ'
>>     DESCRIPTION ENABLED
>>      'ADV_CAT_GZ', {METHOD => 'table_att', coprocessor$1 =>
>>     '|org.apache.phoenix.coprocessor.ScanRegionObser true
>>      ver|1|', coprocessor$2 =>
>>     '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
>>     coproc
>>      essor$3 =>
>>     '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>>     coprocessor$4 => '|org.
>>      apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
>>     coprocessor$5 => '|org.apache.phoenix.hbase.i
>>      ndex.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixIndexBuilder,org.apache.hadoop.hb
>>
>>      ase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'},
>>     {NAME => '0', DATA_BLOCK_ENCODING =
>>      > 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
>>     VERSIONS => '3', COMPRESSION => 'NONE',
>>       MIN_VERSIONS => '0', TTL => '2147483647 <tel:2147483647>',
>>     KEEP_DELETED_CELLS => 'true', BLOCKSIZE => '65536', IN_MEMOR
>>      Y => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>>
>>
>>     CREATE TABLE IF NOT EXISTS adv_cat_snappy (
>>          cat varchar ,
>>          domain VARCHAR
>>          CONSTRAINT PK PRIMARY KEY (cat,domain)
>>     ) COMPRESSION='SNAPPY';
>>
>>     describe 'ADV_CAT_SNAPPY'
>>     DESCRIPTION ENABLED
>>      'ADV_CAT_SNAPPY', {METHOD => 'table_att', coprocessor$1 =>
>>     '|org.apache.phoenix.coprocessor.ScanRegionO true
>>      bserver|1|', coprocessor$2 =>
>>     '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
>>     co
>>      processor$3 =>
>>     '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>>     coprocessor$4 => '|
>>      org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
>>     coprocessor$5 => '|org.apache.phoenix.hba
>>      se.index.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixIndexBuilder,org.apache.hadoo
>>
>>      p.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'},
>>     {NAME => '0', DATA_BLOCK_ENCODI
>>      NG => 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE =>
>>     '0', VERSIONS => '3', COMPRESSION => 'NO
>>      NE', MIN_VERSIONS => '0', TTL => '2147483647 <tel:2147483647>',
>>     KEEP_DELETED_CELLS => 'true', BLOCKSIZE => '65536', IN_M
>>      EMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>>     1 row(s) in 0.0660 seconds
>>
>>     -- 
>>     ------------------------------------------------------------------------
>>
>>     *Puneet Ojha* | Tech Lead: Data Analytics
>>     6th Floor, Amar Paradigm | Baner Road, Pune 411045
>>     o: +91-20-67285700 <tel:%2B91-20-67285700> | m: +91-8605359898
>>     <tel:%2B91-8605359898> | Skype: puneetkr.ojha
>>
>>     PubMatic
>>     <http://www.pubmatic.com/index-eng.php?utm_source=Footer&utm_medium=Email&utm_content=PuneLogo&utm_campaign=Corporate>
>>     Every ad. Every sales channel. Every screen. *One platform*
>>
>>     On the Web: Facebook
>>     <http://www.pubmatic.com/social/facebook.php?utm_source=Footer&utm_medium=Email&utm_content=PuneFacebook&utm_campaign=Corporate>
>>     Twitter
>>     <http://www.pubmatic.com/social/twitter.php?utm_source=Footer&utm_medium=Email&utm_content=PuneTwitter&utm_campaign=Corporate>
>>     LinkedIn
>>     <http://www.pubmatic.com/social/linkedin.php?utm_source=Footer&utm_medium=Email&utm_content=PuneLinkedIn&utm_campaign=Corporate>
>>
>>     Find out more about our services: PubConnect
>>     <http://www.pubmatic.com/pubconnect.php?utm_source=Footer&utm_medium=Email&utm_content=PunePubConnect&utm_campaign=Corporate>
>>     | PubDirect
>>     <http://www.pubmatic.com/pubdirect.php?utm_source=Footer&utm_medium=Email&utm_content=PunePubDirect&utm_campaign=Corporate>
>>     | Solutions for Media Buyers
>>     <http://www.pubmatic.com/media-buyers-overview.php?utm_source=Footer&utm_medium=Email&utm_content=PuneMediaBuyers&utm_campaign=Corporate>
>>     What's New: PubMatic Launches Mobile SDK for app developers.
>>     <http://www.pubmatic.com/press/2013/PubMatic-Launches-Self-Service-Open-SDK.php?utm_source=Footer&utm_medium=Email&utm_content=PuneSDKLink&utm_campaign=Corporate>
>>     Latest Research: Real-Time Bidding in the United States and
>>     Worldwide, 2010-2017 (Q4 2013)
>>     <http://www.pubmatic.com/reports-and-whitepapers.php?utm_source=Footer&utm_medium=Email&utm_content=PuneReportsLink&utm_campaign=Corporate>
>>     Recent Videos: Ad Revenue 6
>>     <http://www.pubmatic.com/ar6_videos.php?utm_source=Footer&utm_medium=Email&utm_content=NYCAR6Link&utm_campaign=Corporate>
>>     | Ad Revenue Europe
>>     <http://www.pubmatic.com/arE_videos.php?utm_source=Footer&utm_medium=Email&utm_content=NYCAR6Link&utm_campaign=Corporate>
>>
>>

-- 
------------------------------------------------------------------------

*Puneet Ojha* | Tech Lead: Data Analytics
6th Floor, Amar Paradigm | Baner Road, Pune 411045
o: +91-20-67285700 | m: +91-8605359898 | Skype: puneetkr.ojha

PubMatic 
<http://www.pubmatic.com/index-eng.php?utm_source=Footer&utm_medium=Email&utm_content=PuneLogo&utm_campaign=Corporate>
Every ad. Every sales channel. Every screen. *One platform*

On the Web: Facebook 
<http://www.pubmatic.com/social/facebook.php?utm_source=Footer&utm_medium=Email&utm_content=PuneFacebook&utm_campaign=Corporate>

Twitter 
<http://www.pubmatic.com/social/twitter.php?utm_source=Footer&utm_medium=Email&utm_content=PuneTwitter&utm_campaign=Corporate>

LinkedIn 
<http://www.pubmatic.com/social/linkedin.php?utm_source=Footer&utm_medium=Email&utm_content=PuneLinkedIn&utm_campaign=Corporate>

Find out more about our services: PubConnect 
<http://www.pubmatic.com/pubconnect.php?utm_source=Footer&utm_medium=Email&utm_content=PunePubConnect&utm_campaign=Corporate>

| PubDirect 
<http://www.pubmatic.com/pubdirect.php?utm_source=Footer&utm_medium=Email&utm_content=PunePubDirect&utm_campaign=Corporate>

| Solutions for Media Buyers 
<http://www.pubmatic.com/media-buyers-overview.php?utm_source=Footer&utm_medium=Email&utm_content=PuneMediaBuyers&utm_campaign=Corporate>
What's New: PubMatic Launches Mobile SDK for app developers. 
<http://www.pubmatic.com/press/2013/PubMatic-Launches-Self-Service-Open-SDK.php?utm_source=Footer&utm_medium=Email&utm_content=PuneSDKLink&utm_campaign=Corporate>
Latest Research: Real-Time Bidding in the United States and Worldwide, 
2010-2017 (Q4 2013) 
<http://www.pubmatic.com/reports-and-whitepapers.php?utm_source=Footer&utm_medium=Email&utm_content=PuneReportsLink&utm_campaign=Corporate>
Recent Videos: Ad Revenue 6 
<http://www.pubmatic.com/ar6_videos.php?utm_source=Footer&utm_medium=Email&utm_content=NYCAR6Link&utm_campaign=Corporate>

| Ad Revenue Europe 
<http://www.pubmatic.com/arE_videos.php?utm_source=Footer&utm_medium=Email&utm_content=NYCAR6Link&utm_campaign=Corporate>


Mime
View raw message