phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeffrey Zhong <jzh...@hortonworks.com>
Subject Re: How to Use Compression in Phoenix 4.0.0
Date Wed, 02 Jul 2014 19:07:36 GMT

Have you tried to set configuration "hfile.compression"= SNAPPY in
hbase-site.xml when using bulkloadtool?

From:  puneet <puneet.kumar@pubmatic.com>
Reply-To:  <user@phoenix.apache.org>
Date:  Wednesday, July 2, 2014 5:20 AM
To:  <user@phoenix.apache.org>
Subject:  Re: How to Use Compression in Phoenix 4.0.0

    
 Hi James,
 
 Thanks for you quick reply.
 ------------------------------------
 
 I had created the table with COMPRESSION='SNAPPY' property but when I
describe it through hbase shell then it shows COMPRESION='NONE'
 
 I did a workaround and performed alter from Hbase shell as below :-
 
 disable 'ADV_CAT_SNAPPY'
 0 row(s) in 2.0320 seconds
 
 hbase(main):027:0> alter 'ADV_CAT_SNAPPY',{NAME=>'0',COMPRESSION=>'SNAPPY'}
 Updating all regions with the new schema...
 1/1 regions updated.
 Done.
 0 row(s) in 1.1440 seconds
 
 hbase(main):028:0> enable 'ADV_CAT_SNAPPY'
 0 row(s) in 2.0970 seconds
 -----------------------------
 After the above steps the compression is modifed to SNAPPY.
 
 But the compression seems to be happening only when I am Loading via PSQL
utility.
 

When I run it from Loading via MaprReduce , then it is not complressing the
data. I have Hbase env file modifed to use Snappy lib in Hadoop
 
 
Need steps for enabling compression via mapreduce.
 


 
 
 
On Wednesday 02 July 2014 03:42 PM, James Taylor wrote:
 
 
>  
> If the table already exists, then CREATE TABLE IF NOT EXISTS is a noop.
> Drop the table first and then create initially with the COMPRESSION='SNAPPY'
> property.
> 
> Thanks,
> James
> 
> 
> On Wed, Jul 2, 2014 at 11:29 AM, puneet <puneet.kumar@pubmatic.com>
> <mailto:puneet.kumar@pubmatic.com>  wrote:
> 
>  
>>  
>>  Hi Team,
>> 
>> I need snappy compression to be used for the Hbase table but it is not
>> creating the table as SNAPPY compressed when I look through the DDL's in
>> the Hbase shell.
>> 
>> Also while loading the data the size of the data remains the same for
>> SNAPPY or NONE.
>> 
>> Please help if I am missing anything.
>> 
>> Details Below:-
>> 
>> CREATE TABLE IF NOT EXISTS adv_cat (
>>      cat varchar ,
>>      domain VARCHAR
>>      CONSTRAINT PK PRIMARY KEY (cat,domain)
>> );
>> 
>> DDL:- describe 'ADV_CAT_GZ'
>> DESCRIPTION ENABLED
>>  'ADV_CAT_GZ', {METHOD => 'table_att', coprocessor$1 =>
>> '|org.apache.phoenix.coprocessor.ScanRegionObser true
>>  ver|1|', coprocessor$2 =>
>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
>> coproc
>>  essor$3 =>
>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>> coprocessor$4 => '|org.
>>  apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|', coprocessor$5
>> => '|org.apache.phoenix.hbase.i
>>  
>> ndex.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixIndexBu
>> ilder,org.apache.hadoop.hb
>> 
>>  ase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'}, {NAME
>> => '0', DATA_BLOCK_ENCODING =
>>>  > 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS
>> => '3', COMPRESSION => 'NONE',
>>   MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'true',
>> BLOCKSIZE => '65536', IN_MEMOR
>>  Y => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>> 
>> 
>> CREATE TABLE IF NOT EXISTS adv_cat_snappy (
>>      cat varchar ,
>>      domain VARCHAR
>>      CONSTRAINT PK PRIMARY KEY (cat,domain)
>> ) COMPRESSION='SNAPPY';
>> 
>> describe 'ADV_CAT_SNAPPY'
>> DESCRIPTION ENABLED
>>  'ADV_CAT_SNAPPY', {METHOD => 'table_att', coprocessor$1 =>
>> '|org.apache.phoenix.coprocessor.ScanRegionO true
>>  bserver|1|', coprocessor$2 =>
>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|', co
>>  processor$3 =>
>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>> coprocessor$4 => '|
>>  org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
>> coprocessor$5 => '|org.apache.phoenix.hba
>>  
>> se.index.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixInd
>> exBuilder,org.apache.hadoo
>> 
>>  p.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'},
>> {NAME => '0', DATA_BLOCK_ENCODI
>>  NG => 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
>> VERSIONS => '3', COMPRESSION => 'NO
>>  NE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS =>
>> 'true', BLOCKSIZE => '65536', IN_M
>>  EMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>> 1 row(s) in 0.0660 seconds
>> 
>>  --
>> ------------------------------
>> 
>> *Puneet Ojha* | Tech Lead: Data Analytics
>>  6th Floor, Amar Paradigm | Baner Road, Pune 411045
>> o: +91-20-67285700 | m: +91-8605359898 | Skype: puneetkr.ojha
>> 
>> [image: PubMatic]
>> <http://www.pubmatic.com/index-eng.php?utm_source=Footer&utm_medium=Email&utm
>> _content=PuneLogo&utm_campaign=Corporate>
>> <http://www.pubmatic.com/index-eng.php?utm_source=Footer&utm_medium=Email&utm
>> _content=PuneLogo&utm_campaign=Corporate>
>> Every ad. Every sales channel. Every screen. *One platform*
>> 
>> On the Web: [image: Facebook]
>> <http://www.pubmatic.com/social/facebook.php?utm_source=Footer&utm_medium=Ema
>> il&utm_content=PuneFacebook&utm_campaign=Corporate>
>> <http://www.pubmatic.com/social/facebook.php?utm_source=Footer&utm_medium=Ema
>> il&utm_content=PuneFacebook&utm_campaign=Corporate>  [image:
>> Twitter]
>> <http://www.pubmatic.com/social/twitter.php?utm_source=Footer&utm_medium=Emai
>> l&utm_content=PuneTwitter&utm_campaign=Corporate>
>> <http://www.pubmatic.com/social/twitter.php?utm_source=Footer&utm_medium=Emai
>> l&utm_content=PuneTwitter&utm_campaign=Corporate>  [image:
>> LinkedIn]
>> <http://www.pubmatic.com/social/linkedin.php?utm_source=Footer&utm_medium=Ema
>> il&utm_content=PuneLinkedIn&utm_campaign=Corporate>
>> <http://www.pubmatic.com/social/linkedin.php?utm_source=Footer&utm_medium=Ema
>> il&utm_content=PuneLinkedIn&utm_campaign=Corporate>
>> 
>>  Find out more about our services: PubConnect
>> <http://www.pubmatic.com/pubconnect.php?utm_source=Footer&utm_medium=Email&ut
>> m_content=PunePubConnect&utm_campaign=Corporate>
>> <http://www.pubmatic.com/pubconnect.php?utm_source=Footer&utm_medium=Email&ut
>> m_content=PunePubConnect&utm_campaign=Corporate>
>> | PubDirect
>> <http://www.pubmatic.com/pubdirect.php?utm_source=Footer&utm_medium=Email&utm
>> _content=PunePubDirect&utm_campaign=Corporate>
>> <http://www.pubmatic.com/pubdirect.php?utm_source=Footer&utm_medium=Email&utm
>> _content=PunePubDirect&utm_campaign=Corporate>
>> | Solutions for Media Buyers
>> <http://www.pubmatic.com/media-buyers-overview.php?utm_source=Footer&utm_medi
>> um=Email&utm_content=PuneMediaBuyers&utm_campaign=Corporate>
>> <http://www.pubmatic.com/media-buyers-overview.php?utm_source=Footer&utm_medi
>> um=Email&utm_content=PuneMediaBuyers&utm_campaign=Corporate>
>> What's New: PubMatic Launches Mobile SDK for app developers.
>> <http://www.pubmatic.com/press/2013/PubMatic-Launches-Self-Service-Open-SDK.p
>> hp?utm_source=Footer&utm_medium=Email&utm_content=PuneSDKLink&utm_campaign=Co
>> rporate> 
>> <http://www.pubmatic.com/press/2013/PubMatic-Launches-Self-Service-Open-SDK.p
>> hp?utm_source=Footer&utm_medium=Email&utm_content=PuneSDKLink&utm_campaign=Co
>> rporate> 
>> Latest Research: Real-Time Bidding in the United States and Worldwide,
>> 2010-2017 (Q4 2013)
>> <http://www.pubmatic.com/reports-and-whitepapers.php?utm_source=Footer&utm_me
>> dium=Email&utm_content=PuneReportsLink&utm_campaign=Corporate>
>> <http://www.pubmatic.com/reports-and-whitepapers.php?utm_source=Footer&utm_me
>> dium=Email&utm_content=PuneReportsLink&utm_campaign=Corporate>
>> Recent Videos: Ad Revenue 6
>> <http://www.pubmatic.com/ar6_videos.php?utm_source=Footer&utm_medium=Email&ut
>> m_content=NYCAR6Link&utm_campaign=Corporate>
>> <http://www.pubmatic.com/ar6_videos.php?utm_source=Footer&utm_medium=Email&ut
>> m_content=NYCAR6Link&utm_campaign=Corporate>
>> | Ad Revenue Europe
>> <http://www.pubmatic.com/arE_videos.php?utm_source=Footer&utm_medium=Email&ut
>> m_content=NYCAR6Link&utm_campaign=Corporate>
>> <http://www.pubmatic.com/arE_videos.php?utm_source=Footer&utm_medium=Email&ut
>> m_content=NYCAR6Link&utm_campaign=Corporate>
>>  
>>   
>>   
>> If the table already exists, then CREATE TABLE IF NOT EXISTS is a noop. Drop
>> the table first and then create initially with the COMPRESSION='SNAPPY'
>> property. 
>>  
>>  
>>  
>> Thanks,
>>  
>> James
>>  
>> 
>>  
>>  
>> On Wed, Jul 2, 2014 at 11:29 AM, puneet <puneet.kumar@pubmatic.com> wrote:
>>  
>>>  
>>>  Hi Team, 
>>>  
>>>  I need snappy compression to be used for the Hbase table but it is not
>>> creating the table as SNAPPY compressed when I look through the DDL's in the
>>> Hbase shell. 
>>>  
>>>  Also while loading the data the size of the data remains the same for
>>> SNAPPY or NONE.
>>>  
>>>  Please help if I am missing anything.
>>>  
>>>  Details Below:-
>>>  
>>>  CREATE TABLE IF NOT EXISTS adv_cat (
>>>       cat varchar ,
>>>       domain VARCHAR
>>>       CONSTRAINT PK PRIMARY KEY (cat,domain)
>>>  ); 
>>>  
>>>  DDL:- describe 'ADV_CAT_GZ'
>>>  DESCRIPTION ENABLED
>>>   'ADV_CAT_GZ', {METHOD => 'table_att', coprocessor$1 =>
>>> '|org.apache.phoenix.coprocessor.ScanRegionObser true
>>>   ver|1|', coprocessor$2 =>
>>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
>>> coproc 
>>>   essor$3 => 
>>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>>> coprocessor$4 => '|org.
>>>   apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',  coprocessor$5
>>> => '|org.apache.phoenix.hbase.i
>>>   
>>> ndex.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixIndexB
>>> uilder,org.apache.hadoop.hb
>>>   ase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'},  {NAME
>>> => '0', DATA_BLOCK_ENCODING =
>>>>   > 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
VERSIONS
>>>> => '3', COMPRESSION => 'NONE',
>>>    MIN_VERSIONS => '0', TTL => '2147483647 <tel:2147483647> ',
>>> KEEP_DELETED_CELLS => 'true', BLOCKSIZE => '65536', IN_MEMOR
>>>   Y => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>>>  
>>>  
>>>  CREATE TABLE IF NOT EXISTS adv_cat_snappy (
>>>       cat varchar ,
>>>       domain VARCHAR
>>>       CONSTRAINT PK PRIMARY KEY (cat,domain)
>>>  ) COMPRESSION='SNAPPY';
>>>  
>>>  describe 'ADV_CAT_SNAPPY'
>>>  DESCRIPTION ENABLED
>>>   'ADV_CAT_SNAPPY', {METHOD => 'table_att', coprocessor$1 =>
>>> '|org.apache.phoenix.coprocessor.ScanRegionO true
>>>   bserver|1|', coprocessor$2 =>
>>> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|', co
>>>   processor$3 =>
>>> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
>>> coprocessor$4 => '|
>>>   org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
>>> coprocessor$5 => '|org.apache.phoenix.hba
>>>   
>>> se.index.Indexer|1073741823|index.builder=org.apache.phoenix.index.PhoenixIn
>>> dexBuilder,org.apache.hadoo
>>>   p.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec'},
>>> {NAME => '0', DATA_BLOCK_ENCODI
>>>   NG => 'FAST_DIFF', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
>>> VERSIONS => '3', COMPRESSION => 'NO
>>>   NE', MIN_VERSIONS => '0', TTL => '2147483647 <tel:2147483647> ',
>>> KEEP_DELETED_CELLS => 'true', BLOCKSIZE => '65536', IN_M
>>>   EMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'}
>>>  1 row(s) in 0.0660 seconds
>>>  
>>>   
>>> -- 
>>>  
>>> 
>>>  
>>> 
>>> Puneet Ojha | Tech Lead: Data Analytics
>>>   6th Floor, Amar Paradigm | Baner Road, Pune 411045
>>>  o: +91-20-67285700 <tel:%2B91-20-67285700>  | m: +91-8605359898
>>> <tel:%2B91-8605359898>  | Skype: puneetkr.ojha
>>>  
>>> 
>>>  
>>> <http://www.pubmatic.com/index-eng.php?utm_source=Footer&utm_medium=Email&ut
>>> m_content=PuneLogo&utm_campaign=Corporate>
>>>  Every ad. Every sales channel. Every screen. One platform
>>>  
>>> 
>>> On the Web:  
>>> <http://www.pubmatic.com/social/facebook.php?utm_source=Footer&utm_medium=Em
>>> ail&utm_content=PuneFacebook&utm_campaign=Corporate>
>>> <http://www.pubmatic.com/social/twitter.php?utm_source=Footer&utm_medium=Ema
>>> il&utm_content=PuneTwitter&utm_campaign=Corporate>
>>> <http://www.pubmatic.com/social/linkedin.php?utm_source=Footer&utm_medium=Em
>>> ail&utm_content=PuneLinkedIn&utm_campaign=Corporate>
>>>  
>>> 
>>>  
>>>  
>>> 
>>> Find out more about our services: PubConnect
>>> <http://www.pubmatic.com/pubconnect.php?utm_source=Footer&utm_medium=Email&u
>>> tm_content=PunePubConnect&utm_campaign=Corporate>  | PubDirect
>>> <http://www.pubmatic.com/pubdirect.php?utm_source=Footer&utm_medium=Email&ut
>>> m_content=PunePubDirect&utm_campaign=Corporate>  | Solutions for Media
>>> Buyers 
>>> <http://www.pubmatic.com/media-buyers-overview.php?utm_source=Footer&utm_med
>>> ium=Email&utm_content=PuneMediaBuyers&utm_campaign=Corporate>
>>>  What's New: PubMatic Launches Mobile SDK for app developers.
>>> <http://www.pubmatic.com/press/2013/PubMatic-Launches-Self-Service-Open-SDK.
>>> php?utm_source=Footer&utm_medium=Email&utm_content=PuneSDKLink&utm_campaign=
>>> Corporate> 
>>>  Latest Research: Real-Time Bidding in the United States and Worldwide,
>>> 2010-2017 (Q4 2013)
>>> <http://www.pubmatic.com/reports-and-whitepapers.php?utm_source=Footer&utm_m
>>> edium=Email&utm_content=PuneReportsLink&utm_campaign=Corporate>
>>>  Recent Videos: Ad Revenue 6
>>> <http://www.pubmatic.com/ar6_videos.php?utm_source=Footer&utm_medium=Email&u
>>> tm_content=NYCAR6Link&utm_campaign=Corporate>  | Ad Revenue Europe
>>> <http://www.pubmatic.com/arE_videos.php?utm_source=Footer&utm_medium=Email&u
>>> tm_content=NYCAR6Link&utm_campaign=Corporate>
>>>  
>>>  
>>>  
>>  
>>  
>>  
>>  
>>  
>  
 
 
-- 
 

 

Puneet Ojha | Tech Lead: Data Analytics
  6th Floor, Amar Paradigm | Baner Road, Pune 411045
 o: +91-20-67285700 | m: +91-8605359898 | Skype: puneetkr.ojha
 

 <http://www.pubmatic.com/index-eng.php?utm_source=Footer&utm_medium=Email&u
tm_content=PuneLogo&utm_campaign=Corporate>
 Every ad. Every sales channel. Every screen. One platform
 

On the Web:  
<http://www.pubmatic.com/social/facebook.php?utm_source=Footer&utm_medium=Em
ail&utm_content=PuneFacebook&utm_campaign=Corporate>
<http://www.pubmatic.com/social/twitter.php?utm_source=Footer&utm_medium=Ema
il&utm_content=PuneTwitter&utm_campaign=Corporate>
<http://www.pubmatic.com/social/linkedin.php?utm_source=Footer&utm_medium=Em
ail&utm_content=PuneLinkedIn&utm_campaign=Corporate>
 

 
 

Find out more about our services: PubConnect
<http://www.pubmatic.com/pubconnect.php?utm_source=Footer&utm_medium=Email&u
tm_content=PunePubConnect&utm_campaign=Corporate>  | PubDirect
<http://www.pubmatic.com/pubdirect.php?utm_source=Footer&utm_medium=Email&ut
m_content=PunePubDirect&utm_campaign=Corporate>  | Solutions for Media
Buyers 
<http://www.pubmatic.com/media-buyers-overview.php?utm_source=Footer&utm_med
ium=Email&utm_content=PuneMediaBuyers&utm_campaign=Corporate>
 What's New: PubMatic Launches Mobile SDK for app developers.
<http://www.pubmatic.com/press/2013/PubMatic-Launches-Self-Service-Open-SDK.
php?utm_source=Footer&utm_medium=Email&utm_content=PuneSDKLink&utm_campaign=
Corporate> 
 Latest Research: Real-Time Bidding in the United States and Worldwide,
2010-2017 (Q4 2013)
<http://www.pubmatic.com/reports-and-whitepapers.php?utm_source=Footer&utm_m
edium=Email&utm_content=PuneReportsLink&utm_campaign=Corporate>
 Recent Videos: Ad Revenue 6
<http://www.pubmatic.com/ar6_videos.php?utm_source=Footer&utm_medium=Email&u
tm_content=NYCAR6Link&utm_campaign=Corporate>  | Ad Revenue Europe
<http://www.pubmatic.com/arE_videos.php?utm_source=Footer&utm_medium=Email&u
tm_content=NYCAR6Link&utm_campaign=Corporate>
 
 



-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message