phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantinos Kougios <kostas.koug...@googlemail.com>
Subject Re: does phoenix+hbase work for tables larger than a few GB?
Date Thu, 01 Oct 2015 16:01:52 GMT
Hi Anil, thanks again for the useful info. My replies inline as well:

On 30/09/15 23:45, anil gupta wrote:
> Please find my reply inline.
>
> On Wed, Sep 30, 2015 at 3:29 PM, Konstantinos Kougios 
> <kostas.kougios@googlemail.com <mailto:kostas.kougios@googlemail.com>> 
> wrote:
>
>     Thanks for the reply and the useful information Anil.
>
>     I am aware of the difficulties of distributed joins and
>     aggregations and that phoenix is a layer on top of hbase. It would
>     be great if it could be configured to run the queries, even if it
>     takes a lot of time for the queries to complete.
>
> Anil: I think, it is doable. But, this might require little bit of hit 
> & trial with HBase and Phoenix conf. I would start with increasing 
> HBase and Phoenix timeouts.
I believe I did that and now my queries run for a long time without 
timeout exceptions (before failing):

   zookeeper.session.timeout = 9000000 # not sure if this should be so high
   hbase.client.pause = 10000
   hbase.client.retries.number = 2
   hbase.client.scanner.timeout.period = 6000000
   phoenix.query.timeoutMs = 60000000
   phoenix.query.keepAliveMs = 60000
   hbase.client.operation.timeout = 60000000
   hbase.client.backpressure.enabled = true
   hbase.client.retries.number = 1
   hbase.rpc.timeout = 6000000

failure might all be due to

2015-10-01 09:13:22,002 WARN 
[B.defaultRpcServer.handler=14,queue=2,port=16020] ipc.RpcServer: 
(responseTooSlow): 
{"call":"Scan(org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ScanRequest)","starttimems":1443685974126,"responsesize":10,"method":"Scan","processingtimems":1227740,"client":"192.168.0.11:34656","queuetimems":0,"class":"HRegionServer"}
2015-10-01 09:13:22,296 ERROR 
[IndexRpcServer.handler=15,queue=0,port=16020] ipc.RpcServer: Unexpected 
throwable object
java.lang.ArrayIndexOutOfBoundsException
         at org.apache.hadoop.hbase.util.Bytes.putBytes(Bytes.java:299)
         at 
org.apache.hadoop.hbase.KeyValue.createByteArray(KeyValue.java:1102)
         at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:650)
         at org.apache.hadoop.hbase.KeyValue.<init>(KeyValue.java:578)
         at 
org.apache.phoenix.util.KeyValueUtil.newKeyValue(KeyValueUtil.java:63)
         at 
org.apache.phoenix.cache.aggcache.SpillManager.getAggregators(SpillManager.java:204)
         at 
org.apache.phoenix.cache.aggcache.SpillManager.toCacheEntry(SpillManager.java:243)
         at 
org.apache.phoenix.cache.aggcache.SpillableGroupByCache$EntryIterator.next(SpillableGroupByCache.java:285)
         at 
org.apache.phoenix.cache.aggcache.SpillableGroupByCache$EntryIterator.next(SpillableGroupByCache.java:261)
         at 
org.apache.phoenix.cache.aggcache.SpillableGroupByCache$2.next(SpillableGroupByCache.java:364)
         at 
org.apache.phoenix.coprocessor.BaseRegionScanner.next(BaseRegionScanner.java:40)
         at 
org.apache.phoenix.coprocessor.BaseRegionScanner.nextRaw(BaseRegionScanner.java:60)
         at 
org.apache.phoenix.coprocessor.DelegateRegionScanner.nextRaw(DelegateRegionScanner.java:77)
         at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2395)
         at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
         at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
         at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
         at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
         at 
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
         at java.lang.Thread.run(Thread.java:745)

The exception appears frequently in the logs when I run the query.
Also an other issue I've noticed is that sqlline.py starts using the cpu 
quite a lot after running my query for a while. Is't that just the 
client and should just wait for the server to respond? I believe it is 
due of using all it's memory (4GB). Also I think it is writing a lot of 
files under /tmp ... is it doing part of the query???

Thanks



>
>     I got mainly 2 tables of 170GB and 550GB. Aggregation queries on
>     both fail and even make region servers crash (there is no info in
>     the logs and still don't know why. My server proved to be rock
>     stable so far on other things but you never know).
>
> Anil: RS should not crash. Are you doing heavy writes along with full 
> table scans at same time? In one of your email, i saw stack trace 
> regarding Region  split and compactions?
I used to, but now I am just trying to run a query without any other 
load on the server. Or create an index.
>
>
>     I am doing full table scans only because so far I was unable to
>     create the indexes. I tried async indexes too with the map reduce
>     job to create them but it runs extremely slowly.
>
> Anil: This doesnt not sounds good. I haven't use those yet. So, i wont 
> be able to help debug the problem. Hopefully, someone else will be 
> able to chime in.
>
>
>     In theory full table scans are possible with hbase, so even if it
>     was slow it shouldn't fail.
>
> Anil: IMO, if you are doing full table scans, then maybe you should 
> turn off blockCache for those queries. Basically, there is a lot of 
> cache churn due to full table scans. Cache churn will lead to JVM GC's.
>
>
>     My setup is a 64GB AMD opteron server with 16 cores. 3 lxc virtual
>     machines as region servers with Xmx8G, each running on a 3TB
>     7200rpm disk. So somehow I simulate 3x low spec servers with
>     enough ram.
>
>     Next thing I will try is give region servers 16GB of RAM. WIth 8GB
>     they seem to have some memory pressure and I see some slow GC's in
>     the logs.
>
> Anil: 16GB ram should help in some cases. Try to disable blockcache 
> for full table scans.
>
>
>     Cheers
>
>
>
>
>
>     On 30/09/15 21:18, anil gupta wrote:
>>     Hi Konstantinos,
>>     Please find my reply inline.
>>
>>     On Wed, Sep 30, 2015 at 12:10 PM, Konstantinos Kougios
>>     <kostas.kougios@googlemail.com
>>     <mailto:kostas.kougios@googlemail.com>> wrote:
>>
>>         Hi all,
>>
>>         I had various issues with big tables while experimenting the
>>         couple last weeks.
>>
>>         The thing that goes to my mind is that hbase (+phoenix) works
>>         only when there is a fairly powerful cluster and say 1/2 the
>>         data can fit into the combined servers memory and disks are
>>         fast (SSD?) as well. It doesn't seem to be able to work when
>>         tables are 2x as large as the memory allocated to region
>>         servers (frankly I think it is less)
>>
>>     Anil: Phoenix is just a SQL layer over HBase. From the query in
>>     your previous emails, it seems like you are doing full table
>>     scans with group by clauses. IMO, HBase is not a DB to be used
>>     for full table scans. If 90% of your use cases are small range
>>     scan or gets then HBase should work nicely with Terabytes of
>>     data. I have a 40 TB table in prod on 60 node cluster where every
>>     RS only has 16GB of heap. What kind of workload you are trying to
>>     run with HBase?
>>
>>
>>         Things that constantly fail:
>>
>>         - non-trivial queries on large tables (with group by, counts,
>>         joins) with region server out of memory errors or crashes
>>         without any reason for Xmx of 4G or 8G
>>
>>     Anil: Can you convert these queries into short range based scans?
>>     If you are always going to do full table scan, then maybe you
>>     need to use MR or Spark for those computation and then tune
>>     cluster for full table scans. Cluster tuning varies with full
>>     table scan workload.
>>
>>         - index creation on the same big tables. Those always fail I
>>         think around the point when hbase has to flush it's memory
>>         regions to the disk and couldn't find a solution
>>
>>         - spark jobs fail unless they are throttled to feed hbase
>>         with the data it can take . No backpressure?
>>
>>
>>         There were no replies to my emails regarding the issues,
>>         which makes me think there aren't solutions (or solutions are
>>         pretty hard to find and not many ppl know them).
>>
>>         So after 21 tweaks to the default config, I am still not able
>>         to operate it as a normal database.
>>
>>     Anil: HBase is actually not a normal RDBMS DB. Its a **keyvalue
>>     store**. Phoenix is providing a SQL layer using HBase API. So,
>>     user will need to deal with pros/cons of a key/value store.
>>
>>
>>         Should I start believing my config is all wrong or that
>>         hbase+phoenix is only working if there is a sufficiently
>>         powerful cluster to handle the data?
>>
>>     Anil: **As per my experience**, HBase+Phoenix will work nicely if
>>     you are doing keyvalue lookups and short range scans.
>>     I would suggest you to evaluate data model of HBase tables and
>>     try to convert queries to small range scan or lookups.
>>
>>
>>         I believe it is a great project and the functionality is
>>         really useful. What's lacking is 3 sample configs for 3
>>         different strength clusters.
>>
>>     Anil: I agree that guidance on configuration of HBase and Phoenix
>>     can be improved so that people can get going quickly.
>>
>>
>>         Thanks
>>
>>
>>
>>
>>     -- 
>>     Thanks & Regards,
>>     Anil Gupta
>
>
>
>
> -- 
> Thanks & Regards,
> Anil Gupta


Mime
View raw message