phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maryann Xue <maryann....@gmail.com>
Subject Re: Join create OOM with java heap space on phoenix client
Date Wed, 27 May 2015 00:45:21 GMT
Hi Krunal,

Sorry for the late reply. I have been on vacation.

1. Can you make sure that the connection/statement is closed after each run
of your query (even with exception)?

2. You might want to try switching the join tables in your query first by
putting the larger table as LHS, and if it still does not work, you can
force a merge join by adding hint "USE_SORT_MERGE_JOIN".


Thanks,
Maryann

On Thursday, May 21, 2015, Krunal Varajiya <Krunal.Varajiya@ask.com> wrote:

>  Hi
>
>  I have 2 issues with phoenix client:
>
>    1. Heap memory is not cleanup after each query is finished. So, it
>    keeps increasing every time when we submit new query.
>    2. I am try to do a normal join operation on two tables but getting
>    exception. Below is the details:
>
>  These are some sample queries I tried:
>
>    1. select p1.host, count(1) from PERFORMANCE_5000000 p1,
>    PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host;
>    2. select p1.host from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2
>    where p1.host = p2.host group by p1.host;
>    3. select count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000
>    p2 where p1.host = p2.host group by p1.host;
>
>  Here is explain plan:
>
>  explain  select count(1) from PERFORMANCE_5000000 p1,
> PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host;
>
> +------------------------------------------+
>
> |                   PLAN                   |
>
> +------------------------------------------+
>
> | CLIENT 9-CHUNK PARALLEL 1-WAY FULL SCAN OVER PERFORMANCE_5000000 |
>
> |     SERVER FILTER BY FIRST KEY ONLY      |
>
> |     SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [HOST] |
>
> | CLIENT MERGE SORT                        |
>
> |     PARALLEL INNER-JOIN TABLE 0 (SKIP MERGE) |
>
> |         CLIENT 18-CHUNK PARALLEL 1-WAY FULL SCAN OVER
> PERFORMANCE_25000000 |
>
> |             SERVER FILTER BY FIRST KEY ONLY |
>
> |     DYNAMIC SERVER FILTER BY HOST IN (P2.HOST) |
>
> +------------------------------------------+
>
> 8 rows selected (0.127 seconds)
>
>   Phoenix client heap size is 16GB. ( noticed that above queries are
> dumping data in local heap, I see millions of instances for
> org.apache.phoenix.expression.literalexpression)
>
> and my exceptions are:
>
> java.sql.SQLException: Encountered exception in sub plan [0] execution.
>
> at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:156)
>
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:235)
>
> at
> org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:226)
>
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>
> at
> org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:225)
>
> at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1066)
>
> at sqlline.Commands.execute(Commands.java:822)
>
> at sqlline.Commands.sql(Commands.java:732)
>
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
>
> at sqlline.SqlLine.begin(SqlLine.java:681)
>
> at sqlline.SqlLine.start(SqlLine.java:398)
>
> at sqlline.SqlLine.main(SqlLine.java:292)
>
> Caused by: java.sql.SQLException: java.util.concurrent.ExecutionException:
> java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
>
> at
> org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:247)
>
> at
> org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:83)
>
> at
> org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:338)
>
> at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:135)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.util.concurrent.ExecutionException: java.lang.Exception:
> java.lang.OutOfMemoryError: Java heap space
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
>
> at
> org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:239)
>
> ... 7 more
>
> Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:212)
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)
>
> ... 4 more
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
>
> May 20, 2015 4:58:01 PM ServerCommunicatorAdmin reqIncoming
>
> WARNING: The server has decided to close this client connection.
>
> 15/05/20 16:56:43 WARN client.HTable: Error calling coprocessor service
> org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService
> for row CSGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
>
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620)
>
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577)
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188)
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
>
> at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:907)
>
> at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:902)
>
> at com.google.protobuf.ByteString.newCodedBuilder(ByteString.java:898)
>
> at
> com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
>
> at
> org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:81)
>
> at
> org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:57)
>
> at
> org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService$Stub.addServerCache(ServerCachingProtos.java:3270)
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:204)
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:189)
>
> at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1608)
>
> ... 4 more
>
> 0: jdbc:phoenix:prbhadoop004iad.io.askjeeves.> 15/05/20 16:56:43 WARN
> client.HTable: Error calling coprocessor service
> org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService
> for row EUGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
>
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java
> heap space
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620)
>
> at
> org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577)
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188)
>
> at
> org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.lang.OutOfMemoryError: Java heap space
>
> 15/05/20 16:59:37 WARN
> client.HConnectionManager$HConnectionImplementation: This client just lost
> it's session with ZooKeeper, closing it. It will be recreated next time
> someone needs it
>
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired
>
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:403)
>
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:321)
>
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
>
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
>
>
>  Error:  (state=,code=0)
>
> java.sql.SQLFeatureNotSupportedException
>
> at
> org.apache.phoenix.jdbc.PhoenixStatement.cancel(PhoenixStatement.java:958)
>
> at sqlline.DispatchCallback.forceKillSqlQuery(DispatchCallback.java:83)
>
> at sqlline.SqlLine.begin(SqlLine.java:695)
>
> at sqlline.SqlLine.start(SqlLine.java:398)
>
> at sqlline.SqlLine.main(SqlLine.java:292)
>
>
>  Can someone please help?
>
>
>  Thanks!
>

Mime
View raw message