phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krunal Varajiya <Krunal.Varaj...@ask.com>
Subject Re: Join create OOM with java heap space on phoenix client
Date Wed, 27 May 2015 20:55:45 GMT
Hey Maryann

Thanks for your reply! I understand it is difficult to access some stuff in China! I can wait
till you come back, meantime I will keep you posted on if we make any progress on this!

My comments are below in blue:

Have a fun time in China!
- Krunal


From: Maryann Xue <maryann.xue@gmail.com<mailto:maryann.xue@gmail.com>>
Reply-To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" <user@phoenix.apache.org<mailto:user@phoenix.apache.org>>
Date: Tuesday, May 26, 2015 at 5:45 PM
To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" <user@phoenix.apache.org<mailto:user@phoenix.apache.org>>
Subject: Re: Join create OOM with java heap space on phoenix client

Hi Krunal,

Sorry for the late reply. I have been on vacation.

1. Can you make sure that the connection/statement is closed after each run of your query
(even with exception)?

I think this is related to second issue mentioned below as it happens for join. I tried some
other queries they are working fine!

2. You might want to try switching the join tables in your query first by putting the larger
table as LHS, and if it still does not work, you can force a merge join by adding hint "USE_SORT_MERGE_JOIN".

I have tried changing lhs and rhs, but no luck. I have also tried "USE_SORT_MERGE_JOIN" but
for some reason it crashes region server while scanning result to local.
I have also tried changing some tuning parameters, but none of the settings worked!
One thing I am curious is why it is dumping data in local heap, I see millions of instances
for org.apache.phoenix.expression.literalexpression. Shouldn't it execute joins on server
side?
I tried increasing memory upto 64gb and still it fails! This is just one single connection
and only one query running at a time, I am not sure what will happen with multiple connection
and multiple queries!
And if you see in example my data size for both LHS and RHS is not that big. These are sample
tables come with Phoenix.

Thanks,
Maryann

On Thursday, May 21, 2015, Krunal Varajiya <Krunal.Varajiya@ask.com<mailto:Krunal.Varajiya@ask.com>>
wrote:
Hi

I have 2 issues with phoenix client:

  1.  Heap memory is not cleanup after each query is finished. So, it keeps increasing every
time when we submit new query.
  2.  I am try to do a normal join operation on two tables but getting exception. Below is
the details:

These are some sample queries I tried:

  1.  select p1.host, count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where
p1.host = p2.host group by p1.host;
  2.  select p1.host from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host =
p2.host group by p1.host;
  3.  select count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host =
p2.host group by p1.host;

Here is explain plan:


explain  select count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host
= p2.host group by p1.host;

+------------------------------------------+

|                   PLAN                   |

+------------------------------------------+

| CLIENT 9-CHUNK PARALLEL 1-WAY FULL SCAN OVER PERFORMANCE_5000000 |

|     SERVER FILTER BY FIRST KEY ONLY      |

|     SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [HOST] |

| CLIENT MERGE SORT                        |

|     PARALLEL INNER-JOIN TABLE 0 (SKIP MERGE) |

|         CLIENT 18-CHUNK PARALLEL 1-WAY FULL SCAN OVER PERFORMANCE_25000000 |

|             SERVER FILTER BY FIRST KEY ONLY |

|     DYNAMIC SERVER FILTER BY HOST IN (P2.HOST) |

+------------------------------------------+

8 rows selected (0.127 seconds)

Phoenix client heap size is 16GB. ( noticed that above queries are dumping data in local heap,
I see millions of instances for org.apache.phoenix.expression.literalexpression)

and my exceptions are:

java.sql.SQLException: Encountered exception in sub plan [0] execution.

at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:156)

at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:235)

at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:226)

at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)

at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:225)

at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1066)

at sqlline.Commands.execute(Commands.java:822)

at sqlline.Commands.sql(Commands.java:732)

at sqlline.SqlLine.dispatch(SqlLine.java:808)

at sqlline.SqlLine.begin(SqlLine.java:681)

at sqlline.SqlLine.start(SqlLine.java:398)

at sqlline.SqlLine.main(SqlLine.java:292)

Caused by: java.sql.SQLException: java.util.concurrent.ExecutionException: java.lang.Exception:
java.lang.OutOfMemoryError: Java heap space

at org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:247)

at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:83)

at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:338)

at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:135)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.OutOfMemoryError:
Java heap space

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:206)

at org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:239)

... 7 more

Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space

at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:212)

at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)

... 4 more

Caused by: java.lang.OutOfMemoryError: Java heap space

May 20, 2015 4:58:01 PM ServerCommunicatorAdmin reqIncoming

WARNING: The server has decided to close this client connection.

15/05/20 16:56:43 WARN client.HTable: Error calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService
for row CSGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620)

at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577)

at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188)

at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.OutOfMemoryError: Java heap space

at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:907)

at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:902)

at com.google.protobuf.ByteString.newCodedBuilder(ByteString.java:898)

at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)

at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:81)

at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:57)

at org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService$Stub.addServerCache(ServerCachingProtos.java:3270)

at org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:204)

at org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:189)

at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1608)

... 4 more

0: jdbc:phoenix:prbhadoop004iad.io.askjeeves.> 15/05/20 16:56:43 WARN client.HTable: Error
calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService
for row EUGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00

java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620)

at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577)

at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188)

at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.OutOfMemoryError: Java heap space

15/05/20 16:59:37 WARN client.HConnectionManager$HConnectionImplementation: This client just
lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs
it

org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired

at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:403)

at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:321)

at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)

at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)


Error:  (state=,code=0)

java.sql.SQLFeatureNotSupportedException

at org.apache.phoenix.jdbc.PhoenixStatement.cancel(PhoenixStatement.java:958)

at sqlline.DispatchCallback.forceKillSqlQuery(DispatchCallback.java:83)

at sqlline.SqlLine.begin(SqlLine.java:695)

at sqlline.SqlLine.start(SqlLine.java:398)

at sqlline.SqlLine.main(SqlLine.java:292)


Can someone please help?


Thanks!

Mime
View raw message