phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: Call queue too small
Date Mon, 26 Mar 2018 21:39:54 GMT
Martin,

Might you be able to upgrade to a newer version of HDP (and implicitly, 
Phoenix)? The default HBase call queue size is 1GB which implies that 
things are getting quite backed up for you to see an error like that.

I'd confidently venture a guess that UPSERT SELECT logic is better in 
newer versions of the code.

Otherwise, you're in for a treat of trying to understand if there's some 
kind of deadlock going on server-side which is preventing the accepted 
RPCs from being worked off in a reasonable amount of time.

On 3/26/18 2:18 AM, Hengesbach, Martin wrote:
> Hi,
> 
> I tried to copy round about 1 million records from one table to another 
> with this sql:
> 
> UPSERT INTO New_Table (ID, data) SELECT ID, data FROM OLD_Table WHERE id 
>  >= '2010000000' AND ID < '2011000000';
> 
> With smaller number of records it works, but with this sql I get the 
> following exception:
> 
> /18/03/26 07:54:02 WARN client.AsyncProcess: #1, table=New_Table, 
> attempt=35/35 failed=28363ops, last exception: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.RpcServer$CallQueueTooBigException):

> *Call queue is full* on /x.x.x.x:16020, is 
> *hbase.ipc.server.max.callqueue.size too small?* on 
> dn13,16020,1521542835853, tracking started Mon Mar 26 07:44:16 CEST 
> 2018; not retrying 28363 - final failure/
> 
> /Error: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
> Failed 28363 actions: RemoteWithExtrasException: 28363 times, 
> (state=,code=0)/
> 
> /org.apache.phoenix.execute.CommitException: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
> Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
> 
> /        at 
> org.apache.phoenix.execute.MutationState.send(MutationState.java:985)/
> 
> /        at 
> org.apache.phoenix.execute.MutationState.send(MutationState.java:1327)/
> 
> /        at 
> org.apache.phoenix.execute.MutationState.commit(MutationState.java:1159)/
> 
> /        at 
> org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:528)/
> 
> /        at 
> org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.java:525)/
> 
> /        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
> 
> /        at 
> org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.java:525)/
> 
> /        at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:346)/
> 
> /        at 
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)/
> 
> /        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
> 
> /        at 
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:329)/
> 
> /        at 
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)/
> 
> /        at sqlline.Commands.execute(Commands.java:822)/
> 
> /        at sqlline.Commands.sql(Commands.java:732)/
> 
> /        at sqlline.SqlLine.dispatch(SqlLine.java:808)/
> 
> /        at sqlline.SqlLine.begin(SqlLine.java:681)/
> 
> /        at sqlline.SqlLine.start(SqlLine.java:398)/
> 
> /        at sqlline.SqlLine.main(SqlLine.java:292)/
> 
> /Caused by: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
> Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
> 
> /        at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:234)/
> 
> /        at 
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:214)/
> 
> /        at 
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1623)/
> 
> /        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:909)/
> 
> /        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:923)/
> 
> /        at 
> org.apache.phoenix.execute.MutationState.send(MutationState.java:952)/
> 
> /        ... 17 more/
> 
> I have searched a lot for correct / better values for 
> */hbase.ipc.server.max.callqueue.size /*but I don’t find anything. And 
> in our configuration there is no entry for that.
> 
> I tried this on a Hortonworks 2.5.3 with Phoenix 4.7.
> 
> Any ideas where I must tune to get this sql running?
> 
> Thank’s in advance
> 
> Martin
> 
> 
> 
> ------------------------------------------------------------------------
> 
> FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
> Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim 
> HRB 101892.
> Geschäftsführerin: Sabine Brünger-Weilandt.
> Vorsitzender des Aufsichtsrats: MinDirig Dr. Stefan Luther.
> 

Mime
View raw message