phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hengesbach, Martin" <Martin.Hengesb...@fiz-Karlsruhe.de>
Subject AW: Call queue too small
Date Tue, 27 Mar 2018 05:34:30 GMT
Thanks Josh,

Until now I thought that newer HDP versions (compared to 2.5.3) only contained bugfixes for
Phoenix 4.7. So we decided to wait for a new HDP Version with Phoenix > 4.7. Am I wrong
about that? And, do you have an idea when there is a new HDP with Phoenix > 4.7?

Best regards
Martin


-----Ursprüngliche Nachricht-----
Von: Josh Elser [mailto:elserj@apache.org] 
Gesendet: Montag, 26. März 2018 23:40
An: user@phoenix.apache.org
Betreff: Re: Call queue too small

Martin,

Might you be able to upgrade to a newer version of HDP (and implicitly, Phoenix)? The default
HBase call queue size is 1GB which implies that things are getting quite backed up for you
to see an error like that.

I'd confidently venture a guess that UPSERT SELECT logic is better in newer versions of the
code.

Otherwise, you're in for a treat of trying to understand if there's some kind of deadlock
going on server-side which is preventing the accepted RPCs from being worked off in a reasonable
amount of time.

On 3/26/18 2:18 AM, Hengesbach, Martin wrote:
> Hi,
> 
> I tried to copy round about 1 million records from one table to 
> another with this sql:
> 
> UPSERT INTO New_Table (ID, data) SELECT ID, data FROM OLD_Table WHERE 
> id  >= '2010000000' AND ID < '2011000000';
> 
> With smaller number of records it works, but with this sql I get the 
> following exception:
> 
> /18/03/26 07:54:02 WARN client.AsyncProcess: #1, table=New_Table,
> attempt=35/35 failed=28363ops, last exception: 
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.RpcServer$CallQueueTooBigException):

> *Call queue is full* on /x.x.x.x:16020, is 
> *hbase.ipc.server.max.callqueue.size too small?* on 
> dn13,16020,1521542835853, tracking started Mon Mar 26 07:44:16 CEST 
> 2018; not retrying 28363 - final failure/
> 
> /Error: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
> Failed 28363 actions: RemoteWithExtrasException: 28363 times, 
> (state=,code=0)/
> 
> /org.apache.phoenix.execute.CommitException: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
> Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
> 
> /        at
> org.apache.phoenix.execute.MutationState.send(MutationState.java:985)/
> 
> /        at
> org.apache.phoenix.execute.MutationState.send(MutationState.java:1327)
> /
> 
> /        at
> org.apache.phoenix.execute.MutationState.commit(MutationState.java:115
> 9)/
> 
> /        at
> org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.jav
> a:528)/
> 
> /        at
> org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.jav
> a:525)/
> 
> /        at 
> org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
> 
> /        at
> org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.jav
> a:525)/
> 
> /        at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:
> 346)/
> 
> /        at
> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:
> 331)/
> 
> /        at 
> org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
> 
> /        at
> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatem
> ent.java:329)/
> 
> /        at
> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java
> :1440)/
> 
> /        at sqlline.Commands.execute(Commands.java:822)/
> 
> /        at sqlline.Commands.sql(Commands.java:732)/
> 
> /        at sqlline.SqlLine.dispatch(SqlLine.java:808)/
> 
> /        at sqlline.SqlLine.begin(SqlLine.java:681)/
> 
> /        at sqlline.SqlLine.start(SqlLine.java:398)/
> 
> /        at sqlline.SqlLine.main(SqlLine.java:292)/
> 
> /Caused by: 
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: 
> Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
> 
> /        at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(
> AsyncProcess.java:234)/
> 
> /        at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(As
> yncProcess.java:214)/
> 
> /        at
> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.get
> Errors(AsyncProcess.java:1623)/
> 
> /        at 
> org.apache.hadoop.hbase.client.HTable.batch(HTable.java:909)/
> 
> /        at 
> org.apache.hadoop.hbase.client.HTable.batch(HTable.java:923)/
> 
> /        at
> org.apache.phoenix.execute.MutationState.send(MutationState.java:952)/
> 
> /        ... 17 more/
> 
> I have searched a lot for correct / better values for 
> */hbase.ipc.server.max.callqueue.size /*but I don't find anything. And 
> in our configuration there is no entry for that.
> 
> I tried this on a Hortonworks 2.5.3 with Phoenix 4.7.
> 
> Any ideas where I must tune to get this sql running?
> 
> Thank's in advance
> 
> Martin
> 
> 
> 
> ----------------------------------------------------------------------
> --
> 
> FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
> Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim 
> HRB 101892.
> Geschäftsführerin: Sabine Brünger-Weilandt.
> Vorsitzender des Aufsichtsrats: MinDirig Dr. Stefan Luther.
> 
------------------------------------------------------------------------------

FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB 101892.
Geschäftsführerin: Sabine Brünger-Weilandt.
Vorsitzender des Aufsichtsrats: MinDirig Dr. Stefan Luther.


Mime
View raw message