phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: AW: Call queue too small
Date Tue, 27 Mar 2018 14:46:21 GMT
I'd prefer not to get into HDP-specific details on Apache mailing lists 
to save those who don't care about HDP. If you have follow up questions, 
I'd encourage you to use Hortonworks' community forum for HDP specific 
question.

The short answer is: no, new HDP versions do not just contain bug-fixes, 
however your problem may also be fixed by an aforementioned bug-fix.

On 3/27/18 1:34 AM, Hengesbach, Martin wrote:
> Thanks Josh,
> 
> Until now I thought that newer HDP versions (compared to 2.5.3) only contained bugfixes
for Phoenix 4.7. So we decided to wait for a new HDP Version with Phoenix > 4.7. Am I wrong
about that? And, do you have an idea when there is a new HDP with Phoenix > 4.7?
> 
> Best regards
> Martin
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Josh Elser [mailto:elserj@apache.org]
> Gesendet: Montag, 26. März 2018 23:40
> An: user@phoenix.apache.org
> Betreff: Re: Call queue too small
> 
> Martin,
> 
> Might you be able to upgrade to a newer version of HDP (and implicitly, Phoenix)? The
default HBase call queue size is 1GB which implies that things are getting quite backed up
for you to see an error like that.
> 
> I'd confidently venture a guess that UPSERT SELECT logic is better in newer versions
of the code.
> 
> Otherwise, you're in for a treat of trying to understand if there's some kind of deadlock
going on server-side which is preventing the accepted RPCs from being worked off in a reasonable
amount of time.
> 
> On 3/26/18 2:18 AM, Hengesbach, Martin wrote:
>> Hi,
>>
>> I tried to copy round about 1 million records from one table to
>> another with this sql:
>>
>> UPSERT INTO New_Table (ID, data) SELECT ID, data FROM OLD_Table WHERE
>> id  >= '2010000000' AND ID < '2011000000';
>>
>> With smaller number of records it works, but with this sql I get the
>> following exception:
>>
>> /18/03/26 07:54:02 WARN client.AsyncProcess: #1, table=New_Table,
>> attempt=35/35 failed=28363ops, last exception:
>> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ipc.RpcServer$CallQueueTooBigException):
>> *Call queue is full* on /x.x.x.x:16020, is
>> *hbase.ipc.server.max.callqueue.size too small?* on
>> dn13,16020,1521542835853, tracking started Mon Mar 26 07:44:16 CEST
>> 2018; not retrying 28363 - final failure/
>>
>> /Error:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 28363 actions: RemoteWithExtrasException: 28363 times,
>> (state=,code=0)/
>>
>> /org.apache.phoenix.execute.CommitException:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
>>
>> /        at
>> org.apache.phoenix.execute.MutationState.send(MutationState.java:985)/
>>
>> /        at
>> org.apache.phoenix.execute.MutationState.send(MutationState.java:1327)
>> /
>>
>> /        at
>> org.apache.phoenix.execute.MutationState.commit(MutationState.java:115
>> 9)/
>>
>> /        at
>> org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.jav
>> a:528)/
>>
>> /        at
>> org.apache.phoenix.jdbc.PhoenixConnection$3.call(PhoenixConnection.jav
>> a:525)/
>>
>> /        at
>> org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
>>
>> /        at
>> org.apache.phoenix.jdbc.PhoenixConnection.commit(PhoenixConnection.jav
>> a:525)/
>>
>> /        at
>> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:
>> 346)/
>>
>> /        at
>> org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:
>> 331)/
>>
>> /        at
>> org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)/
>>
>> /        at
>> org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatem
>> ent.java:329)/
>>
>> /        at
>> org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java
>> :1440)/
>>
>> /        at sqlline.Commands.execute(Commands.java:822)/
>>
>> /        at sqlline.Commands.sql(Commands.java:732)/
>>
>> /        at sqlline.SqlLine.dispatch(SqlLine.java:808)/
>>
>> /        at sqlline.SqlLine.begin(SqlLine.java:681)/
>>
>> /        at sqlline.SqlLine.start(SqlLine.java:398)/
>>
>> /        at sqlline.SqlLine.main(SqlLine.java:292)/
>>
>> /Caused by:
>> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
>> Failed 28363 actions: RemoteWithExtrasException: 28363 times,/
>>
>> /        at
>> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(
>> AsyncProcess.java:234)/
>>
>> /        at
>> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(As
>> yncProcess.java:214)/
>>
>> /        at
>> org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.get
>> Errors(AsyncProcess.java:1623)/
>>
>> /        at
>> org.apache.hadoop.hbase.client.HTable.batch(HTable.java:909)/
>>
>> /        at
>> org.apache.hadoop.hbase.client.HTable.batch(HTable.java:923)/
>>
>> /        at
>> org.apache.phoenix.execute.MutationState.send(MutationState.java:952)/
>>
>> /        ... 17 more/
>>
>> I have searched a lot for correct / better values for
>> */hbase.ipc.server.max.callqueue.size /*but I don't find anything. And
>> in our configuration there is no entry for that.
>>
>> I tried this on a Hortonworks 2.5.3 with Phoenix 4.7.
>>
>> Any ideas where I must tune to get this sql running?
>>
>> Thank's in advance
>>
>> Martin
>>
>>
>>
>> ----------------------------------------------------------------------
>> --
>>
>> FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
>> Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim
>> HRB 101892.
>> Geschäftsführerin: Sabine Brünger-Weilandt.
>> Vorsitzender des Aufsichtsrats: MinDirig Dr. Stefan Luther.
>>
> ------------------------------------------------------------------------------
> 
> FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH.
> Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB 101892.
> Geschäftsführerin: Sabine Brünger-Weilandt.
> Vorsitzender des Aufsichtsrats: MinDirig Dr. Stefan Luther.
> 

Mime
View raw message