phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanooj Padmakumar <p.san...@gmail.com>
Subject Re: RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1 time
Date Sat, 27 Aug 2016 17:11:30 GMT
Hi All,

I am getting the same exception , this time when running a Phoenix MR (
https://phoenix.apache.org/phoenix_mr.html) .. The MR works just fine if I
am doing a select with limit to some 10 rows.. but when I do the same with
a lot of data, I start getting the below exception after 66% of reducer..

6/08/27 10:04:35 INFO mapreduce.Job: Task Id :
attempt_1471862728027_0103_r_000036_0, Status : FAILED
Error: java.lang.RuntimeException:
org.apache.phoenix.execute.CommitException:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
2000 actions: IOException: 2000 times,
        at
org.apache.phoenix.mapreduce.PhoenixRecordWriter.close(PhoenixRecordWriter.java:62)
        at
org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.close(ReduceTask.java:550)
         ............
Caused by:
org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
2000 actions: IOException: 2000 times,
        at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:227)
        at
org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$1700(AsyncProcess.java:207)
        at
org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1568)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
        at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
        at
org.apache.phoenix.execute.MutationState.commit(MutationState.java:444)
        ... 13 more

When I look at the failed reducers log I am seeing a lot of logs like
these..

2016-08-27 10:07:05,834 ERROR [main]
org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0
location for {"totalColumns":25,"families":{"p":[{"timestamp":1472317624237,"tag":[],"qualifier"

But as I said with very less number of rows in the select query, the
MR works just fine and data is populated alright ? Any Phoenix/Hbase
parameters

that is should look into ?

Thanks

Sanooj Padmakumar





On Wed, Aug 24, 2016 at 11:26 AM, Sanooj Padmakumar <p.sanooj@gmail.com>
wrote:

> Hi All,
>
> I get this error one one of the nodes where we have an application
> running. It comes only after a certain duration and once the application is
> restarted things will start working normally..Any inputs as to why this
> might be happening will be of great help.
>
>
> org.springframework.jdbc.UncategorizedSQLException:
> PreparedStatementCallback; uncategorized SQLException for SQL [<<query here
> >>> ]; SQL state [null]; error code [0]; org.apache.hadoop.hbase.client.
> RetriesExhaustedWithDetailsException: Failed 1 action: IOException: 1
> time, ; nested exception is org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action: IOException: 1 time,
>         at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTr
> anslator.translate(AbstractFallbackSQLExceptionTranslator.java:84)
>         at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTr
> anslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
>         at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTr
> anslator.translate(AbstractFallbackSQLExceptionTranslator.java:81)
>         at org.springframework.jdbc.core.JdbcTemplate.execute(
> JdbcTemplate.java:660)
>         at org.springframework.jdbc.core.JdbcTemplate.update(
> JdbcTemplate.java:909)
>         at org.springframework.jdbc.core.JdbcTemplate.update(
> JdbcTemplate.java:933)
>         at org.springframework.jdbc.core.namedparam.
> NamedParameterJdbcTemplate.update(NamedParameterJdbcTemplate.java:313)
>         at Caused by: org.apache.phoenix.execute.CommitException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action: IOException: 1 time,
>         at org.apache.phoenix.execute.MutationState.commit(
> MutationState.java:473)
>         at org.apache.phoenix.jdbc.PhoenixConnection$3.call(
> PhoenixConnection.java:472)
>         at org.apache.phoenix.jdbc.PhoenixConnection$3.call(
> PhoenixConnection.java:469)
>         at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>         at org.apache.phoenix.jdbc.PhoenixConnection.commit(
> PhoenixConnection.java:469)
>         at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
> PhoenixStatement.java:323)
>         at org.apache.phoenix.jdbc.PhoenixStatement$2.call(
> PhoenixStatement.java:312)
>         at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>         at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(
> PhoenixStatement.java:310)
>         at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(
> PhoenixPreparedStatement.java:200)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeUpdate(DelegatingPreparedStatement.java:105)
>         at org.springframework.jdbc.core.JdbcTemplate$2.
> doInPreparedStatement(JdbcTemplate.java:916)
>         at org.springframework.jdbc.core.JdbcTemplate$2.
> doInPreparedStatement(JdbcTemplate.java:909)
>         at org.springframework.jdbc.core.JdbcTemplate.execute(
> JdbcTemplate.java:644)
>         ... 11 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
> Failed 1 action: IOException: 1 time,
>         at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.makeException(AsyncProcess.java:227)
>         at org.apache.hadoop.hbase.client.AsyncProcess$
> BatchErrors.access$1700(AsyncProcess.java:207)
>         at org.apache.hadoop.hbase.client.AsyncProcess$
> AsyncRequestFutureImpl.getErrors(AsyncProcess.java:1568)
>         at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1003)
>         at org.apache.hadoop.hbase.client.HTable.batch(HTable.java:1017)
>         at org.apache.phoenix.execute.MutationState.commit(
> MutationState.java:444)
>         ... 25 more
>
>
> --
> Thanks,
> Sanooj Padmakumar
>
>


-- 
Thanks,
Sanooj Padmakumar

Mime
View raw message