phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nathan Davis <nathan.da...@salesforce.com>
Subject Re: UnknowScanner/ScannerTimeoutException using IndexTool MR
Date Wed, 17 Aug 2016 15:04:03 GMT
One additional problem here (perhaps the more important issue): even though
the MR job failed because of this exception, the underlying index table
still got set to ACTIVE. That I think is a bug.
I'm using v4.7 against HBase 1.2.1 in EMR.

On Wed, Aug 17, 2016 at 10:26 AM, Nathan Davis <nathan.davis@salesforce.com>
wrote:

> Hi All,
> I'm getting the following error (sorry, it is the full error stack) when I
> run the IndexTool MR job to populate an index I created with ASYNC. I have
> been able to use IndexTool successfully previously.
>
> 2016-08-17 13:30:48,024 INFO  [main] mapreduce.Job: Task Id :
>> attempt_1471372816200_0005_m_000051_0, Status : FAILED
>> Error: java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException:
>> 60209ms passed since the last invocation, timeout is currently set to 60000
>> at org.apache.phoenix.mapreduce.PhoenixRecordReader.nextKeyValue(
>> PhoenixRecordReader.java:159)
>> at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.
>> nextKeyValue(MapTask.java:565)
>> at org.apache.hadoop.mapreduce.task.MapContextImpl.
>> nextKeyValue(MapContextImpl.java:80)
>> at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.
>> nextKeyValue(WrappedMapper.java:91)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>> UserGroupInformation.java:1657)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>> Caused by: org.apache.phoenix.exception.PhoenixIOException: 60209ms
>> passed since the last invocation, timeout is currently set to 60000
>> at org.apache.phoenix.util.ServerUtil.parseServerException(
>> ServerUtil.java:111)
>> at org.apache.phoenix.iterate.ScanningResultIterator.next(
>> ScanningResultIterator.java:65)
>> at org.apache.phoenix.iterate.TableResultIterator.next(
>> TableResultIterator.java:110)
>> at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(
>> LookAheadResultIterator.java:47)
>> at org.apache.phoenix.iterate.LookAheadResultIterator.next(
>> LookAheadResultIterator.java:67)
>> at org.apache.phoenix.iterate.RoundRobinResultIterator$
>> RoundRobinIterator.next(RoundRobinResultIterator.java:309)
>> at org.apache.phoenix.iterate.RoundRobinResultIterator.next(
>> RoundRobinResultIterator.java:97)
>> at org.apache.phoenix.jdbc.PhoenixResultSet.next(
>> PhoenixResultSet.java:778)
>> at org.apache.phoenix.mapreduce.PhoenixRecordReader.nextKeyValue(
>> PhoenixRecordReader.java:152)
>> ... 11 more
>> Caused by: org.apache.hadoop.hbase.client.ScannerTimeoutException:
>> 60209ms passed since the last invocation, timeout is currently set to 60000
>> at org.apache.hadoop.hbase.client.ClientScanner.
>> loadCache(ClientScanner.java:438)
>> at org.apache.hadoop.hbase.client.ClientScanner.next(
>> ClientScanner.java:370)
>> at org.apache.phoenix.iterate.ScanningResultIterator.next(
>> ScanningResultIterator.java:55)
>> ... 18 more
>> Caused by: org.apache.hadoop.hbase.UnknownScannerException:
>> org.apache.hadoop.hbase.UnknownScannerException: Name: 149, already
>> closed?
>> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> scan(RSRpcServices.java:2374)
>> at org.apache.hadoop.hbase.protobuf.generated.
>> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
>> RpcExecutor.java:133)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>> at java.lang.Thread.run(Thread.java:745)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(
>> NativeConstructorAccessorImpl.java:57)
>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
>> DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>> at org.apache.hadoop.ipc.RemoteException.instantiateException(
>> RemoteException.java:106)
>> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(
>> RemoteException.java:95)
>> at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(
>> ProtobufUtil.java:329)
>> at org.apache.hadoop.hbase.client.ScannerCallable.call(
>> ScannerCallable.java:262)
>> at org.apache.hadoop.hbase.client.ScannerCallable.call(
>> ScannerCallable.java:64)
>> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(
>> RpcRetryingCaller.java:200)
>> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$
>> RetryingRPC.call(ScannerCallableWithReplicas.java:360)
>> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$
>> RetryingRPC.call(ScannerCallableWithReplicas.java:334)
>> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(
>> RpcRetryingCaller.java:126)
>> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService
>> $QueueingFuture.run(ResultBoundedCompletionService.java:65)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.
>> apache.hadoop.hbase.UnknownScannerException): org.apache.hadoop.hbase.UnknownScannerException:
>> Name: 149, already closed?
>> at org.apache.hadoop.hbase.regionserver.RSRpcServices.
>> scan(RSRpcServices.java:2374)
>> at org.apache.hadoop.hbase.protobuf.generated.
>> ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:33648)
>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(
>> RpcExecutor.java:133)
>> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
>> at java.lang.Thread.run(Thread.java:745)
>> at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(
>> RpcClientImpl.java:1268)
>> at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(
>> AbstractRpcClient.java:226)
>> at org.apache.hadoop.hbase.ipc.AbstractRpcClient$
>> BlockingRpcChannelImplementation.callBlockingMethod(
>> AbstractRpcClient.java:331)
>> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$
>> BlockingStub.scan(ClientProtos.java:34094)
>> at org.apache.hadoop.hbase.client.ScannerCallable.call(
>> ScannerCallable.java:219)
>> ... 9 more
>
>
>
> It seems like I need to set either/both of 'phoenix.query.timeoutMs'/'hbase.rpc.timeout'.
> But not sure how to configure those settings for the MR job's internal
> HBase client...
>
> Thanks for the help,
> -nathan
>
>

Mime
View raw message