phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sunfl@certusnet.com.cn" <su...@certusnet.com.cn>
Subject Mapreduce job exception when using Apache Spark to query phoenix tables
Date Sun, 04 Jan 2015 05:31:05 GMT
Hi,all
When trying to complete the mapreduce job over phoenix table using Apache Spark, we got the
following error. Guess that it is caused by hbase client scanner timeout exeption? 
Do we need to configure something for hbase-site.xml?Thanks for applying any available advice.

By the way, anyone could explain to me about the current mapreduce framework applied by PHOENIX-1454?Noting
that the design aims to transform queryplan to hbase scan instead of 
operating hfiles? Cause we are researching about using HBaseScan and HBaseSnapshotScan with
Apache Spark to read hbase tables more efficiently. And these approaches are actually reading
from hfiles.

Anyone intrested in designing and implementing about the snapshotscan for phoenix table?Or
are there any previous discussions about these kinds of threads?

Thanks,
Sun.

14/12/31 16:59:46 ERROR Executor: Exception in task 17.0 in stage 137.0 (TID 3356) 
java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: 61234ms passed
since the last invocation, timeout is currently set to 60000 
at com.google.common.base.Throwables.propagate(Throwables.java:160) 
at org.apache.phoenix.pig.hadoop.PhoenixRecordReader.nextKeyValue(PhoenixRecordReader.java:138)

at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:145) 
at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39) 
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) 
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327) 
at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:156)

at org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$7.apply(Aggregate.scala:151)

at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601) 
at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:601) 
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) 
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) 
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) 
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) 
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) 
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) 
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) 
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) 
at org.apache.spark.scheduler.Task.run(Task.scala:56) 
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:744) 
Caused by: org.apache.phoenix.exception.PhoenixIOException: 61234ms passed since the last
invocation, timeout is currently set to 60000 
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:108)
at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:52)

at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:104) 
at org.apache.phoenix.iterate.LookAheadResultIterator$1.advance(LookAheadResultIterator.java:47)

at org.apache.phoenix.iterate.LookAheadResultIterator.init(LookAheadResultIterator.java:59)

at org.apache.phoenix.iterate.LookAheadResultIterator.peek(LookAheadResultIterator.java:73)

at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:100)

at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117) 
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:734) 
at org.apache.phoenix.pig.hadoop.PhoenixRecordReader.nextKeyValue(PhoenixRecordReader.java:131)

... 21 more 
Caused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 61234ms passed since the
last invocation, timeout is currently set to 60000 
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:371) 
at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:47)

... 29 more 
Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException:
Name: 7571, already closed? 
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3145) 
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29587)

at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031) 
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) 
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) 
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) 
at java.lang.Thread.run(Thread.java:722) 

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) 
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) 
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:304)

at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:204) 
at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:59) 
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)

at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)




CertusNet 

Mime
View raw message