phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Mahonin <jmaho...@gmail.com>
Subject Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1
Date Thu, 14 Jan 2016 20:33:23 GMT
Hi Li,

I've not seen this error myself, though some searching returns a possible
root cause:
http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=nDkSWyorpt1redB9DQwjBHVVLfnTqw@mail.gmail.com%3E

Could you file a JIRA ticket for this please? It's possible the MapReduce
(and by extension, Spark) integration isn't handling this gracefully.

Also, if possible, any information you have about your environment, table
sizes, read/write workload, etc. would be helpful as well.

Thanks!

Josh

On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <gaol@marinsoftware.com> wrote:

> Hi Phoenix users,
>
>
> We are seeing occasionally (maybe 30 ~ 50% of time) this
> StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix
> 4.5.1-HBase 1.0.
>
> Not sure how to troubleshoot such issue. Any hints and insights are
> greatly appreciated!
>
> Thanks,
>
> Li
>
> PS: The following are the exception stack trace:
>
> 16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw
> exception: org.apache.spark.SparkException: Job aborted due to stage
> failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost
> task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere):
> java.lang.RuntimeException:
> org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108
> (XCL08): Cache of region boundaries are out of date.
>
> at com.google.common.base.Throwables.propagate(Throwables.java:156)
>
> at
> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)
>
> at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)
>
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
>
> at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
>
> at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
>
> at org.apache.spark.scheduler.Task.run(Task.scala:70)
>
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
> at java.lang.Thread.run(Thread.java:745)
>
> Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException:
> ERROR 1108 (XCL08): Cache of region boundaries are out of date.
>
> at
> org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304)
>
> at
> org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)
>
> at
> org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)
>
> at
> org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)
>
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)
>
> at
> org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70)
>
> at
> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
>
> at
> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
>
> at
> org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111)
>
> ... 18 more
>

Mime
View raw message