Hi Li,

I've not seen this error myself, though some searching returns a possible root cause:
http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=nDkSWyorpt1redB9DQwjBHVVLfnTqw@mail.gmail.com%3E

Could you file a JIRA ticket for this please? It's possible the MapReduce (and by extension, Spark) integration isn't handling this gracefully.

Also, if possible, any information you have about your environment, table sizes, read/write workload, etc. would be helpful as well.

Thanks!

Josh

On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <gaol@marinsoftware.com> wrote:

Hi Phoenix users,


We are seeing occasionally (maybe 30 ~ 50% of time) this StaleRegionBoundaryCacheException when running Spark 1.4.1 and Phoenix 4.5.1-HBase 1.0.

Not sure how to troubleshoot such issue. Any hints and insights are greatly appreciated!

Thanks,

Li

PS: The following are the exception stack trace:

16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 110.0 failed 4 times, most recent failure: Lost task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): java.lang.RuntimeException: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date.

at com.google.common.base.Throwables.propagate(Throwables.java:156)

at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)

at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)

at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)

at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)

at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)

at org.apache.spark.scheduler.Task.run(Task.scala:70)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of region boundaries are out of date.

at org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304)

at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)

at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)

at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)

at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)

at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70)

at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)

at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)

at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111)

... 18 more