phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alicia Shu <a...@hortonworks.com>
Subject Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1
Date Thu, 14 Jan 2016 22:55:52 GMT
With the fix of PHOENIX-2447<https://issues.apache.org/jira/browse/PHOENIX-2447> in 4.7.0,
now Phoenix will try multiple times for StaleRegionBoundaryCacheException until query timeout.

Alicia

From: Josh Mahonin <jmahonin@gmail.com<mailto:jmahonin@gmail.com>>
Reply-To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" <user@phoenix.apache.org<mailto:user@phoenix.apache.org>>
Date: Thursday, January 14, 2016 at 12:33 PM
To: "user@phoenix.apache.org<mailto:user@phoenix.apache.org>" <user@phoenix.apache.org<mailto:user@phoenix.apache.org>>
Subject: Re: StaleRegionBoundaryCacheException with Phoenix 4.5.1-HBase 1.0 and Spark 1.4.1

Hi Li,

I've not seen this error myself, though some searching returns a possible root cause:
http://mail-archives.apache.org/mod_mbox/incubator-phoenix-user/201507.mbox/%3CCAAF1JdjNW98dAnxf3kx=nDkSWyorpt1redB9DQwjBHVVLfnTqw@mail.gmail.com%3E

Could you file a JIRA ticket for this please? It's possible the MapReduce (and by extension,
Spark) integration isn't handling this gracefully.

Also, if possible, any information you have about your environment, table sizes, read/write
workload, etc. would be helpful as well.

Thanks!

Josh

On Thu, Jan 14, 2016 at 3:14 PM, Li Gao <gaol@marinsoftware.com<mailto:gaol@marinsoftware.com>>
wrote:

Hi Phoenix users,


We are seeing occasionally (maybe 30 ~ 50% of time) this StaleRegionBoundaryCacheException
when running Spark 1.4.1 and Phoenix 4.5.1-HBase 1.0.

Not sure how to troubleshoot such issue. Any hints and insights are greatly appreciated!

Thanks,

Li

PS: The following are the exception stack trace:

16/01/14 19:40:16 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.SparkException:
Job aborted due to stage failure: Task 5 in stage 110.0 failed 4 times, most recent failure:
Lost task 5.3 in stage 110.0 (TID 35526, datanode-123.somewhere): java.lang.RuntimeException:
org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08): Cache of
region boundaries are out of date.

at com.google.common.base.Throwables.propagate(Throwables.java:156)

at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:126)

at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:133)

at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)

at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)

at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:52)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)

at org.apache.spark.scheduler.Task.run(Task.scala:70)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.phoenix.schema.StaleRegionBoundaryCacheException: ERROR 1108 (XCL08):
Cache of region boundaries are out of date.

at org.apache.phoenix.exception.SQLExceptionCode$13.newException(SQLExceptionCode.java:304)

at org.apache.phoenix.exception.SQLExceptionInfo.buildException(SQLExceptionInfo.java:145)

at org.apache.phoenix.util.ServerUtil.parseRemoteException(ServerUtil.java:131)

at org.apache.phoenix.util.ServerUtil.parseServerExceptionOrNull(ServerUtil.java:115)

at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:104)

at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:70)

at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)

at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)

at org.apache.phoenix.mapreduce.PhoenixRecordReader.initialize(PhoenixRecordReader.java:111)

... 18 more


Mime
View raw message