You might also consider moving back down to 7u79 

On Feb 15, 2016, at 10:35 AM, Jonathan Leech <jonathaz@gmail.com> wrote:

Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is calling the wrong toObject(), and then dumping. The correct toObject() method is a couple parent classes away with some generics and Sun / Oracle must have gotten something wrong with that special case under load. Thinking I may be able to just patch in a call to the correct method to stop the bleeding. Anyone else have other ideas, thoughts, or also want the patch?

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

J 7432 C2 org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object; (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492]

J 14641 C2 org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8]

J 11096 C2 org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c]

J 9742 C2 org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object; (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58]

J 8783 C2 org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState; (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48]

J 10636 C2 org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator; (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4]

J 8646 C2 org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5 bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc]

J 9466 C2 org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12 bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0]

J 7047 C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c]

J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes) @ 0x00007f833420ad2c [0x00007f833420ac80+0xac]

j  java.lang.Thread.run()V+26

v  ~StubRoutines::call_stub

V  [libjvm.so+0x6020a5]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365

V  [libjvm.so+0x600b08]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28

V  [libjvm.so+0x600dd7]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197

V  [libjvm.so+0x600ef7]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47

V  [libjvm.so+0x67f8d5]  thread_entry(JavaThread*, Thread*)+0xe5

V  [libjvm.so+0x95bfff]  JavaThread::thread_main_inner()+0xdf

V  [libjvm.so+0x95c105]  JavaThread::run()+0xf5

V  [libjvm.so+0x8238c8]  java_start(Thread*)+0x108