Thanks a lot, Don! I have assigned the issue to myself and will look into it shortly.

On Thu, Nov 5, 2015 at 7:10 PM, Don Brinn <> wrote:

Hi Maryann,


I have created this defect report in JIRA:




Don Brinn


From: Maryann Xue []
Sent: Thursday, November 5, 2015 5:13 PM
Subject: Re: Error "could not find hash cache for joinId" when doing Inner Join with any table or view with Multi_Tenant=true


Hi Don,

Thank you very much for finding the issue. Would mind filing a Phoenix JIRA?



On Thu, Nov 5, 2015 at 3:08 PM Don Brinn <> wrote:



I am seeing the following error when doing an INNER JOIN of a view with MULTI_TENANT=true with any other table or view:

java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: org.apache.hadoop.hbase.DoNotRetryIOException: Could not find hash cache for joinId: Ys0��%. The cache might have expired and have been removed.

        at org.apache.phoenix.coprocessor.HashJoinRegionScanner.<init>(

        at org.apache.phoenix.coprocessor.ScanRegionObserver.doPostScannerOpen(

        at org.apache.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen(

        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(

        at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(

        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(



        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(

        at org.apache.hadoop.hbase.ipc.RpcExecutor$



        at sqlline.IncrementalRows.hasNext(

        at sqlline.TableOutputFormat.print(

        at sqlline.SqlLine.print(

        at sqlline.Commands.execute(

        at sqlline.Commands.sql(

        at sqlline.SqlLine.dispatch(

        at sqlline.SqlLine.begin(

        at sqlline.SqlLine.start(

        at sqlline.SqlLine.main(


This is with Phoenix version 4.6.0 and HBase version


This seems very strongly related to the MULTI_TENANT=true property on a view or table.  I see the error whenever the view has MULTI_TENANT=true and I have a tenant-specific connection to Phoenix.  I do not see the problem if the MULTI_TENANT=true property is not set on the view or if I do not have a tenant-specific connection to Phoenix.


Here is an example SQL statement that has this error when the view INVENTORY has the MULTI_TENANT=true property and I have a tenant-specific connection, but that succeeds in other cases. (The view PRODUCT_IDS is not Multi-Tenant.)



Note:  “INNER JOIN” fails under these conditions, as does “LEFT OUTER JOIN”.  However, “RIGHT OUTER JOIN” and “FULL OUTER JOIN” do work.  Also, if I tell Phoenix to use a Sort Join for the Inner Join or Left Outer Join then it does work, e.g.  SELECT /*+ USE_SORT_MERGE_JOIN*/ * FROM INVENTORY INNER JOIN PRODUCT_IDS ON (PRODUCT_ID = INVENTORY.ID); works.


This seems to be the same problem that was discussed previously in this mailing list:


Is this a known issue?  Is there a fix or work-around, or something I should be doing differently to avoid the problem?





Don Brinn