Hi James

I tried the upgrade path you mentioned and it worked as far as I can tell. Insert and query existing tables works at least.

The only thing that worries me is an exception thrown at region server start up [1] and frequent periodic exceptions complaining about building the index [2] in runtime. I followed the upgrade procedure multiple times and always seem to end up in this state.

What could be the cause of these exceptions? HashJoiningRegionObserver does indeed not exist in any phoenix 3+ versions.

Cheers,
-Kristoffer

[1]

15/01/13 10:27:27 WARN regionserver.RegionCoprocessorHost: attribute 'coprocessor$4' has invalid coprocessor specification '|org.apache.phoenix.join.HashJoiningRegionObserver|1|'
15/01/13 10:27:27 WARN regionserver.RegionCoprocessorHost: java.io.IOException: No jar path specified for org.apache.phoenix.join.HashJoiningRegionObserver
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:185)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:190)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:154)
at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:473)
at sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4070)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4253)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:329)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:100)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:171)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


[2]

5/01/13 10:27:47 WARN coprocessor.MetaDataRegionObserver: ScheduledBuildIndexTask failed!
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column family 0 does not exist in region SYSTEM.TABLE,,1421139725748.a46320eb144712e231b1dd8ab3da30aa. in table {NAME => 'SYSTEM.TABLE', SPLIT_POLICY => 'org.apache.phoenix.schema.MetaDataSplitPolicy', coprocessor$7 => '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|2|', coprocessor$5 => '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|', coprocessor$6 => '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|1|', coprocessor$3 => '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|', coprocessor$4 => '|org.apache.phoenix.join.HashJoiningRegionObserver|1|', coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|1|', UpgradeTo21 => 'true', coprocessor$2 => '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|', UpgradeTo20 => 'true', UpgradeTo22 => 'true', FAMILIES => [{NAME => '_0', ENCODE_ON_DISK => 'true', BLOOMFILTER => 'NONE', VERSIONS => '1000', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'true', DATA_BLOCK_ENCODING => 'FAST_DIFF', TTL => '2147483647', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}]}
at org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:5341)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1744)
at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1722)
at org.apache.phoenix.coprocessor.MetaDataRegionObserver$BuildIndexScheduleTask.run(MetaDataRegionObserver.java:174)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


On Mon, Jan 12, 2015 at 7:05 PM, James Taylor <jamestaylor@apache.org> wrote:
Hi Kristoffer,
You'll need to upgrade first from 2.2.3 to 3.0.0-incubating, and then
to each minor version (3.1 and then 3.2.2) to trigger the upgrade for
each release. You can access previous releases from the "Download
Previous Releases" link here: http://phoenix.apache.org/download.html.
We'll improve this in future releases such that you can go directly to
any minor release within the same major release in a single step.

When you upgrade, follow these steps:
- Remove the old client and server jar
- Replace both the client jar and the server jar with the new one
- Bounce your cluster
- Establish a connection from the client to the server (i.e. bring up
sqlline, for example). This is what actually triggers the upgrade.

FWIW, since you're going through the trouble of upgrading, you may
want to consider moving to our 4.x releases and upgrading your cluster
to HBase 0.98. I believe the 0.94 HBase releases are close to
end-of-life, and the upcoming 3.3 release of Phoenix will be the last
release in the 3.x series.

Thanks,
James

On Mon, Jan 12, 2015 at 5:38 AM, Kristoffer Sjögren <stoffe@gmail.com> wrote:
> Hi
>
> I'm trying to upgrade phoenix 2.2.3-incubating to phoenix 3.2.2 on my local
> computer first in order to gain confidence that it will work on the
> production cluster. We use HBase 0.94.6 CDH 4.4.0.
>
> 1) My first question is what release to pick? There is no phoenix 3.2.2 jar
> in maven central (only 3.2.1 jar) and there is no 3.2.1 tar.gz distribution
> is available from phoenix.apache.org (only 3.2.2).
>
> Anyway, I tried replacing phoenix-2.2.3-incubating.jar with
> phoenix-core-3.2.2.jar
> (from phoenix-3.2.2-bin.tar.gz) in $HBASE_HOME/lib and restarted HBase.
>
> This triggered warnings in HBase log [2], which is understandable since
> phoenix-core-3.2.2.jar does not include
> org.apache.phoenix.join.HashJoiningRegionObserver.
>
>
> 2) Next step I updated the client from phoenix-2.2.3-incubating.jar to
> phoenix-core-3.2.1.jar and added the following to hbase-site.xml.
>
>   <configuration>
>     <property>
>       <name>phoenix.client.autoUpgradeWhiteList</name>
>       <value>*</value>
>     </property>
>   </configuration>
>
> I think this triggered the upgrade process as soon as the client contacted
> HBase. But all tables are inaccessible after this process [2]. Now there are
> also periodic warnings occurring periodically in HBase log [3].
>
> I also tried to install 3.2.2 manually into maven and run same client and
> server version, but this did not change the behavior.
>
> I'm not sure what have gone wrong?
>
> Cheers,
> -Kristoffer
>
>
> [1]
>
> 15/01/12 14:25:12 WARN regionserver.RegionCoprocessorHost: attribute
> 'coprocessor$6' has invalid coprocessor specification
> '|org.apache.hbase.index.Indexer|1073741823|org.apache.hbase.index.codec.class=org.apache.phoenix.index.PhoenixIndexCodec,index.builder=org.apache.phoenix.index.PhoenixIndexBuilder'
>
> 15/01/12 14:25:12 WARN regionserver.RegionCoprocessorHost:
> java.io.IOException: No jar path specified for
> org.apache.hbase.index.Indexer
>
> 15/01/12 14:25:12 WARN regionserver.RegionCoprocessorHost: attribute
> 'coprocessor$4' has invalid coprocessor specification
> '|org.apache.phoenix.join.HashJoiningRegionObserver|1|'
>
> 15/01/12 14:25:12 WARN regionserver.RegionCoprocessorHost:
> java.io.IOException: No jar path specified for
> org.apache.phoenix.join.HashJoiningRegionObserver
>
> [2]
>
> Caused by: org.apache.phoenix.schema.TableNotFoundException: ERROR 1012
> (42M03): Table undefined. tableName=TRACKING_COUNTER
>
>
> [3]
>
>
> 15/01/12 14:25:24 WARN coprocessor.MetaDataRegionObserver:
> ScheduledBuildIndexTask failed!
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException: Column
> family 0 does not exist in region
> SYSTEM.TABLE,,1421069004810.6b41a24a11a4f106b85d6ae76334baf6. in table {NAME
> => 'SYSTEM.TABLE', SPLIT_POLICY =>
> 'org.apache.phoenix.schema.MetaDataSplitPolicy', UpgradeTo21 => 'true',
> UpgradeTo20 => 'true', coprocessor$7 =>
> '|org.apache.phoenix.coprocessor.MetaDataRegionObserver|2|', UpgradeTo22 =>
> 'true', coprocessor$6 =>
> '|org.apache.phoenix.coprocessor.MetaDataEndpointImpl|1|', coprocessor$5 =>
> '|org.apache.phoenix.coprocessor.ServerCachingEndpointImpl|1|',
> coprocessor$4 => '|org.apache.phoenix.join.HashJoiningRegionObserver|1|',
> coprocessor$3 =>
> '|org.apache.phoenix.coprocessor.GroupedAggregateRegionObserver|1|',
> coprocessor$2 =>
> '|org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver|1|',
> coprocessor$1 => '|org.apache.phoenix.coprocessor.ScanRegionObserver|1|',
> FAMILIES => [{NAME => '_0', DATA_BLOCK_ENCODING => 'FAST_DIFF', BLOOMFILTER
> => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '1000', COMPRESSION =>
> 'NONE', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS =>
> 'true', BLOCKSIZE => '65536', ENCODE_ON_DISK => 'true', IN_MEMORY =>
> 'false', BLOCKCACHE => 'true'}]}
> at
> org.apache.hadoop.hbase.regionserver.HRegion.checkFamily(HRegion.java:5341)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1744)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1722)
> at
> org.apache.phoenix.coprocessor.MetaDataRegionObserver$BuildIndexScheduleTask.run(MetaDataRegionObserver.java:174)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)