From user-return-2417-apmail-phoenix-user-archive=phoenix.apache.org@phoenix.apache.org Thu Mar 5 23:49:31 2015 Return-Path: X-Original-To: apmail-phoenix-user-archive@minotaur.apache.org Delivered-To: apmail-phoenix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3BBF0174A7 for ; Thu, 5 Mar 2015 23:49:31 +0000 (UTC) Received: (qmail 44109 invoked by uid 500); 5 Mar 2015 23:49:24 -0000 Delivered-To: apmail-phoenix-user-archive@phoenix.apache.org Received: (qmail 44058 invoked by uid 500); 5 Mar 2015 23:49:24 -0000 Mailing-List: contact user-help@phoenix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@phoenix.apache.org Delivered-To: mailing list user@phoenix.apache.org Received: (qmail 44039 invoked by uid 99); 5 Mar 2015 23:49:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Mar 2015 23:49:24 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ndimiduk@gmail.com designates 74.125.82.174 as permitted sender) Received: from [74.125.82.174] (HELO mail-we0-f174.google.com) (74.125.82.174) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Mar 2015 23:48:59 +0000 Received: by wesu56 with SMTP id u56so56206438wes.10; Thu, 05 Mar 2015 15:47:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=ncRNcHZc2/PymvJlDmuC5m6/o9/Ct8vtoN99yfKK9IU=; b=YcaD7oD/AI8uLkiUuCOdUIGeS07WoRlxp0JGS81ymcLNjOOaU9+Fhm2jSUpkJ+8JcC oGDOyF8vUqHfdFdnUTbSJDTb905bcRzA7vkr8QtxtEMzgXKolzcBzQVauBZX7MEwwqXF pIBm3WNQZvn8zPJDe3W9pHLE20zlGGA4qDsU5oJi6xVz2O+Wrvx3goqCrEjdJhJuESry fu+TSugjIdb4zFMwlGzwXpnpNi+0v+IPGmzge6ZRNatCeWhisyCyq014XuN2MzZMLZy5 km4fF5HJf7RgDag+159J2EaghFsAiko0XvLG3Mu5Zb5/Re99U9emj8b0JbTiQxwcNtpm vQyw== X-Received: by 10.180.24.65 with SMTP id s1mr27354077wif.30.1425599248066; Thu, 05 Mar 2015 15:47:28 -0800 (PST) MIME-Version: 1.0 Received: by 10.28.186.135 with HTTP; Thu, 5 Mar 2015 15:47:07 -0800 (PST) In-Reply-To: References: From: Nick Dimiduk Date: Thu, 5 Mar 2015 15:47:07 -0800 Message-ID: Subject: Re: HBase Cluster Down: No jar path specified for org.apache.hadoop.hbase.regionserver.LocalIndexSplitter To: "user@phoenix.apache.org" Cc: dev Content-Type: multipart/alternative; boundary=f46d043bdf2ac5ee4505109330e8 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043bdf2ac5ee4505109330e8 Content-Type: text/plain; charset=UTF-8 You need to update the phoenix jar on the servers to match the client version. Both client and server should be the same versions for now, at least until our backward compatibility story is more reliable. Basically, the new client wrote new metadata to hbase schema and the old server jars don't have what's needed at runtime. On Thu, Mar 5, 2015 at 3:32 PM, anil gupta wrote: > Hi All, > > I am using HDP2.1.5, Phoenix4-0.0 was installed on RS. I was running > Phoenix4.1 client because i could not find tar file for > "Phoenix4-0.0-incubating". > I tried to create a view on existing table and then my entire cluster went > down(all the RS went down. MAster is still up). > > > This is the exception i am seeing: > > 2015-03-05 14:30:53,296 FATAL [RS_OPEN_REGION-hdpslave8:60020-2] regionserver.HRegionServer: ABORTING region server bigdatabox.com,60020,1423589420136: The coprocessor org.apache.hadoop.hbase.regionserver.LocalIndexSplitter threw an unexpected exception > java.io.IOException: No jar path specified for org.apache.hadoop.hbase.regionserver.LocalIndexSplitter > at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.java:177) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:207) > at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.(RegionCoprocessorHost.java:163) > at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:555) > at org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:462) > at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:526) > at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:4119) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4430) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4403) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4359) > at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4310) > at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:465) > at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:139) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > We tried to restart the cluster. It died again. It seems, its stucks at this point looking for > > LocalIndexSplitter class. How can i resolve this error? We cant do anything in the cluster until we fix it. > > I was thinking of disabling those tables but none of the RS is coming up. Can anyone suggest me how can i bail out of this BAD situation. > > > -- > Thanks & Regards, > Anil Gupta > --f46d043bdf2ac5ee4505109330e8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
You need to update the phoenix jar on the servers to match= the client version. Both client and server should be the same versions for= now, at least until our backward compatibility story is more reliable.
Basically, the new client wrote new metadata to hbase schem= a and the old server jars don't have what's needed at runtime.

On Thu, Ma= r 5, 2015 at 3:32 PM, anil gupta <anilgupta84@gmail.com>= wrote:
H= i All,

I am using HDP2.1.5, Phoenix4-0.0 was installed on RS. = I was running Phoenix4.1 client because i could not find tar file for "= ;Phoenix4-0.0-incubating".
I tried to create a view on exist= ing table and then my entire cluster went down(all the RS went down. MAster= is still up).


This is the exception i am seeing:
=

2015-03-05 14:30:53,296 FATAL [RS_OPEN_REGION-hdpslave8:60020-2] regio=
nserver.HRegionServer: ABORTING region server bigdatabox.com,60020,1423589420136: The coproces=
sor org.apache.hadoop.hbase.regionserver.LocalIndexSplitter threw an unexpe=
cted exception
java.io.IOException: No jar path specified for org=
.apache.hadoop.hbase.regionserver.LocalIndexSplitter
        at o=
rg.apache.hadoop.hbase.coprocessor.CoprocessorHost.load(CoprocessorHost.jav=
a:177)
        at org.apache.hadoop.hbase.regionserver.RegionCopr=
ocessorHost.loadTableCoprocessors(RegionCoprocessorHost.java:207)
        at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<=
init>(RegionCoprocessorHost.java:163)
at org.apache.ha= doop.hbase.regionserver.HRegion.<init>(HRegion.java:555)
= at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.= java:462)
at sun.reflect.GeneratedConstructorAccessor33.n= ewInstance(Unknown Source)
at sun.reflect.DelegatingConst= ructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.j= ava:526)
at org.apache.hadoop.hbase.regionserver.HRegion.= newHRegion(HRegion.java:4119)
at org.apache.hadoop.hbase.= regionserver.HRegion.openHRegion(HRegion.java:4430)
at or= g.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4403)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegio= n(HRegion.java:4359)
at org.apache.hadoop.hbase.regionser= ver.HRegion.openHRegion(HRegion.java:4310)
at org.apache.= hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHa= ndler.java:465)
at org.apache.hadoop.hbase.regionserver.h= andler.OpenRegionHandler.process(OpenRegionHandler.java:139)
= at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:= 128)
at java.util.concurrent.ThreadPoolExecutor.runWorker= (ThreadPoolExecutor.java:1145)
at java.util.concurrent.Th= readPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
= at java.lang.Thread.run(Thread.java:744)


We tried to = restart the cluster. It died again. It seems, its stucks at this point look= ing for 
LocalIndexSplitter class. How can i resolve this error? We=
 cant do anything in the cluster until we fix it.

I was t=
hinking of disabling those tables but none of the RS is coming up. Can anyo=
ne suggest me how can i bail out of this BAD situation.

--
Tha= nks & Regards,
Anil Gupta

--f46d043bdf2ac5ee4505109330e8--