phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Tarnas <...@biotiquesystems.com>
Subject Re: NoClassDefFoundError org/apache/hadoop/hbase/HBaseConfiguration
Date Tue, 05 Jul 2016 18:05:33 GMT
For HDP 2.4.2 this is what we ended up with to get it to work:

/usr/hdp/2.4.2.0-258/phoenix/lib/phoenix-core-4.4.0.2.4.2.0-258.jar
/usr/hdp/2.4.2.0-258/phoenix/lib/phoenix-spark-4.4.0.2.4.2.0-258.jar
/usr/hdp/2.4.2.0-258/phoenix/lib/hbase-client.jar
/usr/hdp/2.4.2.0-258/phoenix/lib/hbase-common.jar
/usr/hdp/2.4.2.0-258/phoenix/lib/hbase-protocol.jar
/usr/hdp/2.4.2.0-258/hbase/lib/hbase-server.jar



Chris Tarnas
Biotique Systems, Inc
cft@biotiquesystems.com


> On Jul 5, 2016, at 10:38 AM, Sergey Soldatov <sergeysoldatov@gmail.com> wrote:
> 
> Robert,
> you should use the phoenix-4*-spark.jar that is located in  root phoenix directory. 
> 
> Thanks,
> Sergey
> 
> On Tue, Jul 5, 2016 at 8:06 AM, Josh Elser <josh.elser@gmail.com <mailto:josh.elser@gmail.com>>
wrote:
> Looking into this on the HDP side. Please feel free to reach out via HDP channels instead
of Apache channels.
> 
> Thanks for letting us know as well.
> 
> Josh Mahonin wrote:
> Hi Robert,
> 
> I recommend following up with HDP on this issue.
> 
> The underlying problem is that the 'phoenix-spark-4.4.0.2.4.0.0-169 <tel:4.4.0.2.4.0.0-169>.jar'
> they've provided isn't actually a fat client JAR, it's missing many of
> the required dependencies. They might be able to provide the correct JAR
> for you, but you'd have to check with them. It may also be possible for
> you to manually include all of the necessary JARs on the Spark classpath
> to mimic the fat jar, but that's fairly ugly and time consuming.
> 
> FWIW, the HDP 2.5 Tech Preview seems to include the correct JAR, though
> I haven't personally tested it out yet.
> 
> Good luck,
> 
> Josh
> 
> On Tue, Jul 5, 2016 at 2:00 AM, Robert James <srobertjames@gmail.com <mailto:srobertjames@gmail.com>
> <mailto:srobertjames@gmail.com <mailto:srobertjames@gmail.com>>> wrote:
> 
>     I'm trying to use Phoenix on Spark, and can't get around this error:
> 
>     java.lang.NoClassDefFoundError:
>     org/apache/hadoop/hbase/HBaseConfiguration
>              at
>     org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
> 
>     DETAILS:
>     1. I'm running HDP 2.4.0.0-169
>     2. Using phoenix-sqlline, I can access Phoenix perfectly
>     3. Using hbase shell, I can access HBase perfectly
>     4. I added the following lines to /etc/spark/conf/spark-defaults.conf
> 
>     spark.driver.extraClassPath
>     /usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169 <tel:4.4.0.2.4.0.0-169>.jar
>     spark.executor.extraClassPath
>     /usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169 <tel:4.4.0.2.4.0.0-169>.jar
> 
>     5. Steps to reproduce the error:
>     # spark-shell
>     ...
>     scala> import org.apache.phoenix.spark._
>     import org.apache.phoenix.spark._
> 
>     scala> sqlContext.load("org.apache.phoenix.spark", Map("table" ->
>     "EMAIL_ENRON", "zkUrl" -> "localhost:2181"))
>     warning: there were 1 deprecation warning(s); re-run with -deprecation
>     for details
>     java.lang.NoClassDefFoundError:
>     org/apache/hadoop/hbase/HBaseConfiguration
>              at
>     org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
> 
>     // Or, this gets the same error
>     scala> val rdd = sc.phoenixTableAsRDD("EMAIL_ENRON", Seq("MAIL_FROM",
>     "MAIL_TO"), zkUrl=Some("localhost"))
>     java.lang.NoClassDefFoundError:
>     org/apache/hadoop/hbase/HBaseConfiguration
>              at
>     org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
>              at
>     org.apache.phoenix.spark.PhoenixRDD.phoenixConf$lzycompute(PhoenixRDD.scala:38)
> 
>     6. I've tried every permutation I can think of, and also spent hours
>     Googling.  Some times I can get different errors, but always errors.
>     Interestingly, if I manage to load the HBaseConfiguration class
>     manually (by specifying classpaths and then import), I get a
>     "phoenixTableAsRDD is not a member of SparkContext" error.
> 
>     How can I use Phoenix from within Spark?  I'm really eager to do so,
>     but haven't been able to.
> 
>     Also: Can someone give me some background on the underlying issues
>     here? Trial-and-error-plus-google is not exactly high quality
>     engineering; I'd like to understand the problem better.
> 
> 
> 


Mime
View raw message