phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Mahonin <jmaho...@gmail.com>
Subject Re: NoClassDefFoundError org/apache/hadoop/hbase/HBaseConfiguration
Date Tue, 05 Jul 2016 13:32:35 GMT
Hi Robert,

I recommend following up with HDP on this issue.

The underlying problem is that the 'phoenix-spark-4.4.0.2.4.0.0-169.jar'
they've provided isn't actually a fat client JAR, it's missing many of the
required dependencies. They might be able to provide the correct JAR for
you, but you'd have to check with them. It may also be possible for you to
manually include all of the necessary JARs on the Spark classpath to mimic
the fat jar, but that's fairly ugly and time consuming.

FWIW, the HDP 2.5 Tech Preview seems to include the correct JAR, though I
haven't personally tested it out yet.

Good luck,

Josh

On Tue, Jul 5, 2016 at 2:00 AM, Robert James <srobertjames@gmail.com> wrote:

> I'm trying to use Phoenix on Spark, and can't get around this error:
>
> java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
>         at
> org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
>
> DETAILS:
> 1. I'm running HDP 2.4.0.0-169
> 2. Using phoenix-sqlline, I can access Phoenix perfectly
> 3. Using hbase shell, I can access HBase perfectly
> 4. I added the following lines to /etc/spark/conf/spark-defaults.conf
>
> spark.driver.extraClassPath
> /usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar
> spark.executor.extraClassPath
> /usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar
>
> 5. Steps to reproduce the error:
> # spark-shell
> ...
> scala> import org.apache.phoenix.spark._
> import org.apache.phoenix.spark._
>
> scala> sqlContext.load("org.apache.phoenix.spark", Map("table" ->
> "EMAIL_ENRON", "zkUrl" -> "localhost:2181"))
> warning: there were 1 deprecation warning(s); re-run with -deprecation
> for details
> java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
>         at
> org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
>
> // Or, this gets the same error
> scala> val rdd = sc.phoenixTableAsRDD("EMAIL_ENRON", Seq("MAIL_FROM",
> "MAIL_TO"), zkUrl=Some("localhost"))
> java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
>         at
> org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
>         at
> org.apache.phoenix.spark.PhoenixRDD.phoenixConf$lzycompute(PhoenixRDD.scala:38)
>
> 6. I've tried every permutation I can think of, and also spent hours
> Googling.  Some times I can get different errors, but always errors.
> Interestingly, if I manage to load the HBaseConfiguration class
> manually (by specifying classpaths and then import), I get a
> "phoenixTableAsRDD is not a member of SparkContext" error.
>
> How can I use Phoenix from within Spark?  I'm really eager to do so,
> but haven't been able to.
>
> Also: Can someone give me some background on the underlying issues
> here? Trial-and-error-plus-google is not exactly high quality
> engineering; I'd like to understand the problem better.
>

Mime
View raw message