phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert James <srobertja...@gmail.com>
Subject NoClassDefFoundError org/apache/hadoop/hbase/HBaseConfiguration
Date Tue, 05 Jul 2016 06:00:00 GMT
I'm trying to use Phoenix on Spark, and can't get around this error:

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
        at org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)

DETAILS:
1. I'm running HDP 2.4.0.0-169
2. Using phoenix-sqlline, I can access Phoenix perfectly
3. Using hbase shell, I can access HBase perfectly
4. I added the following lines to /etc/spark/conf/spark-defaults.conf

spark.driver.extraClassPath
/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar
spark.executor.extraClassPath
/usr/hdp/current/phoenix-client/lib/phoenix-spark-4.4.0.2.4.0.0-169.jar

5. Steps to reproduce the error:
# spark-shell
...
scala> import org.apache.phoenix.spark._
import org.apache.phoenix.spark._

scala> sqlContext.load("org.apache.phoenix.spark", Map("table" ->
"EMAIL_ENRON", "zkUrl" -> "localhost:2181"))
warning: there were 1 deprecation warning(s); re-run with -deprecation
for details
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
        at org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)

// Or, this gets the same error
scala> val rdd = sc.phoenixTableAsRDD("EMAIL_ENRON", Seq("MAIL_FROM",
"MAIL_TO"), zkUrl=Some("localhost"))
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
        at org.apache.phoenix.spark.PhoenixRDD.getPhoenixConfiguration(PhoenixRDD.scala:82)
        at org.apache.phoenix.spark.PhoenixRDD.phoenixConf$lzycompute(PhoenixRDD.scala:38)

6. I've tried every permutation I can think of, and also spent hours
Googling.  Some times I can get different errors, but always errors.
Interestingly, if I manage to load the HBaseConfiguration class
manually (by specifying classpaths and then import), I get a
"phoenixTableAsRDD is not a member of SparkContext" error.

How can I use Phoenix from within Spark?  I'm really eager to do so,
but haven't been able to.

Also: Can someone give me some background on the underlying issues
here? Trial-and-error-plus-google is not exactly high quality
engineering; I'd like to understand the problem better.

Mime
View raw message