phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mengfei <>
Subject Re: Re: error when get data from Phoenix 4.5.2 on CDH 5.5.x by spark 1.5
Date Tue, 05 Jan 2016 10:38:33 GMT

hi josh:                            thank you for your advice,and it did work . i build the
client-spark jar refreed the patch with thr CDH code and it succeed.                Then i
run some code with the "local" mode ,and the result is correct. But when it comes to the "yarn-client"
mode ,some error happend:
                 java.lang.RuntimeException: java.sql.SQLException: No suitable driver found
                I did try every way i know or i find in the commities: but  they don`t help.
 So i want to get help from you ,Thank you for your patience.                            
   My code is :import org.apache.spark.sql.SQLContextimport org.apache.phoenix.spark._import
org.apache.spark.SparkContextimport org.apache.spark.sql.SQLContext import org.apache.phoenix.jdbc.PhoenixDriverimport
java.sql.DriverManager DriverManager.registerDriver(new PhoenixDriver)val pred = s"MMSI =
'002190048'"val rdd = sc.phoenixTableAsRDD(    "AIS_WS",     Seq("MMSI","LON","LAT","RID"),
   predicate = Some(pred),    zkUrl = Some(",,"))println(rdd.count())
my  scripts are: spark-submit \--master yarn-cluster  \--driver-class-path  "/data/public/mengfei/lib/phoenix-1.2.0-client-spark.jar"
\--conf "spark.executor.extraClassPath=/data/public/mengfei/lib/phoenix-1.2.0-client-spark.jar"
\--conf "spark.driver.extraClassPath=/data/public/mengfei/lib/phoenix-1.2.0-client-spark.jar"
\--jars /data/public/mengfei/lib/phoenix-1.2.0-client-spark.jar \
spark-shell \
--master yarn-client -v \
--driver-class-path  "/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client-spark.jar"
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client-spark.jar"
--conf "spark.driver.extraClassPath=/opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client-spark.jar"
--jars /opt/cloudera/parcels/CLABS_PHOENIX/lib/phoenix/phoenix-1.2.0-client-spark.jar  
ps: i did copied the jar to every node,and give the even the 777 rghts.
 From: Josh MahoninDate: 2015-12-30 00:56To: userSubject: Re: error when get data from Phoenix
4.5.2 on CDH 5.5.x by spark 1.5Hi,
This issue is fixed with the following patch, and using the resulting 'client-spark' JAR after

As an alternative, you may have some luck also including updated com.fasterxml.jackson jackson-databind
JARs in your app that are in sync with Spark's versions. Unfortunately the client JAR right
now is shipping fasterxml jars that conflict with the Spark runtime.
Another user has also had success by bundling their own Phoenix dependencies, if you want
to try that out instead:

On Tue, Dec 29, 2015 at 9:11 AM, <> wrote:
The error is java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.Module$SetupContext.setClassIntrospector(Lcom/fasterxml/jackson/databind/introspect/ClassIntrospector;)V
       at com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospectorModule$$anonfun$1.apply(ScalaClassIntrospector.scala:32)
       at com.fasterxml.jackson.module.scala.introspect.ScalaClassIntrospectorModule$$anonfun$1.apply(ScalaClassIntrospector.scala:32)
       at com.fasterxml.jackson.module.scala.JacksonModule$$anonfun$setupModule$1.apply(JacksonModule.scala:47)
     …..The scala code is val df = sqlContext.load(  "org.apache.phoenix.spark",  Map("table"
-> "AIS ", "zkUrl" -> "")) Maybe I got the resoon ,the Phoenix 4.5.2
on  CDH 5.5.x is build with spark 1.4 ,and cdh5.5`defalut spark version is 1.5.So  how could
I do?  To rebuild a phoenix 4.5.2 version with spark 1.5 Or change the cdh spark to 1.4. 
Apreantly these are difficult for me . Could someone help me ? Thank you vey much. 

View raw message