Hi Ricardo,
Are you talking about below Phoenix jar ?
 --conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-client.jar" 

That also been added.
Please refer to my first post .

Thanks,
Divya 

On 11 April 2016 at 18:33, Ricardo Crespo <ricardo.e.crespo@gmail.com> wrote:
Divya,


I check that the error that you are getting is on the executor, could you add this parameter to spark submit:


--conf spark.executor.extraClassPath=path_to_phoenix_jar


That way the executor should get phoenix in the classpath.


Best Regards,
Ricardo


2016-04-11 12:15 GMT+02:00 Divya Gehlot <divya.htconex@gmail.com>:
Hi Ricardo ,
If would have observed my previous post carefully 
I am already passing  the  below jars 
--jars /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-server.jar  

When I try to test the connection through JDBC able to connect succssfully 
/* Testing Phoenix Connection */
import java.sql.{Connection, DriverManager}
def testConnection(): Unit = {
Class.forName("org.apache.phoenix.jdbc.PhoenixDriver")
    println("start to connect phoenix!")
    var conn = DriverManager.getConnection("jdbc:phoenix:localhost:2181")
println("Connected to Phoenix")
}


StackTrace of above code 


import java.sql.{Connection, DriverManager}
testConnection: ()Unit
start to connect phoenix!
16/04/11 10:04:51 INFO RecoverableZooKeeper: Process identifier=hconnection-0x4260f924 connecting to ZooKeeper ensemble=localhost:2181
16/04/11 10:04:51 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-3485--1, built on 12/16/2015 02:35 GMT
16/04/11 10:04:51 INFO ZooKeeper: Client environment:host.name=ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal
16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.version=1.7.0_67
16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.home=/usr/jdk64/jdk1.7.0_67/jre
16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.class.path=/usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-core-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar:/usr/hdp/current/spark-thriftserver/conf/:/usr/hdp/2.3.4.0-3485/spark/lib/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-api-jdo-3.2.6.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-core-3.2.10.jar:/usr/hdp/2.3.4.0-3485/spark/lib/datanucleus-rdbms-3.2.9.jar:/usr/hdp/current/hadoop-client/conf/
16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
16/04/11 10:04:51 INFO ZooKeeper: Client environment:java.compiler=<NA>
16/04/11 10:04:51 INFO ZooKeeper: Client environment:os.name=Linux
16/04/11 10:04:51 INFO ZooKeeper: Client environment:os.arch=amd64
16/04/11 10:04:51 INFO ZooKeeper: Client environment:os.version=3.10.0-229.el7.x86_64
16/04/11 10:04:51 INFO ZooKeeper: Client environment:user.name=hdfs
16/04/11 10:04:51 INFO ZooKeeper: Client environment:user.home=/home/hdfs
16/04/11 10:04:51 INFO ZooKeeper: Client environment:user.dir=/home/hdfs
16/04/11 10:04:51 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x4260f9240x0, quorum=localhost:2181, baseZNode=/hbase-unsecure
16/04/11 10:04:51 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/04/11 10:04:51 INFO ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/04/11 10:04:51 INFO ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x353f97f0405053a, negotiated timeout = 40000
16/04/11 10:04:51 INFO Metrics: Initializing metrics system: phoenix
16/04/11 10:04:51 INFO MetricsConfig: loaded properties from hadoop-metrics2.properties
16/04/11 10:04:51 INFO MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
16/04/11 10:04:51 INFO MetricsSystemImpl: phoenix metrics system started
16/04/11 10:04:52 INFO RecoverableZooKeeper: Process identifier=hconnection-0x1679b42e connecting to ZooKeeper ensemble=localhost:2181
16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x1679b42e0x0, quorum=localhost:2181, baseZNode=/hbase-unsecure
16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x353f97f0405053b, negotiated timeout = 40000
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x353f97f0405053b
16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053b closed
16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
16/04/11 10:04:52 INFO RecoverableZooKeeper: Process identifier=hconnection-0x6c4546cf connecting to ZooKeeper ensemble=localhost:2181
16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6c4546cf0x0, quorum=localhost:2181, baseZNode=/hbase-unsecure
16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x353f97f0405053c, negotiated timeout = 40000
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x353f97f0405053c
16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053c closed
16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
16/04/11 10:04:52 INFO RecoverableZooKeeper: Process identifier=hconnection-0x5be1d976 connecting to ZooKeeper ensemble=localhost:2181
16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x5be1d9760x0, quorum=localhost:2181, baseZNode=/hbase-unsecure
16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x353f97f0405053d, negotiated timeout = 40000
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x353f97f0405053d
16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053d closed
16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
16/04/11 10:04:52 INFO RecoverableZooKeeper: Process identifier=hconnection-0x46f4dedf connecting to ZooKeeper ensemble=localhost:2181
16/04/11 10:04:52 INFO ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x46f4dedf0x0, quorum=localhost:2181, baseZNode=/hbase-unsecure
16/04/11 10:04:52 INFO ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
16/04/11 10:04:52 INFO ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session
16/04/11 10:04:52 INFO ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x353f97f0405053e, negotiated timeout = 40000
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
16/04/11 10:04:52 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x353f97f0405053e
16/04/11 10:04:52 INFO ZooKeeper: Session: 0x353f97f0405053e closed
16/04/11 10:04:52 INFO ClientCnxn: EventThread shut down
Connected to Phoenix
calling testConnection()


Unable to figure out the root cause 
Why its throwing error when I try to save to Phoenix
Would really appreciate the help.


Thanks,
Divya  

On 11 April 2016 at 17:33, Ricardo Crespo <ricardo.e.crespo@gmail.com> wrote:
Hi Divya,


I am asumming you are using YARN, you have to put phoenix jar on your spark classpath, you can do it with this parameter:

--jars path_to_phoenix_jar

you can check another options here:
http://spark.apache.org/docs/latest/running-on-yarn.html

Another option is to included phoenix on your jar with mvn assembly plugin:

http://maven.apache.org/plugins/maven-assembly-plugin/

Best regards,
Ricardo


2016-04-11 11:17 GMT+02:00 Divya Gehlot <divya.htconex@gmail.com>:
Hi,
I am getting below error when I try to save data to Phoenix 
Below are  Cluster configuration and steps which I followed :
Cluster Configuration :
Hortonworks distribution 2.3.4 version 
Spark 1.5.2
Pheonix 4.4 


Table created in Phoenix 
CREATE TABLE TEST
(
   RPT_DATE varchar(100) PRIMARY KEY
)
;

Spark Scala Script 
val dfLCR = readTable(sqlContext, "", "TEST")
val schemaL = dfLCR.schema
val lcrReportPath = "/TestDivya/Spark/Results/TestData/"
val dfReadReport= sqlContext.read.format("com.databricks.spark.csv").option("header", "true").schema(schemaL).load(lcrReportPath)
dfReadlcrReport.show()
val dfWidCol = dfReadReport.withColumn("RPT_DATE",lit("2015-01-01"))
val dfSelect = dfWidCol.select("RPT_DATE")
dfSelect.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(collection.immutable.Map(
                "zkUrl" -> "localhost",
                "table" -> "TEST")).save()

command line to run the Spark scala job 
spark-shell  --properties-file  /TestDivya/Spark/Phoenix.properties --jars /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-client.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-server.jar  --driver-class-path /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-server.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar --conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-client.jar" --packages com.databricks:spark-csv_2.10:1.4.0  --master yarn-client -i /TestDivya/Spark/WriteToPheonix.scala


Error :

16/04/11 09:03:43 INFO TaskSetManager: Lost task 2.3 in stage 3.0 (TID 409) on executor ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal: java.lang.RuntimeException (java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost:2181:/hbase-unsecure;) [duplicate 7]
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 408, ip-172-31-22-135.ap-southeast-1.compute.internal): java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost:2181:/hbase-unsecure;

Could somebody help me resolving the error.
What am I missing ?

Thanks,
Divya