phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Robinson <Steve.Robin...@aquilainsight.com>
Subject AWS EMR Spark Integration
Date Wed, 03 May 2017 15:08:08 GMT
Hi,


Has anyone successfully integrated Phoenix with Spark on Amazon EMR?


I'm using EMR release version emr-5.5.0, which is phoenix 4.9.0 and spark 2.1.0. This is causing
me issues when I try to run the below in a zeppelin notebook (from here<https://phoenix.apache.org/phoenix_spark.html>):


%spark

import org.apache.phoenix.spark._
import org.apache.spark._

val dataSet = List((1L, "1", 1), (2L, "2", 2), (3L, "3", 3))

sc
  .parallelize(dataSet)
  .saveToPhoenix(
    "OUTPUT_TEST_TABLE",
    Seq("ID","COL1","COL2"),
    zkUrl = Some("phoenix-server:2181")
  )



For reference, the only extra configuration I have performed is:


* Create the above OUTPUT_TEST_TABLE using the phoenix/bin/sqlline.py localhost on the master
node, by running: CREATE TABLE OUTPUT_TEST_TABLE (id BIGINT NOT NULL PRIMARY KEY, col1 VARCHAR,
col2 INTEGER);

* Added the following to spark-defaults.conf:

** spark.jars file:/usr/lib/phoenix/phoenix-4.9.0-HBase-1.2-client.jar,file:/usr/lib/phoenix/phoenix-core-4.9.0-HBase-1.2.jar


When I try to run the above code I get an error saying: java.lang.NoClassDefFoundError: org/apache/spark/Logging.
So it seems that phoenix 4.9.0 is compiled against an earlier version of spark (less than
1.5.2)?


Is there any way to use spark 2.1 with phoenix 4.9.0? Not really sure why AWS hasn't updated
to Phoenix 4.10.0 yet?


Thanks,


Steve Robinson

Steve.Robinson@aquilainsight.com
0131 290 2300

[https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=coZwd1lxzVikDFySR9HQQUgSdTBDtR24zCE7VI3ucKE%3d&docid=0c609694b07114f03a01631f3c5cc4606&rev=1][https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=XWaweiSSd7YO1IFgfwqm3AAn7KKCsmBf%2f73IlT3d0zE%3d&docid=0cea80d160d954b9584aef7090a5c4ef5&rev=1][https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=hTrHq%2fmrgDTxOp4jWXzYVM04wsasy2aNJfmG6EJJ%2f9g%3d&docid=00a6dd20560b1438fadebfb8a1255be41&rev=1]
www.aquilainsight.com<http://www.aquilainsight.com>
[https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=N79xtBiBY4r5ry1TCu0P%2bce%2f%2b3HFTwwamnQ47PieOoo%3d&docid=03f7d1040c43f4fa0bcdf7f17fa89dfcc&rev=1]linkedin.com/aquilainsight<https://www.linkedin.com/company/aquila-insight>
[https://aquilainsight.sharepoint.com/Phoenix/_layouts/15/guestaccess.aspx?guestaccesstoken=fdX1gHdkBdEZ%2bOap1Nr7kTrjMoFxgTZI4RfHFw0R7mw%3d&docid=0869faaa87f6c402fa845a320c225e213&rev=1]twitter.com/aquilainsight<http://twitter.com/aquilainsight>

Mime
View raw message