phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ciureanu Constantin <ciureanu.constan...@gmail.com>
Subject Re: Accessing phoenix tables in Spark 2
Date Fri, 07 Oct 2016 13:30:32 GMT
In Spark 1.4 it worked via JDBC - sure it would work in 1.6 / 2.0 without
issues.

Here's a sample code I used (it was getting data in parallel 24 partitions)


import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

import org.apache.spark.rdd.JdbcRDD
import java.sql.{Connection, DriverManager, ResultSet}

sc.addJar("/usr/lib/hbase/hbase-protocol.jar")
sc.addJar("phoenix-x.y.z-bin/phoenix-core-x.y.z.jar")
sc.addJar("phoenix-x.y.z-bin/phoenix-x.y.z-client.jar")

def createConnection() = {
Class.forName("org.apache.phoenix.jdbc.PhoenixDriver").newInstance();
DriverManager.getConnection("jdbc:phoenix:hd101.lps.stage,hd102.lps.stage,hd103.lps.stage");
// the Zookeeper quorum
}

def extractValues(r: ResultSet) = {
(r.getLong(1),    // datum
r.getInt(2),  // pg
r.getString(3),  // HID
....
 )
}

val data = new JdbcRDD(sc, createConnection,
"SELECT DATUM, PG, HID,  ..... WHERE DATUM >= ? * 1000  AND DATUM <= ? *
1000 and PG = <a value>",
lowerBound = 1364774400, upperBound = 1384774400, numPartitions = 24,
mapRow = extractValues)

data.count()

println(data.collect().toList)


2016-10-07 15:20 GMT+02:00 Ted Yu <yuzhihong@gmail.com>:

> JIRA on hbase side:
> HBASE-16179
>
> FYI
>
> On Fri, Oct 7, 2016 at 6:07 AM, Josh Mahonin <jmahonin@gmail.com> wrote:
>
>> Hi Mich,
>>
>> There's an open ticket about this issue here:
>> https://issues.apache.org/jira/browse/PHOENIX-3333
>>
>> Long story short, Spark changed their API (again), breaking the existing
>> integration. I'm not sure the level of effort to get it working with Spark
>> 2.0, but based on examples from other projects, it looks like there's a
>> fair bit of Maven module work to support both Spark 1.x and Spark 2.x
>> concurrently in the same project. Patches are very welcome!
>>
>> Best,
>>
>> Josh
>>
>>
>>
>> On Fri, Oct 7, 2016 at 8:33 AM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Has anyone managed to read phoenix table in Spark 2 by any chance please?
>>>
>>> Thanks
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>
>>
>

Mime
View raw message