phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Dimiduk <ndimi...@gmail.com>
Subject Re: phoenix-spark and pyspark
Date Sat, 23 Jan 2016 00:32:46 GMT
On Thu, Jan 21, 2016 at 7:36 AM, Josh Mahonin <jmahonin@gmail.com> wrote:

> Amazing, good work.
>

All I did was consume your code and configure my cluster. Thanks though :)

FWIW, I've got a support case in with Hortonworks to get the phoenix-spark
> integration working out of the box. Assuming it gets resolved, that'll
> hopefully help keep these classpath-hell issues to a minimum going forward.
>

That's fine for HWX customers, but it doesn't solve the general problem
community-side. Unless, of course, we assume the only Phoenix users are on
HDP.

Interesting point re: PHOENIX-2535. Spark does offer builds for specific
> Hadoop versions, and also no Hadoop at all (with the assumption you'll
> provide the necessary JARs). Phoenix is pretty tightly coupled with its own
> HBase (and by extension, Hadoop) versions though... do you think it be
> possible to work around (2) if you locally added the HDP Maven repo and
> adjusted versions accordingly? I've had some success with that in other
> projects, though as I recall when I tried it with Phoenix I ran into a snag
> trying to resolve some private transitive dependency of Hadoop.
>

I could specify Hadoop version and build Phoenix locally to remove the
issue. It would work for me because I happen to be packaging my own Phoenix
this week, but it doesn't help for the Apache releases. Phoenix _is_
tightly coupled to HBase versions, but I don't think HBase versions are
that tightly coupled to Hadoop versions. We build HBase 1.x releases
against a long list of Hadoop releases as part of our usual build. I think
what HBase consumes re: Hadoop API's is pretty limited.

Now that you have Spark working you can start hitting real bugs! If you
> haven't backported the full patchset, you might want to take a look at the
> phoenix-spark history [1], there's been a lot of churn there, especially
> with regards to the DataFrame API.
>

Yeah, speaking of which... :)

I find this integration is basically unusable when the underlying HBase
table partitions are in flux; i.e., when you have data being loaded
concurrent to query. Spark RDD's assume stable partitions, multiple queries
against a single DataFrame, or even a single query/DataFrame with lots of
underlying splits and not enough workers, will inevitably fail
with ConcurrentModificationException like [0].

I think in HBase/MR integration, we define split points for the job as
region boundaries initially, but then we're just using the regular API, so
repartitions are handled transiently by the client layer. I need to dig
into this Phoenix/Spark stuff to see if we can do anything similar here.

Thanks again Josh,
-n

[0]:

java.lang.RuntimeException: java.util.ConcurrentModificationException
at
org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:125)
at
org.apache.phoenix.mapreduce.PhoenixInputFormat.createRecordReader(PhoenixInputFormat.java:69)
at org.apache.spark.rdd.NewHadoopRDD$$anon$1.<init>(NewHadoopRDD.scala:131)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:104)
at org.apache.spark.rdd.NewHadoopRDD.compute(NewHadoopRDD.scala:66)
at org.apache.phoenix.spark.PhoenixRDD.compute(PhoenixRDD.scala:57)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:70)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at java.util.Hashtable$Enumerator.next(Hashtable.java:1367)
at org.apache.hadoop.conf.Configuration.iterator(Configuration.java:2154)
at
org.apache.phoenix.util.PropertiesUtil.extractProperties(PropertiesUtil.java:52)
at
org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(ConnectionUtil.java:60)
at
org.apache.phoenix.mapreduce.util.ConnectionUtil.getInputConnection(ConnectionUtil.java:45)
at
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getSelectColumnMetadataList(PhoenixConfigurationUtil.java:278)
at
org.apache.phoenix.mapreduce.util.PhoenixConfigurationUtil.getSelectStatement(PhoenixConfigurationUtil.java:306)
at
org.apache.phoenix.mapreduce.PhoenixInputFormat.getQueryPlan(PhoenixInputFormat.java:113)
... 32 more

On Thu, Jan 21, 2016 at 12:41 AM, Nick Dimiduk <ndimiduk@apache.org> wrote:
>
>> I finally got to the bottom of things. There were two issues at play in
>> my particular environment.
>>
>> 1. An Ambari bug [0] means my spark-defaults.conf file was garbage. I
>> hardly thought of it when I hit the issue with MR job submission; its
>> impact on Spark was much more subtle.
>>
>> 2. YARN client version mismatch (Phoenix is compiled vs Apache 2.5.1
>> while my cluster is running HDP's 2.7.1 build), per my earlier email. Once
>> I'd worked around (1), I was able to work around (2) by setting
>> "/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/phoenix-client/phoenix-client-spark.jar"
>> for both spark.driver.extraClassPath and spark.executor.extraClassPath. Per
>> the linked thread above, I believe this places the correct YARN client
>> version first in the classpath.
>>
>> With the above in place, I'm able to submit work vs the Phoenix tables to
>> the YARN cluster. Success!
>>
>> Ironically enough, I would not have been able to work around (2) if we
>> had PHOENIX-2535 in place. Food for thought in tackling that issue. It may
>> be worth while to ship a uberclient jar that is entirely without Hadoop (or
>> HBase) classes. I believe spark does this for Hadoop with their builds as
>> well.
>>
>> Thanks again for your help here Josh! I really appreciate it.
>>
>> [0]: https://issues.apache.org/jira/browse/AMBARI-14751
>>
>> On Wed, Jan 20, 2016 at 2:23 PM, Nick Dimiduk <ndimiduk@apache.org>
>> wrote:
>>
>>> Well, I spoke too soon. It's working, but in local mode only. When I
>>> invoke `pyspark --master yarn` (or yarn-client), the submitted application
>>> goes from ACCEPTED to FAILED, with a NumberFormatException [0] in my
>>> container log. Now that Phoenix is on my classpath, I'm suspicious that the
>>> versions of YARN client libraries are incompatible. I found an old thread
>>> [1] with the same stack trace I'm seeing, similar conclusion. I tried
>>> setting spark.driver.extraClassPath and spark.executor.extraClassPath
>>> to /usr/hdp/current/hadoop-yarn-client:/usr/hdp/current/phoenix-client/phoenix-client-spark.jar
>>> but that appears to have no impact.
>>>
>>> [0]:
>>> 16/01/20 22:03:45 ERROR yarn.ApplicationMaster: Uncaught exception:
>>> java.lang.IllegalArgumentException: Invalid ContainerId:
>>> container_e07_1452901320122_0042_01_000001
>>> at
>>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
>>> at
>>> org.apache.spark.deploy.yarn.YarnRMClient.getAttemptId(YarnRMClient.scala:93)
>>> at
>>> org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:85)
>>> at
>>> org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:574)
>>> at
>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:66)
>>> at
>>> org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:65)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>>> at
>>> org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:65)
>>> at
>>> org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:572)
>>> at
>>> org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:599)
>>> at
>>> org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
>>> Caused by: java.lang.NumberFormatException: For input string: "e07"
>>> at
>>> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>>> at java.lang.Long.parseLong(Long.java:589)
>>> at java.lang.Long.parseLong(Long.java:631)
>>> at
>>> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
>>> at
>>> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
>>> ... 12 more
>>>
>>> [1]:
>>> http://mail-archives.us.apache.org/mod_mbox/spark-user/201503.mbox/%3CCAAqMD1jSEvfyw9oUBymhZukm7f+WTDVZ8E6Zp3L4a9OBJ-hz=A@mail.gmail.com%3E
>>>
>>> On Wed, Jan 20, 2016 at 1:29 PM, Josh Mahonin <jmahonin@gmail.com>
>>> wrote:
>>>
>>>> That's great to hear. Looking forward to the doc patch!
>>>>
>>>> On Wed, Jan 20, 2016 at 3:43 PM, Nick Dimiduk <ndimiduk@apache.org>
>>>> wrote:
>>>>
>>>>> Josh -- I deployed my updated phoenix build across the cluster, added
>>>>> the phoenix-client-spark.jar to configs on the whole cluster, and now
basic
>>>>> dataframe access is now working. Let me see about updating the docs page
to
>>>>> be more clear, I'll send a patch by you for review.
>>>>>
>>>>> Thanks a lot for the help!
>>>>> -n
>>>>>
>>>>> On Tue, Jan 19, 2016 at 5:59 PM, Josh Mahonin <jmahonin@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Right, this cluster I just tested on is HDP 2.3.4, so it's Spark
on
>>>>>> YARN as well. I suppose the JAR is probably shipped by YARN, though
I don't
>>>>>> see any logging saying it, so I'm not certain how the nuts and bolts
of
>>>>>> that work. By explicitly setting the classpath, we're bypassing Spark's
>>>>>> native JAR broadcast though.
>>>>>>
>>>>>> Taking a quick look at the config in Ambari (which ships the config
>>>>>> to each node after saving), in 'Custom spark-defaults' I have the
following:
>>>>>>
>>>>>> spark.driver.extraClassPath ->
>>>>>> /etc/hbase/conf:/usr/hdp/current/phoenix-client/phoenix-client-spark.jar
>>>>>> spark.executor.extraClassPath ->
>>>>>> /usr/hdp/current/phoenix-client/phoenix-client-spark.jar
>>>>>>
>>>>>> I'm not sure if the /etc/hbase/conf is necessarily needed, but I
>>>>>> think that gets the Ambari generated hbase-site.xml in the classpath.
Each
>>>>>> node has the custom phoenix-client-spark.jar installed to that same
path as
>>>>>> well.
>>>>>>
>>>>>> I can pop into regular spark-shell and load RDDs/DataFrames using:
>>>>>> /usr/hdp/current/spark-client/bin/spark-shell --master yarn-client
>>>>>>
>>>>>> or pyspark via:
>>>>>> /usr/hdp/current/spark-client/bin/pyspark
>>>>>>
>>>>>> I also do this as the Ambari-created 'spark' user, I think there
was
>>>>>> some fun HDFS permission issue otherwise.
>>>>>>
>>>>>> On Tue, Jan 19, 2016 at 8:23 PM, Nick Dimiduk <ndimiduk@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm using Spark on YARN, not spark stand-alone. YARN NodeManagers
>>>>>>> are colocated with RegionServers; all the hosts have everything.
There are
>>>>>>> no spark workers to restart. You're sure it's not shipped by
the YARN
>>>>>>> runtime?
>>>>>>>
>>>>>>> On Tue, Jan 19, 2016 at 5:07 PM, Josh Mahonin <jmahonin@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sadly, it needs to be installed onto each Spark worker (for
now).
>>>>>>>> The executor config tells each Spark worker to look for that
file to add to
>>>>>>>> its classpath, so once you have it installed, you'll probably
need to
>>>>>>>> restart all the Spark workers.
>>>>>>>>
>>>>>>>> I co-locate Spark and HBase/Phoenix nodes, so I just drop
it in
>>>>>>>> /usr/hdp/current/phoenix-client/, but anywhere that each
worker can
>>>>>>>> consistently see is fine.
>>>>>>>>
>>>>>>>> One day we'll be able to have Spark ship the JAR around and
use it
>>>>>>>> without this classpath nonsense, but we need to do some extra
work on the
>>>>>>>> Phoenix side to make sure that Phoenix's calls to DriverManager
actually go
>>>>>>>> through Spark's weird wrapper version of it.
>>>>>>>>
>>>>>>>> On Tue, Jan 19, 2016 at 7:36 PM, Nick Dimiduk <ndimiduk@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Tue, Jan 19, 2016 at 4:17 PM, Josh Mahonin <jmahonin@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> What version of Spark are you using?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Probably HDP's Spark 1.4.1; that's what the jars in my
install
>>>>>>>>> say, and the welcome message in the pyspark console agrees.
>>>>>>>>>
>>>>>>>>> Are there any other traces of exceptions anywhere?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No other exceptions that I can find. YARN apparently
doesn't want
>>>>>>>>> to aggregate spark's logs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Are all your Spark nodes set up to point to the same
>>>>>>>>>> phoenix-client-spark JAR?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes, as far as I can tell... though come to think of
it, is that
>>>>>>>>> jar shipped by the spark runtime to workers, or is it
loaded locally on
>>>>>>>>> each host? I only changed spark-defaults.conf on the
client machine, the
>>>>>>>>> machine from which I submitted the job.
>>>>>>>>>
>>>>>>>>> Thanks for taking a look Josh!
>>>>>>>>>
>>>>>>>>> On Tue, Jan 19, 2016 at 5:02 PM, Nick Dimiduk <ndimiduk@apache.org
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi guys,
>>>>>>>>>>>
>>>>>>>>>>> I'm doing my best to follow along with [0], but
I'm hitting some
>>>>>>>>>>> stumbling blocks. I'm running with HDP 2.3 for
HBase and Spark. My phoenix
>>>>>>>>>>> build is much newer, basically 4.6-branch + PHOENIX-2503,
PHOENIX-2568. I'm
>>>>>>>>>>> using pyspark for now.
>>>>>>>>>>>
>>>>>>>>>>> I've added phoenix-$VERSION-client-spark.jar
to both
>>>>>>>>>>> spark.executor.extraClassPath and spark.driver.extraClassPath.
This allows
>>>>>>>>>>> me to use sqlContext.read to define a DataFrame
against a Phoenix table.
>>>>>>>>>>> This appears to basically work, as I see PhoenixInputFormat
in the logs and
>>>>>>>>>>> df.printSchema() shows me what I expect. However,
when I try df.take(5), I
>>>>>>>>>>> get "IllegalStateException: unread block data"
[1] from the workers. Poking
>>>>>>>>>>> around, this is commonly a problem with classpath.
Any ideas as to
>>>>>>>>>>> specifically which jars are needed? Or better
still, how to debug this
>>>>>>>>>>> issue myself. Adding "/usr/hdp/current/hbase-client/lib/*"
to the classpath
>>>>>>>>>>> gives me a VerifyError about netty method version
mismatch. Indeed I see
>>>>>>>>>>> two netty versions in that lib directory...
>>>>>>>>>>>
>>>>>>>>>>> Thanks a lot,
>>>>>>>>>>> -n
>>>>>>>>>>>
>>>>>>>>>>> [0]: http://phoenix.apache.org/phoenix_spark.html
>>>>>>>>>>> [1]:
>>>>>>>>>>>
>>>>>>>>>>> java.lang.IllegalStateException: unread block
data
>>>>>>>>>>> at
>>>>>>>>>>> java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2424)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1383)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>>>>>>>>>>> at
>>>>>>>>>>> java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:69)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:95)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:194)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>>>>>>>>> at
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Dec 21, 2015 at 8:33 AM, James Taylor
<
>>>>>>>>>>> jamestaylor@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for remembering about the docs, Josh.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Dec 21, 2015 at 8:27 AM, Josh Mahonin
<
>>>>>>>>>>>> jmahonin@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Just an update for anyone interested,
PHOENIX-2503 was just
>>>>>>>>>>>>> committed for 4.7.0 and the docs have
been updated to include these samples
>>>>>>>>>>>>> for PySpark users.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://phoenix.apache.org/phoenix_spark.html
>>>>>>>>>>>>>
>>>>>>>>>>>>> Josh
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Dec 10, 2015 at 1:20 PM, Josh
Mahonin <
>>>>>>>>>>>>> jmahonin@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hey Nick,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think this used to work, and will
again once PHOENIX-2503
>>>>>>>>>>>>>> gets resolved. With the Spark DataFrame
support, all the necessary glue is
>>>>>>>>>>>>>> there for Phoenix and pyspark to
play nice. With that client JAR (or by
>>>>>>>>>>>>>> overriding the com.fasterxml.jackson
JARS), you can do something like:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> df = sqlContext.read \
>>>>>>>>>>>>>>   .format("org.apache.phoenix.spark")
\
>>>>>>>>>>>>>>   .option("table", "TABLE1") \
>>>>>>>>>>>>>>   .option("zkUrl", "localhost:63512")
\
>>>>>>>>>>>>>>   .load()
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> df.write \
>>>>>>>>>>>>>>   .format("org.apache.phoenix.spark")
\
>>>>>>>>>>>>>>   .mode("overwrite") \
>>>>>>>>>>>>>>   .option("table", "TABLE1") \
>>>>>>>>>>>>>>   .option("zkUrl", "localhost:63512")
\
>>>>>>>>>>>>>>   .save()
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, this should be added to the
documentation. I hadn't
>>>>>>>>>>>>>> actually tried this till just now.
:)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Dec 9, 2015 at 6:39 PM, Nick
Dimiduk <
>>>>>>>>>>>>>> ndimiduk@apache.org> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Heya,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Has anyone any experience using
phoenix-spark integration
>>>>>>>>>>>>>>> from pyspark instead of scala?
Folks prefer python around here...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I did find this example [0] of
using HBaseOutputFormat from
>>>>>>>>>>>>>>> pyspark, haven't tried extending
it for phoenix. Maybe someone with more
>>>>>>>>>>>>>>> experience in pyspark knows better?
Would be a great addition to our
>>>>>>>>>>>>>>> documentation.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Nick
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [0]:
>>>>>>>>>>>>>>> https://github.com/apache/spark/blob/master/examples/src/main/python/hbase_outputformat.py
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message