phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Mahonin <jmaho...@gmail.com>
Subject Re: Problem with arrays in phoenix-spark
Date Mon, 30 Nov 2015 21:22:18 GMT
Hi David,

Thanks for the bug report and the proposed patch. Please file a JIRA and
we'll take the discussion there.

Josh

On Mon, Nov 30, 2015 at 1:01 PM, Dawid Wysakowicz <
wysakowicz.dawid@gmail.com> wrote:

> Hi,
>
> I've recently found some behaviour that I found buggy when working with
> phoenix-spark and arrays.
>
> Take a look at those unit tests:
>
>   test("Can save arrays from custom dataframes back to phoenix") {
>     val dataSet = List(Row(2L, Array("String1", "String2", "String3")))
>
>     val sqlContext = new SQLContext(sc)
>
>     val schema = StructType(
>         Seq(StructField("ID", LongType, nullable = false),
>             StructField("VCARRAY", ArrayType(StringType))))
>
>     val rowRDD = sc.parallelize(dataSet)
>
>     // Apply the schema to the RDD.
>     val df = sqlContext.createDataFrame(rowRDD, schema)
>
>     df.write
>       .format("org.apache.phoenix.spark")
>       .options(Map("table" -> "ARRAY_TEST_TABLE", "zkUrl" ->
> quorumAddress))
>       .mode(SaveMode.Overwrite)
>       .save()
>   }
>
>   test("Can save arrays of AnyVal type back to phoenix") {
>     val dataSet = List((2L, Array(1, 2, 3), Array(1L, 2L, 3L)))
>
>     sc
>       .parallelize(dataSet)
>       .saveToPhoenix(
>         "ARRAY_ANYVAL_TEST_TABLE",
>         Seq("ID", "INTARRAY", "BIGINTARRAY"),
>         zkUrl = Some(quorumAddress)
>       )
>
>     // Load the results back
>     val stmt = conn.createStatement()
>     val rs = stmt.executeQuery("SELECT INTARRAY, BIGINTARRAY FROM
> ARRAY_ANYVAL_TEST_TABLE WHERE ID = 2")
>     rs.next()
>     val intArray = rs.getArray(1).getArray().asInstanceOf[Array[Int]]
>     val longArray = rs.getArray(2).getArray().asInstanceOf[Array[Long]]
>
>     // Verify the arrays are equal
>     intArray shouldEqual dataSet(0)._2
>     longArray shouldEqual dataSet(0)._3
>   }
>
> Both fail with some ClassCastExceptions.
>
> In attached patch I've proposed a solution. The tricky part is with
> Array[Byte] as this would be same for both VARBINARY and TINYINT[].
>
> Let me know If I should create an issue for this, and if my solution
> satisfies you.
>
> Regards
> Dawid Wysakowicz
>
>
>

Mime
View raw message