phoenix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Wysakowicz <>
Subject Problem with arrays in phoenix-spark
Date Mon, 30 Nov 2015 18:01:53 GMT

I've recently found some behaviour that I found buggy when working with
phoenix-spark and arrays.

Take a look at those unit tests:

  test("Can save arrays from custom dataframes back to phoenix") {
    val dataSet = List(Row(2L, Array("String1", "String2", "String3")))

    val sqlContext = new SQLContext(sc)

    val schema = StructType(
        Seq(StructField("ID", LongType, nullable = false),
            StructField("VCARRAY", ArrayType(StringType))))

    val rowRDD = sc.parallelize(dataSet)

    // Apply the schema to the RDD.
    val df = sqlContext.createDataFrame(rowRDD, schema)

      .options(Map("table" -> "ARRAY_TEST_TABLE", "zkUrl" -> quorumAddress))

  test("Can save arrays of AnyVal type back to phoenix") {
    val dataSet = List((2L, Array(1, 2, 3), Array(1L, 2L, 3L)))

        Seq("ID", "INTARRAY", "BIGINTARRAY"),
        zkUrl = Some(quorumAddress)

    // Load the results back
    val stmt = conn.createStatement()
    val rs = stmt.executeQuery("SELECT INTARRAY, BIGINTARRAY FROM
    val intArray = rs.getArray(1).getArray().asInstanceOf[Array[Int]]
    val longArray = rs.getArray(2).getArray().asInstanceOf[Array[Long]]

    // Verify the arrays are equal
    intArray shouldEqual dataSet(0)._2
    longArray shouldEqual dataSet(0)._3

Both fail with some ClassCastExceptions.

In attached patch I've proposed a solution. The tricky part is with
Array[Byte] as this would be same for both VARBINARY and TINYINT[].

Let me know If I should create an issue for this, and if my solution
satisfies you.

Dawid Wysakowicz

View raw message