livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsch, Tim" <>
Subject %json magick of dataframe drops the datatype information
Date Fri, 11 May 2018 19:53:13 GMT

I'm trying to understand why, when I use %json magic, dataType is empty?  As you can see in
this gist:

all structs in the schema come back with just "name" and "nullable" populated.  But, as you
can see in values I have two columns that are integer and the rest are string.  This matches
the definition of the hive table.  and I can verify it by sending d.printSchema() to livy.

Should this be populated with the datatype, or am I missing something?



View raw message