livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsch, Tim" <Tim.Har...@Teradata.com>
Subject %json magick of dataframe drops the datatype information
Date Fri, 11 May 2018 19:53:13 GMT
Hi,

I'm trying to understand why, when I use %json magic, dataType is empty?  As you can see in
this gist:


https://gist.github.com/harschware/a29dc3224d3dea6bb749790244d39d72


<https://gist.github.com/harschware/a29dc3224d3dea6bb749790244d39d72>
all structs in the schema come back with just "name" and "nullable" populated.  But, as you
can see in values I have two columns that are integer and the rest are string.  This matches
the definition of the hive table.  and I can verify it by sending d.printSchema() to livy.

Should this be populated with the datatype, or am I missing something?


Thanks,

Tim

Mime
View raw message