livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wandong Wu <wu...@husky.neu.edu>
Subject Some questions about cached data in Livy
Date Wed, 11 Jul 2018 09:46:46 GMT
Dear Sir or Madam:
 
      I am a Livy beginner. I use Livy, because within an interactive session, different spark
jobs could share cached RDDs or DataFrames.
 
      When I read some parquet files and create a table called “TmpTable”. The following
queries will use this table. Does it mean this table has been cached?
      If cached, where is the table cached? The table is cached in Livy or Spark cluster?
 
      Spark also supports cache function.  When I read some parquet files and create a table
called “TmpTable2”. I add such code: sql_ctx.cacheTable('tmpTable2').
      In the next query using this table. It will be cached in Spark cluster. Then the following
queries could use this cached table.
 
      What is the difference between cached in Livy and cached in Spark cluster?
 
Thanks!
 
Yours
Wandong
 
Mime
View raw message