livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsch, Tim" <>
Subject Re: Use existing SparkSession in POST/batches request
Date Fri, 16 Nov 2018 19:31:35 GMT
Interactive sessions also allow you to upload jars.  Consider trying to transform your batch
job, or wrapping your job, into interactive code by giving your batch job an entry point method.

class JumboBatchRunner {
   public static runBatch( SparkContext sc, String startDate, String endDate, String pipeline,
   runSpiders ) {

Now execute a statement using the session, with code like:

Import JumboBatchRunner;
JumboBatchRunner.runBatch( sc, startDate, endData, “db-importer”, true);

vary the parameters and submit your 250 statements, you can even submit them in parallel so
long as you don’t have more than livy.rsc.retained-statements job results outstanding


From: Shubham Gupta <>
Reply-To: "" <>
Date: Friday, November 16, 2018 at 4:43 AM
To: "" <>
Subject: Use existing SparkSession in POST/batches request

[External Email]
Hi committers and contributors

I have this query since long, which I think I know the answer to, but am still looking for

Can I make use of the SparkSession that I created using POST/sessions request for submitting
my Spark job using POST/batches request?

Here's the link<> to my complete (elaborated) question
on StackOverflow.
Luqman Ghani's response<>
already hints that this isn't possible, but

  *   Is it on the roadmap?
  *   Any workarounds?


Shubham Gupta
Software Engineer
View raw message