livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsch, Tim" <Tim.Har...@Teradata.com>
Subject Re: Use existing SparkSession in POST/batches request
Date Fri, 16 Nov 2018 19:31:35 GMT
Interactive sessions also allow you to upload jars.  Consider trying to transform your batch
job, or wrapping your job, into interactive code by giving your batch job an entry point method.

class JumboBatchRunner {
   public static runBatch( SparkContext sc, String startDate, String endDate, String pipeline,
boolean
   runSpiders ) {
   ....
   }
}

Now execute a statement using the session, with code like:

Import JumboBatchRunner;
JumboBatchRunner.runBatch( sc, startDate, endData, “db-importer”, true);

vary the parameters and submit your 250 statements, you can even submit them in parallel so
long as you don’t have more than livy.rsc.retained-statements job results outstanding

--


From: Shubham Gupta <y2k.shubhamgupta@gmail.com>
Reply-To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Date: Friday, November 16, 2018 at 4:43 AM
To: "user@livy.incubator.apache.org" <user@livy.incubator.apache.org>
Subject: Use existing SparkSession in POST/batches request

[External Email]
________________________________
Hi committers and contributors

I have this query since long, which I think I know the answer to, but am still looking for
confirmation.

Can I make use of the SparkSession that I created using POST/sessions request for submitting
my Spark job using POST/batches request?

Here's the link<https://stackoverflow.com/q/51746286> to my complete (elaborated) question
on StackOverflow.
Luqman Ghani's response<https://groups.google.com/a/cloudera.org/d/msg/livy-user/TFLc7wIaQ8Y/EydZM-4HBAAJ>
already hints that this isn't possible, but

  *   Is it on the roadmap?
  *   Any workarounds?

Thanks

Shubham Gupta
Software Engineer
 zomato
Mime
View raw message