livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Meisam Fathi <meisam.fa...@gmail.com>
Subject Re: Use existing SparkSession in POST/batches request
Date Fri, 16 Nov 2018 14:32:09 GMT
Hi Shubham!

POST/sessions is to create a session and submit statements to it. Once the
session is up, it can accept as many statements as needed. Users can
interactively submit new statements to it.
POST/batches is to send one Spark application. Anything that the job needs
to run is submit with the POST request (jar files, or a python script, or).
Once the job is submitted, users cannot interactively send new statements
to the application to run.

The only interesting case is when you submit a streaming job with
POST/batches which runs forever (or until it fails/dies).

Thanks,
Meisam

On Fri, Nov 16, 2018 at 4:43 AM Shubham Gupta <y2k.shubhamgupta@gmail.com>
wrote:

> Hi committers and contributors
>
> I have this query since long, which I think I know the answer to, but am
> still looking for confirmation.
>
> Can I make use of the SparkSession that I created using POST/sessions request
> for submitting my Spark job using POST/batches request?
>
> Here's the link <https://stackoverflow.com/q/51746286> to my complete
> (elaborated) question on StackOverflow.
> *Luqman Ghani'*s response
> <https://groups.google.com/a/cloudera.org/d/msg/livy-user/TFLc7wIaQ8Y/EydZM-4HBAAJ>
> already hints that this isn't possible, but
>
>    - Is it on the roadmap?
>    - Any workarounds?
>
>
> Thanks
>
> *Shubham Gupta*
> Software Engineer
>  zomato
>

Mime
View raw message