livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Seybold <daniel.seyb...@uni-ulm.de>
Subject How to deploy generic Spark applications via Livy using the Java client
Date Tue, 04 Sep 2018 15:36:24 GMT
Hi guys,

I'd like to use Livy Server and its Java client to deploy generic Spark 
applications by integrating the Java client into a custom orchestration 
engine.

After going through the docs and experimenting the code I am not sure if 
this is already possible with Livy, see the following example:

The orchestration engine can reveive generic Spark binaries and 
additional input parameters, which should be executed programmitcally at 
a Spark Cluster (with the Livy Server).

Yet, according to the PiJob 
<https://livy.incubator.apache.org/docs/latest/programmatic-api.html>example 
it seems that I need to wrap the code of any Spark application to submit 
it via the Java client?

Hence, the following code snippet would not work:

LivyClient client =new LivyClientBuilder()
     .setURI(new URI("http://IP:8998")).build(); client.uploadJar(new File("genberic-spark-app.jar"));

Yet, it seems that this kind of execution would be possible by using the 
REST-API directly (and not the Java Client)?

Thanks for any advice!

Cheers,
Daniel

Mime
View raw message