livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sarthak Singhal <Sarthak.Sing...@hitachiconsulting.com>
Subject Question for Apache Livy
Date Mon, 14 May 2018 13:15:26 GMT
Hi,

How to set --master, --deploy-mode, --driver-class-path and --driver-java-options through
Apache Livy?
I want to set the master, spark deploy-mode, driver-class-path and driver-java-options for
the Spark job when the job is triggered through Apache Livy without having to restart the
Livy server when these settings change. How to do this since there are no direct options to
do this in Livy?
I understand these are used to set the parameters for Client mode. I want to set these when
I spawn a job in client mode.

Also when I try to set "spark.master" through REST through "conf" param, it is not being set.
This is the JSON I'm sending to the Livy API.
{
                "file": "/user/livy/spark-examples.jar",
                "conf" : {
                                "spark.master": "yarn",
                                "spark.submit.deployMode": "cluster"
                },
                "args": ["2"],
                "className": "org.apache.spark.examples.SparkPi"
}

Thanks and Regards,
Sarthak

Privileged/Confidential Information may be contained in this message and is intended only
for the use of the addressee.
If you are not the addressee, or person responsible for delivering it to the addressee, you
should not copy or deliver
this to anyone else. If you receive this message by mistake, please delete the message from
any/all computer/s and
notify the sender immediately by reply Email. We appreciate your assistance in preserving
the confidentiality of our
correspondence. Any information in this message that does not relate to the official business
of the organization
shall be understood as neither given nor endorsed by it. Please advise immediately if you
or your employer
does not consent to Internet Email for messages of this kind. Thank you. (HCID0411)

Mime
View raw message