livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ghosh, Parthapratim " <>
Subject RE: Livy session
Date Mon, 21 May 2018 07:54:36 GMT

                Not sure of your exact use case. I was also mentioning different user code
in the same job’s codebase. My use case is different input while calling the same job multiple

Thanks and Regards
Partha Pratim Ghosh

From: Malveeka Bhandari []
Sent: Monday, May 21, 2018 3:43 AM
Subject: Re: Livy session


Thanks for the idea. However, my use case is a little bit complicated. The job code remains
constant, but the user code goes in as a dependent class. For e.g.  the job might be to query
data, filter for some constraints and run a simulation (multiple transformations and actions)
on the data. This job is constant. One of the functions inside the simulation would need the
user code. So I would need to add the user code as a jar as opposed to a job class.

Is that possible in any way ?

Thanks !

On Mon, 21 May 2018 at 12:46 PM, Ghosh, Parthapratim <<>>
Hi Malveeka,

                I am another user in the forum. However, your requirement may be met without
creating a new jar. You may put the user code in a defined job as an instance level variable
and use it everytime you use the client to submit a job. To call this, create a LivyClient
and submit the job everytime with a new code as required, instantiating the job with the client
code in overloaded constructor.

Thanks and Regards
Partha Pratim Ghosh

From: Malveeka Bhandari [<>]
Sent: Monday, May 21, 2018 2:58 AM
Subject: Fwd: Livy session

Hi everyone.

We are using Livy for our custom use case where a user can submit a batch request. We send
the user code in spark.jars configuration. But we are having to make a new session everytime.
Is there any way that there could be one running session for a user and we could keep submitting
new jobs but with a new dependent jar each time.

View raw message