livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <>
Subject Re: What happens if Livy server crashes ? All the spark jobs are gone?
Date Mon, 12 Mar 2018 08:58:27 GMT
Sorry I see there is a recovery mode and also I can set state store to
zookeeper but looks like I need YARN? because I get the error message below

"requirement failed: Session recovery requires YARN"

I am using spark standalone and I don't use YARN anywhere in my cluster. is
there any other option for recovery in this case?

On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <> wrote:

> Hi All,
> When my live server crashes it looks like all my spark jobs are gone. I am
> trying to see how I can make it more resilient? other words, I would like
> spark jobs that were spawned by Livy to be running even if my Livy server
> crashes because in theory Livy server can crash anytime and Spark Jobs
> should run for weeks or months in my case. How can I achieve this?
> Thanks!

View raw message