livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: What happens if Livy server crashes ? All the spark jobs are gone?
Date Tue, 20 Mar 2018 22:33:46 GMT
@Meisam Fathi I am running with yarn and zookeeper as a state store. I
spawned a job via livy that reads from kafka and writes to Kafka
but the moment I kill the livy server the job also is getting killed. not
sure why? I believe once the livy server crashes the spark context also
get's killed so do I need to need to set the livy.spark.deploy.mode ? if
so, what value should I set it to?


On Mon, Mar 12, 2018 at 12:30 PM, Meisam Fathi <meisam.fathi@gmail.com>
wrote:

> On YARN, your application keeps running even if the launcher fails. So
> after recovery, Livy reconnects to the application. On Spark standalone, I
> am not sure what happens to the application of the launcher fails.
>
> Thanks,
> Meisam
>
> On Mon, Mar 12, 2018 at 10:34 AM kant kodali <kanth909@gmail.com> wrote:
>
>> can someone please explain how YARN helps here? And why not spark master?
>>
>> On Mon, Mar 12, 2018 at 3:41 AM, Matteo Durighetto <
>> m.durighetto@miriade.it> wrote:
>>
>>>
>>>
>>> 2018-03-12 9:58 GMT+01:00 kant kodali <kanth909@gmail.com>:
>>>
>>>> Sorry I see there is a recovery mode and also I can set state store to
>>>> zookeeper but looks like I need YARN? because I get the error message below
>>>>
>>>> "requirement failed: Session recovery requires YARN"
>>>>
>>>>
>>>> I am using spark standalone and I don't use YARN anywhere in my
>>>> cluster. is there any other option for recovery in this case?
>>>>
>>>>
>>>> On Sun, Mar 11, 2018 at 11:57 AM, kant kodali <kanth909@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> When my live server crashes it looks like all my spark jobs are gone.
>>>>> I am trying to see how I can make it more resilient? other words, I would
>>>>> like spark jobs that were spawned by Livy to be running even if my Livy
>>>>> server crashes because in theory Livy server can crash anytime and Spark
>>>>> Jobs should run for weeks or months in my case. How can I achieve this?
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>> Hello,
>>>              to enable recovery in Livy you need Spark on YARN
>>>
>>> ( https://spark.apache.org/docs/latest/running-on-yarn.html )
>>>
>>>
>>>
>>> Kind Regards
>>>
>>
>>

Mime
View raw message