livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Romanenko <aromanenko....@gmail.com>
Subject Re: Unexpected behaviour of config option “livy.server.session.max-creation”
Date Thu, 22 Nov 2018 16:32:32 GMT
+dev@

I’ve created a Jira [1] and PR [2] for that.

[1] https://issues.apache.org/jira/browse/LIVY-535 <https://issues.apache.org/jira/browse/LIVY-535>
[2] https://github.com/apache/incubator-livy/pull/129 <https://github.com/apache/incubator-livy/pull/129>


> On 19 Nov 2018, at 19:00, Alexey Romanenko <aromanenko.dev@gmail.com> wrote:
> 
> Hi all,
> 
> I’d like to confirm that the current behaviour of config option “livy.server.session.max-creation”
 is correct or not.
> 
> For example, I set the value of this option to “1” and try to submit 5 batch jobs
(example SparkPi) on Livy almost in the same time using a command like this: 
> “for i in {1..5}; do curl -X POST --data '{"file": "/tmp/spark-examples-2.jar", "className":
"org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" http://localhost:8999/batches
<http://localhost:8999/batches>; done”. 
> 
> In such case, I expect that the only one job (first) will be submitted properly and others
will be rejected with a response message “Rejected, too many sessions are being created!”.
This is what I actually have when I submit them manually, one by one. Though, in the case
given above, almost always all 5 jobs will be submitted properly (sometimes only one will
be rejected) despite of the value of config option.
> 
> Looks like, that code, when Livy checks the number of already created sessions and launch
new one, is not atomic:
> https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130
<https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130>
> 
> Also, the calculation of the total number of child processes in "tooManySessions()" is
not atomic operation.
> 
> So, when I submit the jobs almost in the same moment (POST request takes only about tens
of milliseconds) then I can spawn more spark-submit processes despite of maximum limitation.

> 
> If it’s a bug then I will create a Jira and try to fix it.
> 
> 
> 


Mime
View raw message