livy-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Romanenko <aromanenko....@gmail.com>
Subject Unexpected behaviour of config option “livy.server.session.max-creation”
Date Mon, 19 Nov 2018 18:00:36 GMT
Hi all,

I’d like to confirm that the current behaviour of config option “livy.server.session.max-creation”
 is correct or not.

For example, I set the value of this option to “1” and try to submit 5 batch jobs (example
SparkPi) on Livy almost in the same time using a command like this: 
“for i in {1..5}; do curl -X POST --data '{"file": "/tmp/spark-examples-2.jar", "className":
"org.apache.spark.examples.SparkPi"}' -H "Content-Type: application/json" http://localhost:8999/batches;
done”. 

In such case, I expect that the only one job (first) will be submitted properly and others
will be rejected with a response message “Rejected, too many sessions are being created!”.
This is what I actually have when I submit them manually, one by one. Though, in the case
given above, almost always all 5 jobs will be submitted properly (sometimes only one will
be rejected) despite of the value of config option.

Looks like, that code, when Livy checks the number of already created sessions and launch
new one, is not atomic:
https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130
<https://github.com/apache/incubator-livy/blob/4cfb6bcb8fb9ac6b2d6c8b3d04b20f647b507e1f/server/src/main/scala/org/apache/livy/server/SessionServlet.scala#L130>

Also, the calculation of the total number of child processes in "tooManySessions()" is not
atomic operation.

So, when I submit the jobs almost in the same moment (POST request takes only about tens of
milliseconds) then I can spawn more spark-submit processes despite of maximum limitation.


If it’s a bug then I will create a Jira and try to fix it.




Mime
View raw message