[LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue#529
[LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue#529nileshrathi345 wants to merge 5 commits into
Conversation
|
@ArnavBalyan, @gyogal and @roczei could you please take a look when possible thanks! |
ArnavBalyan
left a comment
ArnavBalyan
left a comment
There was a problem hiding this comment.
Thanks for the changes @nileshrathi345, could you pls look into the CI failures, overall LGTM
gyogal
left a comment
gyogal
left a comment
There was a problem hiding this comment.
Thanks for submitting a PR @nileshrathi345 ! I left two minor comments, could you please check and let me know your thoughts?
| m.owner, | ||
| m.proxyUser, | ||
| sessionStore, | ||
| None, |
There was a problem hiding this comment.
I think I now understand why the queue was added to the state store in the previous version, it was so that this value could be populated correctly. Is the queue in BatchSessions used by anything else other than the unit test?
There was a problem hiding this comment.
Currently, queue is primarily used during session creation and tracking. However, passing None during recover means that if the Livy server restarts, any recovered batch session will permanently lose its original queue context.
To maintain that have added to the state in previous version
There was a problem hiding this comment.
It might be possible to remove this and change the unit test in BatchSessionSpec to verify that the queue is set via the mockApp. I am not entirely sure though, but if that is the case this PR could be further simplified.
3 participants
What changes were proposed in this pull request?
This PR introduces support for configuring a default YARN queue at the Livy server level via a new configuration property:
livy.spark.yarn.queue.Previously, if a client did not explicitly specify a YARN queue parameter (queue) in their session creation payload, the session would fall back to the global Hadoop cluster's default YARN queue. This change allows cluster administrators to isolate Livy-generated workloads into a specific default queue without requiring end-users to pass it manually in every API request.
How was this patch tested?
This patch has been thoroughly validated using both newly added automated unit tests and manual API validation: