[SPARK-36058][K8S] Add support for statefulset APIs in K8s#33508

Closed

holdenk wants to merge 107 commits into

apache:masterfrom

holdenk:SPARK-36058-support-replicasets-or-job-api-like-things

Commits on Aug 4, 2021

Work in progress supporting statefulsets for Spark. TBD if we want statefulset or job
holdenk
committed
It compiles, yaygit diff!
holdenk
committed
Add applicationId to setTotalExpectedExecutors so that we can use this as the entry point to adjusting Job/Statefulsets
holdenk
committed
Put in a restart policy of always. Next TODO (likely we can look at the volcano example) is getting execID from the podname inside of the container. Then after that switch the one at a time deploym…
holdenk
committed
Add podname parsing logic based on https://github.com/spark-volcano-wip/spark-3-volcano/blob/main/docker/src/main/dockerfiles/spark/entrypoint.sh from the volcano folks
holdenk
committed
Try and plumb through the SPARK_EXECUTOR_POD_NAME so we can process it, ignore snapshots in replicaset mode
holdenk
committed
Move the restart policy change into basic exec feature step instead of doing weird stuff
holdenk
committed
Move more of the hijinks into the featuresteps where they fit
holdenk
committed
Add a parallel pod management property, lets hope this doesn't screw with the IDs too _much_
holdenk
committed
Fix typo )) -> )
holdenk
committed
Get it to compile again
holdenk
committed
Turns out we do want to track snapshots so know about dead pods earlier than timeouts anyways. Extract exec ids from pod names if the exec label is null or set to the generic EXECID value.
holdenk
committed
Refactor the stateful allocator out from the base allocator (TODO: actually construct it and test it)
holdenk
committed
Use scale to update statefulset scale
holdenk
committed
Construct the pod allocator based on user configuration.
holdenk
committed
Start adding new tests (slowly) for statefulset allocator and update basicexecfeaturestepsuite to match the new podname env
holdenk
committed
Initial statefulset mock test
holdenk
committed
Add second resource profile and scaleup test to StatefulsetAllocatorSuite, explicitly set wait to false for scaleup in StatefulsetPodsAllocator, fix bug of not keeping track of setsCreated
holdenk
committed
Validate the deletions as well
holdenk
committed
Verify that we can allocate with statefulsets. Next up: clean up the stateful set and it's pods automatically and verify this in an integration test.
holdenk
committed
Start work to cleanup and validate removal of statefulset on driver exit
holdenk
committed
Fix addowner ref
holdenk
committed
Delegate the pod cleanup to the pod allocator so that the statefulset or pods by label can be deleted as needed.
holdenk
committed
Use eventually when checking for set delition because it depends on the K8s GC which could take some time.
holdenk
committed
Make the KubernetesSuite pod log collection resilent to pending pods.
holdenk
committed
Add a minireadwrite test for use with PVCs and not proper DFS
holdenk
committed
Add some tests around the new allocator with PVs
holdenk
committed
maaaybe exec mount
holdenk
committed
Revert "maaaybe exec mount"
holdenk
committed
Update the mini-read-write test to handle the fact the exec PVCs are dynamically allocated and don't have the same view as the driver PVC
holdenk
committed
Switch the PV tests back tohaving pvTestTag and MiniKubeTag as needed for Jenkins env
holdenk
committed
Scala style cleanups
holdenk
committed
Delete block when putting over an existing block incase our in-memory block info doesn't match on-disk (e.g. a restarted container.)
holdenk
committed
We do the deletion of the pods inside of the executorpodsallocator now because the method of doing the delete can change, change the tests to validate the stop is called and then validate the stop …
holdenk
committed
Handle empty pod specs
holdenk
committed
Update StatefulsetPodsAllocator.scala
holdenk
committed
code review feedback, cleanup the SPARK_LOCAL_DIRS when executing entrypoint if defined, allow user to specify a arbitrary abstract pods allocator, more precise exception
holdenk
committed
Expose the AbstractPodsAllocator as a @DeveloperApi as suggested/requested by Dongjoon & Aaruna
holdenk
committed
Move the getItems inside of the eventually otherwise we still could have a race with the deletion propegating
holdenk
committed
pvTestTag was removed upstream
holdenk
committed

Commits on Aug 5, 2021

Commits on Aug 9, 2021

Commits on Aug 13, 2021

Commits on Aug 17, 2021

Commits on Aug 18, 2021

Commits on Aug 20, 2021

Merge branch 'master' into SPARK-36058-support-replicasets-or-job-api-like-things
holdenk
committed

Commits on Aug 23, 2021

Commits on Aug 24, 2021

Merge branch 'master' into SPARK-36058-support-replicasets-or-job-api-like-things
holdenk
committed

Commits on Aug 25, 2021

Merge branch 'master' into SPARK-36058-support-replicasets-or-job-api-like-things
holdenk
committed