Add async recursive delete for apps and service instances by johha · Pull Request #5299 · cloudfoundry/cloud_controller_ng

johha · 2026-07-20T15:41:32Z

Behind temporary_enable_async_recursive_delete flag (default off). Recursive delete jobs re-enqueue instead of failing when sub-resources (service bindings) are deleted asynchronously, waiting for them to settle and surfacing the original broker error on failure. Adds root/sub job tracking via root_job_guid on the jobs table. Foundation for future recursive deletes (org, space).

Thanks for contributing to cloud_controller_ng. To speed up the process of reviewing your pull request please provide us with:

A short explanation of the proposed change:
An explanation of the use cases your change solves
Links to any other associated PRs
I have reviewed the contributing guide
I have viewed, signed, and submitted the Contributor License Agreement
I have made this pull request to the main branch
I have run all the unit tests using bundle exec rake
I have run CF Acceptance Tests

Behind `temporary_enable_async_recursive_delete flag` (default off). Recursive delete jobs re-enqueue instead of failing when sub-resources (service bindings) are deleted asynchronously, waiting for them to settle and surfacing the original broker error on failure. Adds root/sub job tracking via `root_job_guid` on the jobs table. Foundation for future recursive deletes (org, space).

philippthun · 2026-08-03T11:48:28Z


        result
+      rescue SubResourceError => e
+        raise if !@fail_if_in_progress && e.any_in_progress? # re-raise SubResourceError so that root job continues to run


At this point @fail_if_in_progress will always be false.

philippthun · 2026-08-03T13:16:25Z

+        resource_model: AppModel,
+        resource_guid: app.guid,
+        operation: 'app.delete'
+      ) { VCAP::CloudController::V3::RecursiveDeleteAppJob.new(app.guid, user_audit_info) }


Use do |locked_instance| for symmetry.

philippthun · 2026-08-03T13:56:42Z

+      end
+
+      def raise_if_sub_jobs_failed
+        return if sub_job_errors.empty?


Constructs sub_job_errors only to check if it is empty or not. Should do something like sub_jobs.any? { |s| s.state == PollableJobModel::FAILED_STATE } instead.

philippthun · 2026-08-03T14:03:14Z

+      def sub_jobs_in_flight?
+        return false if active_sub_jobs.empty?
+
+        add_in_progress_warning(root_job)


Somewhat unexpected that this predicate method has a side-effect (the warning).

philippthun · 2026-08-03T14:09:41Z

+          AppStop.stop(app: app, user_audit_info: @user_audit_info, delete_triggered: true) if app.desired_state != ProcessModel::STOPPED
+          AppDelete.new(@user_audit_info).delete([app])
+          finish
+        end


Instead of log_failed_bindings (and log_failed_children in RecursiveDeleteServiceInstanceJob):

rescue CloudController::Errors::CompoundError => e // log errors from the CompoundError raise

Maybe this could even go into the RootJobMixin.

philippthun · 2026-08-03T14:39:59Z

+    end
+
+    def any_in_progress?
+      in_progress_operations.any?


@errors.any? { |e| e.is_a?(AsyncOperationInProgress) } is slightly more efficient.

philippthun · 2026-08-03T15:08:47Z

      end
+
+      def with_root_job_guid_set
+        return yield if @root_job_guid.nil?


Does this make sense? This would mean that there could be logs with an old root_job_guid, or?

philippthun · 2026-08-03T15:16:21Z

+      drop_column :jobs, :root_job_guid, if_exists: true
+    end
+
+    if database_type == :mysql


philippthun · 2026-08-03T15:26:06Z

+        update_last_operation_with_failure(e.message) unless service_instance.operation_in_progress?
+        raise e
      rescue StandardError => e
        update_last_operation_with_failure(e.message) unless service_instance.operation_in_progress?
        raise e


Most parts of the rescue blocks are identical, so maybe merge them:

rescue StandardError => e # In-progress sub-resource deletions aren't failures: re-raise so the root job re-enqueues and waits. raise if e.is_a?(SubResourceError) && e.any_in_progress? update_last_operation_with_failure(e.message) unless service_instance.operation_in_progress? raise e end

philippthun · 2026-08-03T15:28:35Z

+      def initialize(app_guid, user_audit_info)
+        super()
+        @app_guid = app_guid


I would favor the usage of resource_guid here as well (consistent with RecursiveDeleteServiceInstanceJob and the RootJobMixin).

philippthun · 2026-08-04T11:28:31Z

    end
  end
-
-  describe 'DeleteAppErrorTranslatorJob' do


I think this context should be restored and just use the new VCAP::CloudController::SubResourceError class.

philippthun · 2026-08-04T12:02:20Z

@@ -21,14 +23,26 @@ def enqueue(job, run_at: nil, priority_increment: nil)
      end

      def enqueue_pollable(job, existing_guid: nil, run_at: nil, priority_increment: nil, preserve_priority: false)
-        wrapped_job = PollableJobWrapper.new(job, existing_guid:)
+        wrapped_job = PollableJobWrapper.new(job, existing_guid:, root_job_guid:)


Instead of accessing a variable that is set by a sub-class the following would be easier to read:

class Enqueuer def enqueue_pollable(job, existing_guid: nil, run_at: nil, priority_increment: nil, preserve_priority: false) wrapped_job = PollableJobWrapper.new(job, existing_guid:, root_job_guid: current_root_job_guid) end private def current_root_job_guid nil end

class GenericEnqueuer < Enqueuer def current_root_job_guid @root_job_guid end end

- Rename RootJobMixin -> RecursiveDeleteRootJobMixin (it is delete-specific) - Move defer-while-in-flight and raise-if-failed into the mixin; keep sub_jobs_in_flight? side-effect-free - Log sub-resource failures via rescue (log_immediate_failures / log_recursive_delete_failure) instead of looping over sub_resource_errors - Merge the two SubResourceError/StandardError rescue blocks in ServiceInstanceDelete#delete - raise_if_sub_jobs_failed short-circuits with any? instead of building errors - Memoize @service_instance per pass (cleared in ensure); use resource_guid consistently in both recursive-delete jobs - Add ServiceInstance#route_bindings association and use it - SubResourceError#any_in_progress? uses any? (drop in_progress_operations) - Enqueuer exposes current_root_job_guid overridden by GenericEnqueuer - apps_controller: build the job from the locked resource for symmetry - logging_context: align save/restore variable naming - migration: use elsif in the down block - Restore DeleteAppErrorTranslatorJob spec; assert FOR UPDATE via have_queried_db_times

johha force-pushed the recursive-app-service-delete branch 8 times, most recently from 1539023 to a87ea6c Compare July 22, 2026 12:17

johha added 2 commits July 22, 2026 17:33

temp logging

05ca1aa

johha force-pushed the recursive-app-service-delete branch from a87ea6c to 05ca1aa Compare July 22, 2026 15:45

johha added 4 commits July 23, 2026 13:29

fix logger

52c1b1f

improve audit log so it shows in CF CLI

1029646

Ensure last operation is set to in progress/failed + better logging

8dbfcc5

revert setting SI last_operation for sub resource state

8e4afae

philippthun reviewed Aug 3, 2026

View reviewed changes

philippthun reviewed Aug 4, 2026

View reviewed changes

johha added 2 commits August 4, 2026 16:25

Resructure mixin

c52da16

johha force-pushed the recursive-app-service-delete branch 2 times, most recently from d472e26 to c52da16 Compare August 5, 2026 12:59

simplify

547f774

Navigation Menu

Uh oh!

Add async recursive delete for apps and service instances - #5299

Add async recursive delete for apps and service instances#5299
johha wants to merge 9 commits into
mainfrom
recursive-app-service-delete

johha commented Jul 20, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 3, 2026

Uh oh!

philippthun Aug 4, 2026

Uh oh!

Uh oh!

philippthun Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

johha commented Jul 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants