Skip to content

fix(toolkit-lib): finalizing deployment after refactor intermittently fails while stacks are still in UPDATE_IN_PROGRESS - #1805

Open
go-to-k wants to merge 5 commits into
aws:mainfrom
go-to-k:fix-refactor-stabilize
Open

fix(toolkit-lib): finalizing deployment after refactor intermittently fails while stacks are still in UPDATE_IN_PROGRESS#1805
go-to-k wants to merge 5 commits into
aws:mainfrom
go-to-k:fix-refactor-stabilize

Conversation

@go-to-k

@go-to-k go-to-k commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #1804

Problem

CloudFormation marks a stack refactor's ExecutionStatus as EXECUTE_COMPLETE while the affected stacks may still be in UPDATE_IN_PROGRESS for a few more seconds (observed: 0.5-3 seconds). RefactoringContext.execute() only waits for the refactor status (waitUntilStackRefactorExecuteComplete), and Toolkit._refactor() then immediately starts the finalizing deployment. Since synthesis is done and the assets are already published at that point, the deployment reaches CreateChangeSet within ~1-2 seconds and frequently fails with:

ValidationError: Stack:arn:... is in UPDATE_IN_PROGRESS state and can not be updated.

Whether it fails depends on where the waiter's 6-second polling of DescribeStackRefactor lands relative to this window, so the failure is intermittent but frequent (2 out of 3 attempts in the reporter's environment).

Fix

After waitUntilStackRefactorExecuteComplete, RefactoringContext.execute() now waits for every stack involved in the mappings (source and destination) to stabilize, reusing the existing stabilizeStack helper from api/deployments/cfn-api. The wait is placed inside execute() rather than in Toolkit._refactor() so that any caller of execute() is protected.

Testing

  • New unit tests in test/api/refactoring/context-execute.test.ts (fake timers, same style as cfn-api-stabilization-polling-interval.test.ts): execute() does not resolve while the stack is UPDATE_IN_PROGRESS and resolves only after it reaches UPDATE_COMPLETE; in a cross-stack refactor both source and destination stacks are waited on.
  • Verified against a real AWS environment: with this fix, in one of the runs the CLI observed the stack still in UPDATE_IN_PROGRESS (Refactor ... initiated) right after the refactor waiter returned, waited for it to stabilize, and the finalizing deployment then succeeded:
[03:17:33] Waiting for stack RefactorReproStack to finish creating or updating...
[03:17:33] Stack RefactorReproStack has an ongoing operation in progress and is not stable (UPDATE_IN_PROGRESS (Refactor 7689cad2-... initiated))
✅  Stack refactor complete
Deploying updated stacks to finalize refactor...
✅  RefactorReproStack (no changes)

This is exactly the timing where the current CLI fails.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

auto-merge was automatically disabled August 9, 2026 18:34

Head branch was pushed to by a user without write access

@codecov-commenter

codecov-commenter commented Aug 9, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.27%. Comparing base (6551740) to head (e2744be).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1805      +/-   ##
==========================================
- Coverage   90.32%   90.27%   -0.05%     
==========================================
  Files          80       80              
  Lines       12124    12124              
  Branches     1716     1714       -2     
==========================================
- Hits        10951    10945       -6     
- Misses       1139     1145       +6     
  Partials       34       34              
Flag Coverage Δ
suite.unit 90.27% <ø> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@go-to-k go-to-k changed the title fix(toolkit-lib): wait for stacks to stabilize before finalizing refactor with a deployment fix(toolkit-lib): refactor finalizing deployment intermittently fails while stacks are still in UPDATE_IN_PROGRESS Aug 9, 2026
@go-to-k go-to-k changed the title fix(toolkit-lib): refactor finalizing deployment intermittently fails while stacks are still in UPDATE_IN_PROGRESS fix(toolkit-lib): finalizing deployment after refactor intermittently fails while stacks are still in UPDATE_IN_PROGRESS Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(refactor): finalizing deployment fails because stacks are still in UPDATE_IN_PROGRESS after the refactor completes

2 participants