implement Pause/UnpauseActivityExecution for standalone activities

[spkane31]

What changed?

Implement PauseActivityExecution and UnpauseActivityExecution for standalone activities. Previously both handlers returned Unimplemented for the SAA path. They now use chasm.UpdateComponent to apply pause/unpause state directly to the CHASM Activity component, matching the semantics of
the existing workflow-activity implementation.

• Proto (activity_state.proto): Added ActivityPauseState message (pause_time, identity, reason) and a pause_state field on ActivityState.
• handlePauseRequested: Sets PauseState on the component. If the activity is in SCHEDULED state, increments the attempt stamp so the existing ActivityDispatchTask is invalidated — preventing the activity from being dispatched to a worker while paused. For STARTED activities the stamp is left unchanged; the worker retains a valid token and receives ActivityPaused: true on its next heartbeat.
• handleUnpauseRequested: Clears PauseState, optionally resets the attempt count and/or heartbeat details, and if the activity is SCHEDULED bumps the stamp and enqueues a new ActivityDispatchTask with optional jitter.
• RecordHeartbeat: Wires up the ActivityPaused response field.
• buildActivityExecutionInfo: Maps pause state to PENDING_ACTIVITY_STATE_PAUSED (activity is scheduled but not running) or PENDING_ACTIVITY_STATE_PAUSE_REQUESTED (activity is running on the worker) in the RunState field of DescribeActivityExecution.

Why?

PauseActivityExecution / UnpauseActivityExecution were already implemented for workflow-embedded activities via the history service. Standalone activities had stub handlers that returned Unimplemented, this brings SAA to feature parity with workflow activities for the pause/unpause lifecycle operations.

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

Potential risks

Minimal, this is a translation of an existing api (Pause/UnpauseActivity)

[fretz12]

Did a quick scan with AI, something to double check or ignore if not applicable.

Missing test cases

1. Terminate while PAUSED — Design doc says PAUSED + terminate → TERMINATED. No test covers this.

2. Schedule-to-close timeout while PAUSED — Design doc says PAUSED + S2C timeout → TIMED_OUT. No test for this. This is an important edge case since the S2C task must still fire on a PAUSED activity.

3. Unpause with ResetHeartbeat — UnpauseWithResetAttempts is tested but ResetHeartbeat is not. These are independent flags in the design doc.

4. Unpause of CANCEL_REQUESTED + PauseState — This is the bug we identified earlier. There's no test for unpausing a CANCEL_REQUESTED activity. Per the design doc it should be a no-op, but the code falls through. Adding this test would catch the bug.

5. Worker completes successfully while STARTED+paused — Design doc says STARTED + paused + worker completes → COMPLETED. No test verifies this — the PauseWhileRunning test only covers the fail+retry path.

6. Worker fails with non-retryable failure while paused — Design doc says STARTED + paused + non-retryable fail → FAILED. No test.

7. Describe run state mapping for CANCEL_REQUESTED + PauseState — Design doc says CANCEL_REQUESTED + PauseState should report CANCEL_REQUESTED (cancel takes precedence). The PauseWhileCancelRequested test checks heartbeat flags but doesn't verify the RunState in Describe.

8. Pause/Unpause request validation — No tests for invalid inputs to pause/unpause (e.g., empty activity_id, overly long identity/reason). This aligns with the missing validation TODOs in the frontend handler code.

Issues in existing tests

9. PauseWhileWaiting has a race — The test fails attempt 1, then polls DescribeActivityExecution until attempt == 2 to confirm rescheduling, then pauses. But between the Describe check and the Pause call, the retry dispatch task could fire and a worker could pick it up (the task queue has no poller, but the dispatch task still enters the queue). The test relies on the 1s retry interval being slow enough. This is
   fragile — consider using a longer retry interval (like PauseWhileRetryNoWait does with 30s).

10. PauseWhileCancelRequested doesn't verify RunState — It checks heartbeat flags but not DescribeActivityExecution RunState. Should add:
    require.Equal(t, enumspb.PENDING_ACTIVITY_STATE_CANCEL_REQUESTED, descResp.GetInfo().GetRunState())

[fretz12]

I think these do the exact same thing and can be combined into an || conditional

[fretz12]

nit: unpause method already cause a.PauseState = nil. Maybe just move this inside the clause that doesn't call unpause

[fretz12]

Can you add a test that pauses the activity, updates options, then unpauses?

[fretz12]

this is called when in STARTED state, but we shouldn't do it otherwise it'll invalidate timeouts. Probably move the stamp bump outside of this function

[spkane31]

Moved into the TransitionPaused apply func, which is not called for the flag-only pause (when in started)

[fretz12]

I believe we want attempt.Stamp++