feat(compute): add EC2 fleet compute strategy

[MichaelWalker-git]

Summary

• Add EC2 fleet compute strategy with SSM Run Command dispatch — a third compute backend alongside AgentCore (default) and ECS Fargate
• New Ec2ComputeStrategy handler: finds idle instances via tags, uploads payload to S3, dispatches via SSM AWS-RunShellScript, polls GetCommandInvocation, cancels with CancelCommand
• New Ec2AgentFleet CDK construct: Auto Scaling Group with launch template (AL2023 ARM64), security group (443 egress only), S3 payload bucket, IAM role with scoped permissions, Docker user data for pre-pulling images
• Wire orchestrator polling, cancel-task SSM dispatch, and task-api SSM permissions for EC2
• Stack wiring is commented-out (same pattern as ECS) — ready to enable per-repo via blueprint compute_type: 'ec2'
• Add instance_type field to RepoConfig and BlueprintConfig for future GPU/custom instance type support

Test plan

• mise //cdk:compile — no TypeScript errors
• mise //cdk:test — 43 suites, 697 tests all passing (including new ec2-strategy and ec2-agent-fleet tests)
• mise //cdk:synth — synthesizes without errors (EC2 block commented out)
• mise //cdk:build — full build including lint passes
• Deploy with EC2 block uncommented and run an end-to-end task with compute_type: 'ec2'

[MichaelWalker-git]

Recreating from a clean branch off main to avoid conflicts from prior commits

[Copilot]

Pull request overview

This PR adds a third compute backend (“EC2 fleet”) alongside the existing AgentCore and ECS options, wiring strategy selection into the orchestrator and extending CDK constructs/tests/docs to support the new backend.

Changes:

• Introduces compute-strategy abstraction with implementations for AgentCore, ECS Fargate, and EC2 fleet (SSM Run Command + S3 payload).
• Updates orchestrator start-session and polling to use the selected compute strategy and persist compute metadata for cancellation.
• Adds CDK constructs/tests for ECS agent cluster and EC2 agent fleet, plus small docs/CI updates.

Reviewed changes

Copilot reviewed 31 out of 33 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
yarn.lock	Adds AWS SDK clients (EC2/ECS/S3/SSM) and transitive deps.
docs/src/content/docs/design/Architecture.md	Adds rationale section on separating orchestrator vs agent loops.
docs/design/ARCHITECTURE.md	Same rationale section mirrored into top-level design doc.
cdk/test/handlers/start-session-composition.test.ts	Integration-style orchestration step composition tests.
cdk/test/handlers/shared/strategies/agentcore-strategy.test.ts	Unit tests for AgentCore compute strategy.
cdk/test/handlers/shared/strategies/ecs-strategy.test.ts	Unit tests for ECS compute strategy.
cdk/test/handlers/shared/strategies/ec2-strategy.test.ts	Unit tests for EC2 compute strategy.
cdk/test/handlers/shared/preflight.test.ts	Normalizes compute_type casing in tests.
cdk/test/handlers/shared/compute-strategy.test.ts	Tests strategy resolution for agentcore/ecs/ec2.
cdk/test/handlers/orchestrate-task.test.ts	Removes older startSession tests now handled by strategies/composition tests.
cdk/test/handlers/cancel-task.test.ts	Adds ECS cancellation coverage and behavior tests.
cdk/test/constructs/task-orchestrator.test.ts	Adds ECS env var + IAM wiring tests.
cdk/test/constructs/task-api.test.ts	Adds cancel-task ECS env var + IAM wiring tests.
cdk/test/constructs/ecs-agent-cluster.test.ts	New tests for ECS cluster construct.
cdk/test/constructs/ec2-agent-fleet.test.ts	New tests for EC2 fleet construct.
cdk/src/stacks/agent.ts	Adds commented wiring blocks for ECS/EC2 backends.
cdk/src/handlers/shared/types.ts	Persists compute_type + compute_metadata on task records.
cdk/src/handlers/shared/strategies/agentcore-strategy.ts	New AgentCore compute strategy implementation.
cdk/src/handlers/shared/strategies/ecs-strategy.ts	New ECS compute strategy implementation.
cdk/src/handlers/shared/strategies/ec2-strategy.ts	New EC2 compute strategy implementation.
cdk/src/handlers/shared/repo-config.ts	Adds ComputeType union + instance_type config field.
cdk/src/handlers/shared/orchestrator.ts	Adds PollState fields + instance_type wiring; removes old startSession helper.
cdk/src/handlers/shared/compute-strategy.ts	New strategy interface + resolver.
cdk/src/handlers/orchestrate-task.ts	Uses compute strategies for start + compute-level polling.
cdk/src/handlers/cancel-task.ts	Adds ECS StopTask + EC2 SSM CancelCommand cancellation paths.
cdk/src/constructs/task-orchestrator.ts	Adds optional ECS/EC2 config env vars + IAM grants.
cdk/src/constructs/task-api.ts	Adds optional ECS/EC2 cancellation wiring + IAM grants.
cdk/src/constructs/ecs-agent-cluster.ts	New ECS cluster construct (Fargate task def + SG + IAM).
cdk/src/constructs/ec2-agent-fleet.ts	New EC2 ASG-based fleet construct (SSM-managed instances).
cdk/src/constructs/blueprint.ts	Extends blueprint compute type to include ec2.
cdk/package.json	Adds AWS SDK clients needed for new strategies.
.gitignore	Ignores local-docs directory.
.github/workflows/build.yml	Sets GitHub token env vars for CI tools; adjusts MISE_DISABLE_TOOLS.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The instance is tagged bgagent:status=busy before the SSM command is dispatched, but if SendCommand throws or returns no CommandId, the instance will remain stuck in busy (and with bgagent:task-id set). Wrap the dispatch in a try/catch/finally that reverts tags on failure, or tag busy only after a successful SendCommand response.

Suggested change

	// 3. Tag instance as busy
	await getEc2Client().send(new CreateTagsCommand({
	Resources: [instanceId],
	Tags: [
	{ Key: 'bgagent:status', Value: 'busy' },
	{ Key: 'bgagent:task-id', Value: taskId },
	],
	}));
	// 4. Build the boot command (mirrors ECS strategy env vars and Python boot command)
	// 3. Build the boot command (mirrors ECS strategy env vars and Python boot command)

[Copilot]

This strategy serializes the full orchestrator payload (including hydrated_context) into an ECS environment variable (AGENT_PAYLOAD). ECS task overrides have fairly small limits on environment size, so larger contexts can cause RunTask to fail at runtime. Consider switching to an S3-backed payload (pass an S3 URI/key in env) or another mechanism that doesn't depend on env-var size limits.

[Copilot]

For compute_type: 'ec2', cancel currently only calls SSM CancelCommand. If the command is cancelled mid-script, the cleanup/tag-reset section in the boot script may never run, leaving the instance stuck in bgagent:status=busy and effectively reducing fleet capacity. Consider also re-tagging the instance back to idle here (and deleting bgagent:task-id), or invoking the EC2 strategy’s stopSession logic from the cancel handler.

[Copilot]

The compute-level polling block is gated on blueprintConfig.compute_type === 'ecs' || 'ec2', but the variables/log messages/errors are all ECS-specific (e.g., consecutiveEcs*, "ECS container failed"). This will produce misleading failures for EC2 and makes the logic harder to extend. Consider renaming this to backend-neutral terminology and branching error messages based on sessionHandle.strategyType (or splitting ECS vs EC2 polling).

Suggested change

	let consecutiveEcsPollFailures = 0;
	let consecutiveEcsCompletedPolls = 0;
	// ECS compute-level crash detection: if DDB is not terminal, check ECS task status
	if (
	ddbState.lastStatus &&
	!TERMINAL_STATUSES.includes(ddbState.lastStatus) &&
	computeStrategy
	) {
	try {
	const ecsStatus = await computeStrategy.pollSession(sessionHandle);
	if (ecsStatus.status === 'failed') {
	const errorMsg = 'error' in ecsStatus ? ecsStatus.error : 'ECS task failed';
	logger.warn('ECS task failed before DDB terminal write', {
	task_id: taskId,
	error: errorMsg,
	});
	await failTask(taskId, ddbState.lastStatus, `ECS container failed: ${errorMsg}`, task.user_id, true);
	return { attempts: ddbState.attempts, lastStatus: TaskStatus.FAILED };
	}
	if (ecsStatus.status === 'completed') {
	consecutiveEcsCompletedPolls = (state.consecutiveEcsCompletedPolls ?? 0) + 1;
	if (consecutiveEcsCompletedPolls >= MAX_CONSECUTIVE_ECS_COMPLETED_POLLS) {
	// ECS task exited successfully but DDB never reached terminal — the agent
	// likely crashed after container exit code 0 but before writing status.
	logger.error('ECS task completed but DDB never caught up — failing task', {
	task_id: taskId,
	consecutive_completed_polls: consecutiveEcsCompletedPolls,
	});
	await failTask(taskId, ddbState.lastStatus, `ECS task exited successfully but agent never wrote terminal status after ${consecutiveEcsCompletedPolls} polls`, task.user_id, true);
	return { attempts: ddbState.attempts, lastStatus: TaskStatus.FAILED };
	}
	logger.warn('ECS task completed but DDB not terminal — waiting for DDB catchup', {
	task_id: taskId,
	consecutive_completed_polls: consecutiveEcsCompletedPolls,
	});
	}
	} catch (err) {
	consecutiveEcsPollFailures = (state.consecutiveEcsPollFailures ?? 0) + 1;
	if (consecutiveEcsPollFailures >= MAX_CONSECUTIVE_ECS_POLL_FAILURES) {
	logger.error('ECS pollSession failed repeatedly — failing task', {
	task_id: taskId,
	consecutive_failures: consecutiveEcsPollFailures,
	error: err instanceof Error ? err.message : String(err),
	});
	await failTask(taskId, ddbState.lastStatus, `ECS poll failed ${consecutiveEcsPollFailures} consecutive times: ${err instanceof Error ? err.message : String(err)}`, task.user_id, true);
	return { attempts: ddbState.attempts, lastStatus: TaskStatus.FAILED };
	}
	logger.warn('ECS pollSession check failed (non-fatal)', {
	task_id: taskId,
	consecutive_failures: consecutiveEcsPollFailures,
	error: err instanceof Error ? err.message : String(err),
	});
	}
	}
	return { ...ddbState, consecutiveEcsPollFailures, consecutiveEcsCompletedPolls };
	let consecutiveComputePollFailures = 0;
	let consecutiveComputeCompletedPolls = 0;
	const computeBackendLabel = sessionHandle.strategyType === 'ec2' ? 'EC2' : 'ECS';
	// Compute-level crash detection: if DDB is not terminal, check compute session status.
	if (
	ddbState.lastStatus &&
	!TERMINAL_STATUSES.includes(ddbState.lastStatus) &&
	computeStrategy
	) {
	try {
	const computeStatus = await computeStrategy.pollSession(sessionHandle);
	if (computeStatus.status === 'failed') {
	const errorMsg =
	'error' in computeStatus ? computeStatus.error : `${computeBackendLabel} task failed`;
	logger.warn(`${computeBackendLabel} task failed before DDB terminal write`, {
	task_id: taskId,
	error: errorMsg,
	});
	await failTask(
	taskId,
	ddbState.lastStatus,
	`${computeBackendLabel} compute failed: ${errorMsg}`,
	task.user_id,
	true,
	);
	return { attempts: ddbState.attempts, lastStatus: TaskStatus.FAILED };
	}
	if (computeStatus.status === 'completed') {
	consecutiveComputeCompletedPolls = (state.consecutiveEcsCompletedPolls ?? 0) + 1;
	if (consecutiveComputeCompletedPolls >= MAX_CONSECUTIVE_ECS_COMPLETED_POLLS) {
	// Compute session exited successfully but DDB never reached terminal —
	// the agent likely crashed after compute completion but before writing status.
	logger.error(`${computeBackendLabel} task completed but DDB never caught up — failing task`, {
	task_id: taskId,
	consecutive_completed_polls: consecutiveComputeCompletedPolls,
	});
	await failTask(
	taskId,
	ddbState.lastStatus,
	`${computeBackendLabel} task exited successfully but agent never wrote terminal status after ${consecutiveComputeCompletedPolls} polls`,
	task.user_id,
	true,
	);
	return { attempts: ddbState.attempts, lastStatus: TaskStatus.FAILED };
	}
	logger.warn(`${computeBackendLabel} task completed but DDB not terminal — waiting for DDB catchup`, {
	task_id: taskId,
	consecutive_completed_polls: consecutiveComputeCompletedPolls,
	});
	}
	} catch (err) {
	consecutiveComputePollFailures = (state.consecutiveEcsPollFailures ?? 0) + 1;
	if (consecutiveComputePollFailures >= MAX_CONSECUTIVE_ECS_POLL_FAILURES) {
	logger.error(`${computeBackendLabel} pollSession failed repeatedly — failing task`, {
	task_id: taskId,
	consecutive_failures: consecutiveComputePollFailures,
	error: err instanceof Error ? err.message : String(err),
	});
	await failTask(
	taskId,
	ddbState.lastStatus,
	`${computeBackendLabel} poll failed ${consecutiveComputePollFailures} consecutive times: ${err instanceof Error ? err.message : String(err)}`,
	task.user_id,
	true,
	);
	return { attempts: ddbState.attempts, lastStatus: TaskStatus.FAILED };
	}
	logger.warn(`${computeBackendLabel} pollSession check failed (non-fatal)`, {
	task_id: taskId,
	consecutive_failures: consecutiveComputePollFailures,
	error: err instanceof Error ? err.message : String(err),
	});
	}
	}
	return {
	...ddbState,
	consecutiveEcsPollFailures: consecutiveComputePollFailures,
	consecutiveEcsCompletedPolls: consecutiveComputeCompletedPolls,
	};

[Copilot]

The EC2 orchestrator IAM statements are very broad (ec2:CreateTags on *, ssm:* on *) and the iam:PassRole grant conditioned on ec2.amazonaws.com doesn’t appear to be used by the EC2 strategy (it doesn’t call APIs that pass a role). Consider removing the EC2 iam:PassRole statement, and scoping EC2 tagging permissions with conditions (e.g., ec2:ResourceTag/${fleetTagKey} == fleetTagValue and limiting aws:TagKeys) to reduce blast radius.

[Copilot]

ec2FleetConfig includes instanceRoleArn, but that value isn’t used (the construct only conditionally grants ssm:CancelCommand on *). This makes the API misleading and suggests a scoping that doesn’t exist. Either remove instanceRoleArn from the props or use it to implement a more scoped permission model (e.g., document/service role, or additional permissions needed to reset instance tags on cancel).

Suggested change

	*/
	readonly ec2FleetConfig?: {
	readonly instanceRoleArn: string;
	};
	* This object currently serves only as an enablement flag for EC2-backed cancellation.
	*/
	readonly ec2FleetConfig?: {};

[Copilot]

this.autoScalingGroup.node.defaultChild; is a no-op statement and can be removed. If the intent was to access/modify the underlying CFN resource, assign it to a variable and use it explicitly (otherwise this line adds noise without effect).

Suggested change

this.autoScalingGroup.node.defaultChild;

[Copilot]

The EC2 boot script uses aws ecr get-login-password --region $AWS_REGION, but AWS_REGION is never set in the script. On stock AL2023 instances this is typically unset, which will make ECR login/pull fail. Consider deriving the region from instance metadata (like the user-data does) and exporting AWS_REGION/AWS_DEFAULT_REGION before the AWS CLI and docker run steps.

Suggested change

	'',
	'',
	'# Resolve AWS region from instance metadata',
	'IMDS_TOKEN=$(curl -sS -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")',
	'AWS_REGION=$(curl -sS -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document | sed -n \'s/.*"region"[[:space:]]*:[[:space:]]*"\\([^"]*\\)".*/\\1/p\')',
	'export AWS_REGION',
	'export AWS_DEFAULT_REGION="$AWS_REGION"',
	'',

[Copilot]

export AGENT_PAYLOAD=$(cat /tmp/payload.json) is unsafe: command substitution strips newlines and can mangle JSON, and large hydrated payloads may exceed shell/env-var limits. Prefer keeping the payload as a file (e.g., mount /tmp/payload.json into the container and have the Python entrypoint read it) or at least quote the assignment to preserve content.