Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions docs/adr/0003-ios-ax-snapshot-failure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# ADR 0003: iOS AX Snapshot Failure Handling

## Status

Accepted

## Context

iOS XCTest can fail hierarchy capture with `kAXErrorIllegalArgument` when an accessibility tree is
too deep to serialize. Appium's XCUITest guidance documents the practical depth limit: callers may
raise `snapshotMaxDepth` only up to `62`, and elements at depth `63` or greater cannot be returned by
XCTest. React Native screens are a common source of this shape.

Before this ADR, Agent Device let that XCTest failure escape as a slow runner command. The daemon
could wait for the command deadline, invalidate or kill the runner session as a transport failure,
and then later commands reported `SESSION_NOT_FOUND`. The app tree may still need flattening, but a
snapshot limitation should not break screenshot, logs, app lifecycle, or direct selector commands in
the same runner session.

Maestro handles this class of failure in its iOS view hierarchy route by using a depth cap of `60`,
detecting `kAXErrorIllegalArgument`, and retrying from a child/window subtree when the app root
cannot be serialized.

## Decision

Agent Device treats iOS AX snapshot serialization failure as a typed snapshot failure, not as a
runner transport failure.

The runner snapshot path now:

- caps traversal depth at `60`, with lower user-provided `--depth` values still honored
- catches Swift errors and Objective-C exceptions from `XCUIElement.snapshot()`
- classifies `kAXErrorIllegalArgument` as `IOS_AX_SNAPSHOT_FAILED`
- retries app-root failures from `windows.firstMatch`, first child, and first `.other` subtree
- returns a partial snapshot with a warning when fallback succeeds
- returns `IOS_AX_SNAPSHOT_FAILED` with an app-side flattening hint when fallback fails

Daemon and CLI output preserve runner warnings and runner error hints. Because the error code is not
`COMMAND_FAILED`, runner-session retry and invalidation policy does not treat this typed failure as a
dead transport.

Direct iOS selector interaction remains the first path for simple selector clicks, and `find id
<value> click` now probes the runner `querySelector` path before taking a full snapshot. If the
direct probe misses or has a transport fallback condition, the normal snapshot-based find path still
executes.

## Alternatives Considered

- Flatten every problematic app screen: still useful when the screen must be fully inspectable, but
it moves a tooling failure mode into each app codebase and does not protect other sessions.
- Copy WebDriverAgent/Appium source generation: too broad for Agent Device. The immediate need is
typed fast failure, partial recovery, and session preservation.
- Copy Maestro's hierarchy implementation wholesale: Maestro builds a different AX model and has
its own swizzled max-depth path. Agent Device keeps its existing snapshot model and adopts only the
small recovery behavior that fits the runner protocol.
- Always return an empty snapshot on AX failure: simple, but ambiguous. Users need to know this is an
iOS AX serialization limit and that app-side flattening may be required.

## Consequences

Partial fallback snapshots are explicitly marked `truncated` and include a warning. Selectors may be
less accurate against partial trees, so callers should treat screenshot as visual truth and flatten
the app-side accessibility tree when full inspectability is required.

`IOS_AX_SNAPSHOT_FAILED` should remain a snapshot-domain error. Do not add it to generic retryable
runner transport errors, and do not invalidate the runner session for it.

Future improvements can add a dedicated regression fixture for a minimal React Native tree that
reproduces the XCTest depth failure. Until then, TypeScript tests guard warning propagation, typed
error preservation, and direct `find id ... click` routing.
Original file line number Diff line number Diff line change
Expand Up @@ -646,12 +646,25 @@ extension RunnerTests {
scope: command.scope,
raw: command.raw ?? false
)
if options.raw {
do {
let payload: DataPayload
if options.raw {
payload = try snapshotRaw(app: activeApp, options: options)
} else {
payload = try snapshotFast(app: activeApp, options: options)
}
needsPostSnapshotInteractionDelay = true
return Response(ok: true, data: snapshotRaw(app: activeApp, options: options))
return Response(ok: true, data: payload)
} catch let failure as SnapshotCaptureFailure {
return Response(
ok: false,
error: ErrorPayload(
code: failure.code,
message: failure.message,
hint: failure.hint
)
)
}
needsPostSnapshotInteractionDelay = true
return Response(ok: true, data: snapshotFast(app: activeApp, options: options))
case .screenshot:
let screenshot: XCUIScreenshot
#if os(macOS)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ struct DataPayload: Codable {
let items: [String]?
let nodes: [SnapshotNode]?
let truncated: Bool?
let warnings: [String]?
let gestureStartUptimeMs: Double?
let gestureEndUptimeMs: Double?
let x: Double?
Expand All @@ -115,6 +116,7 @@ struct DataPayload: Codable {
items: [String]? = nil,
nodes: [SnapshotNode]? = nil,
truncated: Bool? = nil,
warnings: [String]? = nil,
gestureStartUptimeMs: Double? = nil,
gestureEndUptimeMs: Double? = nil,
x: Double? = nil,
Expand All @@ -135,6 +137,7 @@ struct DataPayload: Codable {
self.items = items
self.nodes = nodes
self.truncated = truncated
self.warnings = warnings
self.gestureStartUptimeMs = gestureStartUptimeMs
self.gestureEndUptimeMs = gestureEndUptimeMs
self.x = x
Expand All @@ -154,10 +157,12 @@ struct DataPayload: Codable {
struct ErrorPayload: Codable {
let code: String?
let message: String
let hint: String?

init(code: String? = nil, message: String) {
init(code: String? = nil, message: String, hint: String? = nil) {
self.code = code
self.message = message
self.hint = hint
}
}

Expand Down
Loading