Investigate ApplicationRunsWithDebuggerAndBreaks flakiness: Fast Deployment + guest user (XA0137) + test-code bugs (follow-up to #11320)

[simonrozsival]

Investigate ApplicationRunsWithDebuggerAndBreaks flakiness (follow-up to #11320, #11203)

Deep dive into the recurring ApplicationRunsWithDebuggerAndBreaks failures called out in
#11203 and #11320. After pulling the actual AZDO build logs for last 14 days of 🔴 hits, the
flakiness is not primarily debugger-attach timing — it's a mix of one known product bug,
two test-code bugs, and shard infra noise.

This is a follow-up to #11320; the broader analysis lives there.

Where the failures cluster

In the last 14d, every 🔴 hit was on an embedAssemblies=False (Fast Deployment)
parametrization. (True, …, …) variants produced zero 🔴 events.

(embed, user, fmt)	🔴	🟢	⚠️	FastDev	Guest user
(False, "guest1", "aab")	1	3	2	yes	yes
(False, "guest1", "apk")	2	1	0	yes	yes
(False, null, "aab")	1	0	0	yes	no

Failure modes (from actual AZDO build logs)

1. XA0137 / Fast Deployment vs. guest user — primary root cause

Build 14020311 for
#11277, test (False,"guest1","aab",True,MonoVM):

error XA0137: The 'run-as' command failed with 'run-as: couldn't stat
    /data/user/11/com.xamarin.applicationrunswithdebuggerandbreaks: No such file or directory'
... Fast Deployment is not currently supported on this device.

This is exactly the symptom in open issue #7821. Fast Dev's run-as query against the
guest user's data dir races against the install — when pm install ran under user 11
("guest1"), the app's /data/user/11/<pkg> directory doesn't yet exist (or hasn't been
re-stat-able through run-as). The test does SwitchUser + WaitFor(5000) + install
(DebuggingTest.cs:474-491),
but the 5 s wait isn't always enough.

2. Shard-wide infra failures (secondary, not test-specific)

Build 14048827
auto-retry run 162017497: 17 unrelated tests all failed together on one macOS shard,
including (True,"guest1","apk"). Emulator/agent died — not a bug in this test.

3. Build duration eats the debugger-attach budget on slow shards

Same failed run: Build,Install step alone took 3 m 01 s, plus two uiautomator dump
timeouts. The test's debugger-attach budget is fixed at 60 s + 30 s connect — on a slow
shard, the install alone overruns it.

Bugs in the test code itself (independent of environment)

4. Shared timeout variable across two wait loops

DebuggingTest.cs:635-650:

TimeSpan timeout = TimeSpan.FromSeconds (60);
int expected = 4;
while (session.IsConnected && breakcountHitCount < 3 && timeout >= TimeSpan.Zero) {
    Thread.Sleep (10);
    timeout = timeout.Subtract (TimeSpan.FromMilliseconds (10));
}
WaitFor (2000);
Assert.AreEqual (expected, breakcountHitCount, $"Should have hit {expected} breakpoints. Only hit {breakcountHitCount}");
breakcountHitCount = 0;
ClearAdbLogcat ();
ClearBlockingDialogs ();
Assert.True (ClickButton (app.PackageName, "myXFButton", "CLICK ME"), "Button should have been clicked!");
while (session.IsConnected && breakcountHitCount < 1 && timeout >= TimeSpan.Zero) {
    Thread.Sleep (10);
    timeout = timeout.Subtract (TimeSpan.FromMilliseconds (10));
}

The second loop reuses whatever's left of timeout after the first. If the initial
wait-for-3-breakpoints took 55 s, only 5 s remains for the post-click breakpoint.
Real bug — should reset timeout = TimeSpan.FromSeconds (60) before the second loop.

5. Synchronous, untimed HttpClient.GetStringAsync in MainActivity static ctor

DebuggingTest.cs:528-542:

static MainActivity()
{
    try {
        var text = new HttpClient().GetStringAsync("https://www.google.com").GetAwaiter().GetResult();
        Console.WriteLine("Web request:" + text);
    } catch (Exception ex) {
        Console.WriteLine("Web request failed:" + ex);
    }
}

No timeout. Blocks the type initializer until DNS+TLS+HTTP resolves. The accompanying comment
says "Doesn't actually matter if succeeds" — but the test still waits synchronously. On a
slow agent network, this can delay reaching OnCreate past WaitForDebuggerToStart's
budget. Should at minimum apply a short HttpClient.Timeout, or move the call off the type
initializer.

6. Random debugger port may collide

DebuggingTest.cs:597:
rnd.Next(10000, 20000). Low collision probability but not zero on shards running many test
processes. Better to bind port 0 (let the OS pick) or retry on bind failure.

7. catch (Exception ex) { Assert.Fail(...); } swallows environment failures

DebuggingTest.cs:655-656.
Build failures, adb timeouts, emulator deaths all surface as test failure rather than
Assert.Inconclusive — same anti-pattern that #11249 fixed for the perf tests and #11110
fixed for emulator-acquisition. The XA0137 Fast Dev failure (#7821) is a perfect candidate
for Inconclusive.

Why [Retry(5)] + AZDO auto-retry don't help

The dominant failure (Fast Dev + guest user run-as) is systematic, not transient. The
guest user's home-dir state is the same on each retry within seconds. Retrying without
backoff or recreating the guest user changes nothing — the test keeps consuming the retry
budget on the same bug.

Recommendations

#	Action	Why	Effort
1	Fix the shared-timeout bug (#4)	Pure correctness bug, eliminates one flake class	trivial
2	On XA0137 run-as build failure during Install: catch FailedBuildException, grep the log, call Assert.Inconclusive and link to #7821	The combo is a known product bug; test shouldn't gate PRs on it	small
3	Add a short HttpClient.Timeout (e.g. 5 s) to the injected static ctor, or move the call off the type initializer	Unblocks startup on slow networks	small
4	Assert.Inconclusive on the obvious environment failures in the outer catch (adb timeout, emulator gone, shard-wide failure) — follows the #11249/#11110 pattern	Stops shard infra failures from showing as 🔴 on this test	small
5	Bind a free port (new TcpListener(IPAddress.Loopback, 0)) instead of rnd.Next	Eliminates one rare flake source	small
6	Consider splitting (embedAssemblies=False, username="guest1") parametrizations into a separate [Category("KnownIssue")] test until #7821 is fixed	Removes 4/9 parameterizations from gating CI	small

Out of scope here

• Real fix for When using FastDeployment get error message XA0136: The 'run-as' command failed with 'run-as: Package 'packagename' has corrupt installation #7821 (Fast Deployment vs. guest user) — that's a product change in
  Xamarin.Android.Common.Debugging.targets / FastDeploy task, not a test fix.
• CoreCLR debugging — the test is MonoVM-only by design today
  (DebuggingTest.cs:370-373).

Related

• Flaky test analysis: Xamarin.Android-PR pipeline, April 27 – May 11 2026 (follow-up to #11203) #11320 — broader flaky-test analysis (this issue's parent)
• Flaky test analysis: Xamarin.Android-PR pipeline, April 10–24 2026 #11203 — prior broader flaky-test analysis
• When using FastDeployment get error message XA0136: The 'run-as' command failed with 'run-as: Package 'packagename' has corrupt installation #7821 — 'run-as' command failed (the underlying product bug)
• [tests] Skip PerformanceTest on slow CI machines using evaluation time #11249 — example pattern of converting environment-sensitive failures to Inconclusive
• Use Assert.Inconclusive for emulator acquisition failures #11110 — Assert.Inconclusive for emulator-acquisition failures
• App stops/closes while on splash screen when debugging #9511 — debugger / splash-screen interaction (likely unrelated but in the area)