Problem
packages/server/test/vendor/vendor.test.js jspmGenerate: install order does not affect output intermittently FAILS by hitting the LIVE ga.jspm.io / api.jspm.io CDN and getting order-dependent results (observed: a clsx transitive resolved in one install ordering but not the other). It is gated behind a NETWORK_OK env, so it runs in CI and on a networked dev machine. Across ~18 PRs in one session it flaked roughly 1 run in 4, each time passing on a plain re-run. A flaky test that depends on a third-party CDN's response stability is a CI-reliability gap: a green PR can fail CI for reasons unrelated to the change, and the failure looks like a regression.
Design / approach
Make the test deterministic rather than order-dependent against a live CDN. Options: (a) record/replay the jspm responses for this case (a fixture of the two orderings' expected importmaps) so no live fetch happens; (b) relax the assertion to compare only the order-INDEPENDENT subset (the set of resolved top-level specifiers), tolerating CDN-side transitive-resolution differences; or (c) gate this specific assertion behind a stricter opt-in (a dedicated env) so the default networked run does not include the order-sensitivity check. Prefer (a) or (b): the property under test (install order should not change OUR output) should be asserted against OUR resolution logic, not against the CDN's live transitive graph.
Acceptance criteria
Observed during the production-readiness work (recurred across ~18 PRs in one session, always passing on re-run). Theme: testing / CI reliability. Priority: P2.
Problem
packages/server/test/vendor/vendor.test.jsjspmGenerate: install order does not affect outputintermittently FAILS by hitting the LIVEga.jspm.io/api.jspm.ioCDN and getting order-dependent results (observed: aclsxtransitive resolved in one install ordering but not the other). It is gated behind aNETWORK_OKenv, so it runs in CI and on a networked dev machine. Across ~18 PRs in one session it flaked roughly 1 run in 4, each time passing on a plain re-run. A flaky test that depends on a third-party CDN's response stability is a CI-reliability gap: a green PR can fail CI for reasons unrelated to the change, and the failure looks like a regression.Design / approach
Make the test deterministic rather than order-dependent against a live CDN. Options: (a) record/replay the jspm responses for this case (a fixture of the two orderings' expected importmaps) so no live fetch happens; (b) relax the assertion to compare only the order-INDEPENDENT subset (the set of resolved top-level specifiers), tolerating CDN-side transitive-resolution differences; or (c) gate this specific assertion behind a stricter opt-in (a dedicated env) so the default networked run does not include the order-sensitivity check. Prefer (a) or (b): the property under test (install order should not change OUR output) should be asserted against OUR resolution logic, not against the CDN's live transitive graph.
Acceptance criteria
install order does not affect outputtest no longer flakes against the live jspm CDN (deterministic, or relaxed to the order-independent property)Observed during the production-readiness work (recurred across ~18 PRs in one session, always passing on re-run). Theme: testing / CI reliability. Priority: P2.