Add Microsoft.Testing.Extensions.JUnitReport extension (#4268)#8850
Conversation
Adds a new MTP extension that emits JUnit-format XML reports compatible with Jenkins, GitLab, Azure DevOps and other CI parsers. CLI options: --report-junit --report-junit-filename <name> Design highlights (see docs/RFCs/016-JUnit-Report.md): - Flat <testsuites>/<testsuite> for broad parser compatibility. - MTP tree-of-tests preserved per testcase via <property name="testpath"/> rather than nested <testsuite>, which Jenkins/Surefire handle inconsistently. - Strict <testcase> child ordering (properties -> skipped/error/failure -> system-out/-err) required by Surefire/Jenkins. - Atomic write via temp file + rename. - Defensive caps (testpath length, parent-chain depth with cycle detection) and stream/message/identity/trait memory limits. - Outcome mapping: passed -> empty, skipped -> <skipped>, failed -> <failure>, errored/timedOut -> <error>. Discovered/InProgress nodes are not emitted as testcases but participate in the parent chain. Implementation: - 17 files under src/Platform/Microsoft.Testing.Extensions.JUnitReport (net8.0, net9.0, netstandard2.0) modelled on the existing HtmlReport extension. - MSBuild auto-registration GUID 9F3CD35A-4E62-43EF-B7F1-426D30E850ED wired into the platform csproj and TestFx.slnx. - Localized resources mirrored from HtmlReport; XLF regenerated via UpdateXlf. Tests (all green): - 5 new acceptance tests x 3 TFMs (JUnitReportTests). - HelpInfoAllExtensionsTests and MSBuild.KnownExtensionRegistration updated with the new option/registration assertions. Closes microsoft#4268 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a new Microsoft.Testing.Platform (MTP) extension, Microsoft.Testing.Extensions.JUnitReport, to generate JUnit-style XML reports (CLI + MSBuild known-extension registration), along with acceptance-test and help/info updates.
Changes:
- Introduces the new
Microsoft.Testing.Extensions.JUnitReportextension (capture + engine + CLI options + MSBuild builder hook + packaging assets). - Updates acceptance/integration tests to cover registration and basic report generation + updates help/info expected output for “all extensions”.
- Adds an RFC documenting schema/design choices and wires the new project into the solution/platform internals visibility.
Show a summary per file
| File | Description |
|---|---|
| TestFx.slnx | Adds the new JUnitReport project to the solution. |
| src/Platform/Microsoft.Testing.Platform/Microsoft.Testing.Platform.csproj | Grants InternalsVisibleTo for the new extension assembly. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/MSBuild.KnownExtensionRegistration.cs | Extends known-extension registration assertions to include JUnitReport. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/JUnitReportTests.cs | Adds acceptance tests for JUnit report generation and filename validation. |
| test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HelpInfoAllExtensionsTests.cs | Updates --help/--info expected output to include JUnitReport options/provider. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Microsoft.Testing.Extensions.JUnitReport.csproj | New extension project definition (TFMs, packaging layout, linked helpers). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportEngine.cs | Core JUnit XML generation logic (suites/testcases, XML writing, atomic write). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportGenerator.cs | Session handler + data consumer that captures test results and emits the artifact. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportGeneratorCommandLine.cs | Defines/validates --report-junit and --report-junit-filename. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/JUnitReportExtensions.cs | Public builder extension AddJUnitReportProvider. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/TestingPlatformBuilderHook.cs | MSBuild hook entrypoint for known-extension registration. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/TestResultCapture.cs | Projects TestNodeUpdateMessage into capped DTOs and tracks parent chain entries. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/CapturedTestResult.cs | DTO representing a captured terminal test result. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/BannedSymbols.txt | Adds banned API usage rules for the new project. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/PACKAGE.md | NuGet package readme for the new extension. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/PublicAPI/PublicAPI.Shipped.txt | Initializes shipped API baseline for the new package. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/PublicAPI/PublicAPI.Unshipped.txt | Declares new public APIs for API review/baseline. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/build/Microsoft.Testing.Extensions.JUnitReport.props | Build assets for MSBuild consumption. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/buildTransitive/Microsoft.Testing.Extensions.JUnitReport.props | Transitive build assets import. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/buildMultiTargeting/Microsoft.Testing.Extensions.JUnitReport.props | MSBuild known-extension registration item + GUID. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/ExtensionResources.resx | Adds localized strings for CLI/help/errors/artifact metadata. |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.cs.xlf | Localized resource XLF (cs). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.de.xlf | Localized resource XLF (de). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.es.xlf | Localized resource XLF (es). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.fr.xlf | Localized resource XLF (fr). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.it.xlf | Localized resource XLF (it). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.ja.xlf | Localized resource XLF (ja). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.ko.xlf | Localized resource XLF (ko). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.pl.xlf | Localized resource XLF (pl). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.pt-BR.xlf | Localized resource XLF (pt-BR). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.ru.xlf | Localized resource XLF (ru). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.tr.xlf | Localized resource XLF (tr). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.zh-Hans.xlf | Localized resource XLF (zh-Hans). |
| src/Platform/Microsoft.Testing.Extensions.JUnitReport/Resources/xlf/ExtensionResources.zh-Hant.xlf | Localized resource XLF (zh-Hant). |
| docs/RFCs/016-JUnit-Report.md | Adds the RFC describing schema/design/testing strategy. |
Copilot's findings
- Files reviewed: 35/35 changed files
- Comments generated: 8
Verifies that {tfm}, {pname} and {asm} placeholders in the JUnit report file name template are resolved end-to-end.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Youssef1313
left a comment
There was a problem hiding this comment.
Is there a specification for JUnit that this implementation followed?
IIRC, there were multiple versions of JUnit results file (possibly with breaking changes?). Have you researched that? If so, can we explicitly document what version this is?
- Rename root/per-suite `disabled` attribute to `skipped` (Jenkins/Surefire correct name) - Map cancelled outcome to <error>/errors++ (was incorrectly <failure>/failures++) - Emit <skipped> element for skipped outcome (was missing) - Fix BuildTestPath to always append the leaf DisplayName even when no parent - Rename property name=`test-uid` to `uid` for consistency - Prefix trait property keys with `trait.` - Classify CancelledTestNodeStateProperty as `cancelled` in TestResultCapture (with CS0618/MTP0001 pragma) - Add acceptance test JUnit_WhenTestsFailOrSkip_JUnitReportContainsExpectedOutcomes covering passed/failed/skipped/errored mix with parent container Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@Youssef1313 great question — short answer: there is no single canonical ""JUnit XML"" specification. The format originated as the XML output of the Ant For this extension I targeted the Jenkins / Maven Surefire flavor, which is the closest thing to a de-facto industry standard and the one consumed by the widest set of CI systems (Jenkins, Azure DevOps, GitLab, CircleCI, etc.). This is documented in RFC 016 ( Sources I consulted while designing the schema and writing the emitter:
If you'd like, I'm happy to add the explicit links above as a Normative references section in the RFC so the schema lineage is preserved alongside the design. |
- Use unique per-run temp file name (Path.GetRandomFileName) to avoid collisions between concurrent runs that share the same default report name. Replaces the fixed '<final>.tmp' path. - Add XDocument-based <testcase> child ordering assertion in the JUnit_WhenTestsFailOrSkip acceptance test to enforce the RFC 016 normative element order beyond substring checks. - Register Microsoft.Testing.Extensions.JUnitReport in Microsoft.Testing.Platform.slnf and NonWindowsTests.slnf so the package is produced for Linux/MacOS Debug Test jobs. - Fix markdown lint MD038 (spaces inside code spans) in RFC 016 by moving the leading space out of '\[attempt 2]\' / '\[attempt 3]\'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…STest acceptance - RFC 016: add JUnit flavors comparison table; expand retry-handling section covering both MSTest [Retry] and MTP --retry-failed-tests; clarify duplicate-name suffix wording. - MTP acceptance: introduce AssertWellFormedJUnitReport helper using XDocument; assert <testcase> child ordering; add DummyTestFramework mixed-outcomes test. - MSTest acceptance: add JUnitReportTests covering [Retry(N)] attribute and the MTP --retry-failed-tests extension. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
@Evangelink follow-ups addressed in bfc4b46: 1. Flavors comparison table — Added a new table in RFC 016 listing the differences between Ant, Maven Surefire, Jenkins, IntelliJ, and Gradle JUnit flavors (root element, attributes, retry handling, 2. Retry handling — Expanded the RFC's retry section to cover both modes:
3. Full-XML acceptance assertions — Refactored 4. MSTest-side acceptance tests — New file
About the spec Youssef asked about: I didn't follow a single official spec — JUnit XML has no canonical schema. The RFC documents which flavors were consulted (Ant, Maven Surefire, Jenkins, IntelliJ, Gradle) and which conventions were adopted. The "JUnit XML flavors" section in RFC 016 now spells out the differences explicitly. The closest thing to a normative reference is the Apache Ant Build is green, all acceptance tests pass locally (21 MTP + 2 MSTest). |
- Use --list-tests (the actual CLI option) instead of --discover-tests. - Document that reserved Windows filenames are sanitized (prefixed with _), not rejected, matching ReplaceInvalidFileNameChars and the HtmlReport behavior. - Spell out the full list of reserved names and the invalid character set. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e / RFC 016 Address review feedback on PR microsoft#8850. The previous marker was inconsistent with the repo's standard truncation format (TestResultCapture.Truncate) and with RFC 016, which both specify the marker as `\n…[truncated, original length: N]` where N is the real untruncated length. Two issues are fixed: 1. The marker now has the leading `\n` prefix so XML consumers and log readers can split on it cleanly. 2. The reported "original length" is now the full untruncated testpath length (segments + separators), computed up front, rather than the partially-built StringBuilder length at the moment we exceeded the cap (which omits remaining segments and is therefore strictly smaller). Also adds the surrogate-pair safety check from TestResultCapture so the cut at MaxTestPathLength never splits a UTF-16 high surrogate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Follow-ups:
|
Match the versioning scheme used by other experimental Microsoft.Testing.Platform extensions (Logging, OpenTelemetry, AI). Packages will now ship as 1.0.0-alpha.* instead of the repo-default *-preview.*. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Also switched to the experimental |
Replace XDocument-based structural checks with verbatim XML snapshots covering both the single-passing-test default and the mixed-outcomes (pass/fail/skip/error across two suites) scenario. Runtime-variable attributes (time, timestamp, hostname) are normalized via attribute-scoped regexes; everything else — attribute order, suite ids, counts, exit-code, properties block contents, self-closing failure/skipped/error elements — must match byte-for-byte. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
After the JUnitReport package was moved to the experimental versioning scheme (1.0.0-alpha) it no longer matches the platform version requested by the acceptance test asset csproj. Introduce a dedicated MicrosoftTestingExtensionsJUnitReportVersion property on AcceptanceTestBase and a \\\$\ placeholder so the dummy app references the package at the version actually produced by the local pack. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Follow-up commits on this branch:
All 21 |
- Truncate RawUid, ParentRawUid, and the _parentChain dictionary key to MaxIdentityFieldLength, matching the existing truncation of Uid/DisplayName. Without this cap, an attacker-controlled or pathological test framework could exhaust memory by emitting a very long TestNode.Uid. - Switch the HelpInfoAllExtensionsTests asset csproj to the dedicated \\$ placeholder (the JUnit extension now follows the experimental versioning scheme and no longer matches \\$). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Follow-up commit on this branch:
|
The JUnitReport extension ships under the experimental 1.0.0-alpha versioning scheme, so the acceptance test assets that reference it must use MicrosoftTestingExtensionsJUnitReportVersion instead of MicrosoftTestingPlatformVersion. HelpInfoAllExtensionsTests was already updated; this fixes the remaining two assets (MSBuild known-extension registration and MSTest JUnit retry tests) where NuGet restore was failing with NU1102. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Follow up to the AzDO/TRX dogfooding pattern: now also exercise the newest report extension (JUnitReport, microsoft#8850) and the OpenTelemetry processing pipeline end-to-end as part of our own CI test runs. JUnit: - test/Directory.Build.targets adds Microsoft.Testing.Extensions.JUnitReport as a ProjectReference for every test project that opts into MTP and wires --report-junit / --report-junit-filename into the MSBuild path. - azure-pipelines.yml and eng/pipelines/steps/test-non-windows.yml also pass --report-junit --report-junit-filename "{asm}_{tfm}.xml" on the Release dotnet test --test-modules path (which bypasses MSBuild evaluation). - Each test project's Program.cs explicitly calls AddJUnitReportProvider() (TestingPlatformBuilderHook auto-registration only fires for NuGet consumers, not source ProjectReferences). OpenTelemetry: - test/Directory.Build.targets adds Microsoft.Testing.Extensions.OpenTelemetry as a ProjectReference. - Each test project's Program.cs registers AddOpenTelemetryProvider with AddTestingPlatformInstrumentation on both the tracer and meter builders. No exporter is wired: the SDK still creates real Activities and Counters because the sources/meters now have listeners, so data flows through the full OpenTelemetryResultHandler pipeline and is dropped at the export stage. Sufficient to exercise the handler in CI without polluting logs. - VSTestBridge.UnitTests gates OTel registration on !NATIVE_AOT to match its existing AOT-aware Program.cs pattern (the OTel SDK uses reflection). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Adds a new MTP extension
Microsoft.Testing.Extensions.JUnitReportthat emits JUnit-format XML reports compatible with Jenkins, GitLab, Azure DevOps and other CI parsers.Closes #4268
CLI
--report-junit— enable the report--report-junit-filename <name>— override the default filenameDesign
Full RFC:
docs/RFCs/016-JUnit-Report.mdHighlights:
<testsuites>/<testsuite>for broad parser compatibility. Nested<testsuite>was rejected because Jenkins/Surefire handle depth-2+ inconsistently.<property name="testpath" value="A/B/C"/>. This is the v1 representation; we can evolve later if/when parsers gain richer hierarchy support.<testcase>child ordering (properties → skipped/error/failure → system-out/-err) — required by Surefire/Jenkins.MaxTestPathLength = 64K,MaxParentChainDepth = 1024with cycle detection; bounded stream/message/identity/trait sizes.passed→ empty body,skipped→<skipped>,failed→<failure>,errored/timedOut→<error>.Discovered/InProgressnodes are not emitted as testcases but participate in the parent-chain capture.XmlWriterSettings.CheckCharacters = false+XmlSafeTexthelper replacing invalid chars with U+FFFD.Implementation
src/Platform/Microsoft.Testing.Extensions.JUnitReport/(net8.0, net9.0, netstandard2.0) modelled on the existingHtmlReportextension.9F3CD35A-4E62-43EF-B7F1-426D30E850EDwired intoMicrosoft.Testing.Platform.csprojandTestFx.slnx.HtmlReport; XLF regenerated viaUpdateXlf(never hand-edited).Tests
All green locally:
JUnitReportTests):JUnit_WhenReportJUnitIsNotSpecified_JUnitReportIsNotGeneratedJUnit_WhenReportJUnitIsSpecified_JUnitReportIsGeneratedInDefaultLocationJUnit_WhenReportJUnitFilenameIsSpecified_JUnitReportIsGeneratedWithThatNameJUnit_WhenReportJUnitFilenameContainsPath_JUnitReportIsGeneratedInThatPathJUnit_WhenTestsFailOrSkip_JUnitReportContainsExpectedOutcomesHelpInfoAllExtensionsTestsandMSBuild.KnownExtensionRegistrationupdated with the new option/registration assertions (43/43 passing).Verification commands
Compatibility note
The extension option
--report-junitcollides with xUnit pre-4.0's identically-named CLI option. This is fatal-by-design and documented in the RFC: enabling both at once will fail fast.