Skip to content

docs: revamp README with feature showcase and auto-updating benchmarks#6430

Merged
thomhurst merged 4 commits into
mainfrom
docs/readme-revamp
Jul 16, 2026
Merged

docs: revamp README with feature showcase and auto-updating benchmarks#6430
thomhurst merged 4 commits into
mainfrom
docs/readme-revamp

Conversation

@thomhurst

Copy link
Copy Markdown
Owner

Summary

Rewrites the README to actually showcase TUnit, and adds automation so the new benchmark table never goes stale.

README changes

  • Hero section: realistic test snippet followed by the actual failure output TUnit produces (expression capture + focused differs at member object diff) — all message formats verified against TUnit.Assertions.Tests.
  • Performance section: scenario × framework table (TUnit AOT / TUnit / xUnit v3 / NUnit / MSTest) populated from docs/static/benchmarks/latest.json, with an honest note that source generation trades a slightly slower build for faster runs.
  • Feature tour: data-driven tests, chained/custom assertions ([GenerateAssertion]), shared fixtures via ClassDataSource + IAsyncInitializer, parallelism control (DependsOn, NotInParallel, ParallelLimiter<T>), hooks, and TUnit.Mocks (previously just a table row).
  • Integrations: ASP.NET Core, Aspire, Playwright, and FsCheck snippets — all copied/adapted from real test code in this repo.
  • Added TUnit.FsCheck and TUnit.OpenTelemetry to the packages table; noted F#/VB.NET support.

Benchmark automation

  • New .github/scripts/update-readme-benchmarks.js renders the table between <!-- benchmarks:start/end --> markers from latest.json (skips NA cells, fails loudly if markers vanish).
  • speed-comparison.yml runs it after process-benchmarks.js and includes README.md in the existing weekly auto-PR, so the table refreshes with zero manual effort.

Verification

  • Ran the injection script locally — table populated with the committed 2026-07-12 results (visible in this diff).
  • Snippet syntax cross-checked against real usages: HasCount(n), Contains(predicate), .Because(...), [Before(Test)], IFoo.Mock()/Returns/WasCalled(Times.Once), ClassDataSource shared scopes, TestWebApplicationFactory<T>, AspireFixture<T>, PageTest, [FsCheckProperty].

Rewrites the README around what TUnit can actually do: hero snippet with
real failure output, focused object-diff example, benchmark table vs
xUnit/NUnit/MSTest, and sections for mocking, shared fixtures, custom
assertions, and the ASP.NET Core/Aspire/Playwright/FsCheck integrations.

The benchmark table lives between <!-- benchmarks:start/end --> markers
and is regenerated weekly: a new update-readme-benchmarks.js step in the
speed-comparison workflow rewrites it from latest.json inside the
existing automated benchmark PR.
@codacy-production

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 2 minor

Alerts:
⚠ 2 issues (≤ 0 issues of at least minor severity)

Results:
2 new issues

Category Results
BestPractice 1 minor
CodeStyle 1 minor

View in Codacy

🟢 Metrics 12 complexity

Metric Results
Complexity 12

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@claude

claude Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Code review

Reviewed the README rewrite, the new update-readme-benchmarks.js script, and the speed-comparison.yml workflow changes.

Verification performed: cross-checked every API surface shown in the new README snippets against the actual source (HasCount, Contains(predicate), .And chaining, Because, [GenerateAssertion], T.Mock() (C# 14 extension static members), Arg.Any<T>() (auto-imported via global using static), Mock.HttpClient/OnGet/RespondWithJson, Mock.Logger<T>/VerifyLog().AtLevel().ContainingMessage(), TestWebApplicationFactory<T>, AspireFixture<T>, PageTest, [FsCheckProperty]) — all exist and match the shown usage. Also traced the benchmark script's field access (latest.categories[...], Mean/Version/NA handling) against the real docs/static/benchmarks/latest.json produced in this PR — matches, including the NA fallback for the missing TUnit_AOT SetupTeardownTests cell.

Finding:

README.md — "Assertions that explain themselves" section:

await Assert.That(order.Items)
    .HasCount(3)
    .And.Contains(item => item.Sku == "ABC-123");

CollectionAssertionBase.HasCount(int, ...) is marked [Obsolete("Use Count().IsEqualTo(expectedCount) instead.")] (TUnit.Assertions/Sources/CollectionAssertionBase.cs:332). This compiles (it's a warning, not an error), but the flagship README — the first thing new users read — is showcasing a deprecated API as the idiomatic way to assert on collection size. Since the whole point of this section is "here's the current best way to write assertions," recommend swapping to .Count().IsEqualTo(3).And.Contains(...) so the example doesn't steer newcomers toward something the library itself is trying to move away from.

Everything else — the benchmark automation script, the workflow wiring (git add ... README.md so the auto-PR now includes README changes), and the rest of the README content — looks correct and well cross-checked.

HasCount(int) is [Obsolete]; showcase the recommended Count().IsEqualTo(3)
path instead. Chains identically via CollectionCountEqualsAssertion.
'a fraction of a second slower' understates source-gen cost on very large
suites (scales with test count). Reworded to 'pay a little up front'.
Property injection lets subclasses inherit shared fixtures without
re-threading constructor parameters; note constructor injection still works.
@thomhurst
thomhurst merged commit 9b4317a into main Jul 16, 2026
3 of 4 checks passed
@thomhurst
thomhurst deleted the docs/readme-revamp branch July 16, 2026 12:12
@claude

claude Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Code review (follow-up)

Re-reviewed after the four fixup commits. Diff scope: README.md, .github/scripts/update-readme-benchmarks.js (new), .github/workflows/speed-comparison.yml.

Previously flagged issue — resolved: the HasCount(3) obsolete-API example was swapped for .Count().IsEqualTo(3) in e56c5a5. Confirmed the current diff no longer references the obsolete overload. The two other fixup commits (softened build-time claim, property-injection framing) are accurate reflections of the code and don't introduce new issues.

New finding — maintainability (minor, non-blocking):

update-readme-benchmarks.js hardcodes its own SCENARIOS map (AsyncTests, DataDrivenTests, MatrixTests, ScaleTests, MassiveParallelTests, SetupTeardownTests → display labels), duplicating the category key list that already exists as RUNTIME_DESCRIPTIONS in the sibling process-benchmarks.js (.github/scripts/process-benchmarks.js:12-19). The two maps have diverged failure semantics: process-benchmarks.js falls back to a generated description for an unrecognized category (RUNTIME_DESCRIPTIONS[testClass] || \Performance benchmark for ${testClass}.`), while the new script's .filter(([key]) => latest.categories[key])` silently drops any category not in its hardcoded map — so if a new benchmark scenario is ever added to the runtime suite, it shows up on the docs benchmarks page but silently never appears in the README table, with no CI failure to catch the omission.

Since this table is now stitched into the weekly automated PR with zero human review expected (per the PR description), a silent-drop failure mode is the one place this automation could quietly go stale. Two options that would close the gap:

  • Have update-readme-benchmarks.js warn/console.error (and optionally fail) when latest.json contains a category key with no entry in SCENARIOS, rather than passing over it silently.
  • Or derive the scenario list from latest.json's own category keys plus a single shared label map imported by both scripts, so there's one source of truth instead of two that can drift.

Not a blocker for this PR — the automation as written matches today's six scenarios correctly — but worth a quick follow-up so the "never goes stale" guarantee in the PR description actually holds for future scenario additions.

Everything else — the marker-replacement logic, the workflow step ordering (runs after process-benchmarks.js, before the git add that now includes README.md), and the rest of the README content/snippets — checks out.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e4ed6d0c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread README.md
### ASP.NET Core

```csharp
public class ApiFactory : TestWebApplicationFactory<Program>;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use valid class declarations in integration snippets

Readers copying the new ASP.NET Core sample hit a C# syntax error here because public class ...; is not a valid class declaration; the Aspire sample below uses the same semicolon-only form. These examples need an empty body such as { } (or another valid declaration) before users can try the TUnit integration code.

Useful? React with 👍 / 👎.

This was referenced Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant