chore: consolidate docs/ and fern/ into one documentation tree

[andreatnvidia]

Priority Level

Low

Task Summary

Consolidate the repository's split documentation layout so Fern configuration, source content, notebook inputs, generated Colab notebooks, scripts, and downloadable assets live under one top-level documentation tree.

Fern is now the canonical documentation system, but fern/ still consumes support inputs from docs/. The split duplicates path handling across Make targets, scripts, and GitHub workflows and makes ownership less obvious to contributors.

Technical Details & Implementation Plan

1. Inventory the remaining content under docs/ and every reference to it from fern/, the Makefile, scripts, workflow path filters, contributor documentation, and generated notebook links.
2. Choose one canonical top-level documentation tree. Prefer moving the remaining support inputs under fern/ because Fern is the active documentation system, unless Colab or external-link compatibility requires a different layout.
3. Move the remaining documentation inputs, including:
   • notebook sources
   • generated Colab notebooks
   • documentation scripts
   • downloadable recipe assets
4. Update all consumers:
   • Make targets and script defaults
   • GitHub Actions path filters and artifact paths
   • Fern generation and validation scripts
   • Colab badge and repository links
   • README, CONTRIBUTING, and agent instructions
5. Remove stale references to the old paths and document the final source/generated-artifact boundaries.
6. Validate the notebook generation, Fern preparation, Fern link checks, and preview/publish workflow syntax.

Acceptance criteria:

• All active documentation source and support files live under one top-level documentation tree.
• docs/ and fern/ no longer form a coupled two-root documentation system.
• Notebook generation and Colab links continue to work.
• Fern local check, preview, and publish workflows reference the new paths.
• Repository documentation clearly identifies source files versus generated artifacts.
• No stale references to the removed paths remain.

Investigation / Context

PR #581 introduced Fern alongside the existing MkDocs tree. PR #687 later removed MkDocs and intentionally retained docs/ only for Fern support inputs such as notebook sources, generated Colab notebooks, scripts, and recipe assets.

The current split was appropriate during migration, but Fern is now canonical and the transition period has ended. Issue #637 is adjacent but tracks separating docs preview build and deploy workflows, not directory consolidation.

A repository issue search found no dedicated issue tracking this consolidation.

Agent Plan / Findings

Treat this as a structural move, not a content rewrite.

Start by mapping path consumers and deciding the canonical root. Make the move in one PR so workflow filters, generated links, scripts, and documentation do not pass through an inconsistent intermediate state. Preserve public-facing notebook and documentation behavior, and call out any external links that cannot be redirected.

Dependencies

Coordinate with #637 if both tasks modify docs preview workflow paths; otherwise no hard dependency.