feat(seo): AI-crawler baseline + content layer + validation gates by rubenvdlinde · Pull Request #55 · ConductionNL/conduction-website

rubenvdlinde · 2026-05-18T12:27:24Z

Summary

Makes conduction.nl AI-crawler-friendly across three layers, then locks the contract with a CI gate.

Discovery. robots.txt allows every AI bot Conduction wants (citation and training), points at both locale sitemaps. llms.txt lists the 15 shipping apps with one-line descriptions for AI dev-tools. Both per-locale sitemaps emit clean URL sets.
Structured data. Every page carries Organization + WebSite JSON-LD (from preset 3.4.0). Every /apps/<slug> page adds a SoftwareApplication block emitted by <DetailHero> with category-mapped applicationCategory and operatingSystem: Nextcloud. /support, /install, /iso each ship a <FAQ> that auto-emits FAQPage JSON-LD.
Content. A Wikipedia-style 1-paragraph lede sits under <DetailHero> on every shipping app page so AI fetchers extract a clean "what is this app" answer in plain text, not from a hero component. New OG card (1200x630, cobalt + point-up hex) so LinkedIn / Slack / AI previews don't render a 404.
Gates. scripts/validate-ai-baseline.mjs runs as a postbuild step on every build. 15 checks, hard-fail. Any future PR that drops the baseline gets blocked at CI time.

Posture is open: we allow both citation and training. Documented at the top of static/robots.txt; flipping any subset to Disallow is a one-line edit.

What's in here

static/robots.txt, static/llms.txt, static/img/og-conduction.png (new files)
docusaurus.config.js (PR 55 commit 1 wrapped with site-local JSON-LD; later commit drops the wrap once preset 3.4.0 ships the same defaults via headTags + themeConfig.image + themeConfig.metadata)
@conduction/docusaurus-preset bumped to ^3.4.0
src/pages/apps/*.mdx (15 files): Wikipedia-style lede injected under <DetailHero>
src/pages/install.mdx: new 7-question FAQ block
src/pages/iso.mdx: new 6-question FAQ block
src/pages/support.mdx: untouched; its existing 6-question <FAQ> now auto-emits FAQPage JSON-LD via preset 3.4.0
scripts/validate-ai-baseline.mjs: 15-check hard-fail validator, wired as postbuild

CI gate (15 checks, all hard-fail)

✓ robots.txt exists and is non-empty - 1620 bytes
✓ robots.txt has Sitemap line - 2 sitemap line(s)
✓ robots.txt allows AI search bots - OAI-SearchBot, Claude-SearchBot, PerplexityBot
✓ robots.txt has at least User-agent: * Allow: / - default-allow present
✓ llms.txt exists and has content - 4428 bytes
✓ sitemap.xml has at least 50 URLs - 67 URLs
✓ nl/sitemap.xml has at least 50 URLs - 67 URLs
✓ homepage has >= 2 JSON-LD blocks, all valid JSON - 2 blocks, all valid
✓ homepage JSON-LD includes Organization and WebSite - Organization + WebSite
✓ homepage has og:image, og:type, twitter:site, twitter:card - all four present
✓ og:image URL resolves to a file in the build - img/og-conduction.png (50269 bytes)
✓ /support has FAQPage JSON-LD with >= 5 questions - /support: 6 Q&A
✓ /install has FAQPage JSON-LD with >= 5 questions - /install: 7 Q&A
✓ /iso has FAQPage JSON-LD with >= 5 questions - /iso: 6 Q&A
✓ /apps/openregister has SoftwareApplication JSON-LD - BusinessApplication/Nextcloud
All 15 AI-baseline checks passed.

Adding a check: append to the CHECKS list in scripts/validate-ai-baseline.mjs. Keep the failure message specific so a CI annotation tells the reader exactly what regressed.

Notes

Every lede ends with "Released under EUPL-1.2". That is canonical for the whole fleet. Some apps declare AGPL in appinfo/info.xml or README only because the Nextcloud app store does not include EUPL in its license picker, so AGPL is the closest copyleft option there. Not a contradiction; both the marketing copy and the schema.org license field carry EUPL-1.2.
softwarecatalog's docs have two readings (IT-asset management vs internal user-sync). The lede matches the conduction.nl positioning (IT-asset management).
Rollout to the 17 per-app docs sites (openregister.conduction.nl etc.) is the next big piece. They will inherit most of the baseline automatically via preset 3.4.0 (same DetailHero, same FAQ, same robots-txt fallback); per-site overrides are sitemap content, og:image, and llms.txt.

Test plan

CI postbuild step runs validation and reports all 15 checks
Visit /robots.txt, /llms.txt, view-source of /, /apps/openregister, /support, /install, /iso to confirm JSON-LD blocks
LinkedIn / Slack URL preview shows the new OG card
https://search.google.com/test/rich-results validates Organization + WebSite on /
https://validator.schema.org/ validates SoftwareApplication on /apps/openregister
https://validator.schema.org/ validates FAQPage on /support, /install, /iso
Read each of the 15 lede paragraphs in-context on the app pages; flag any that miss the app's actual positioning

…SON-LD, OG card) Conduction.nl was technically crawlable (SSG, no JS-rendering risk) but commercially invisible to AI agents: no robots.txt, no llms.txt, zero JSON-LD, no og:image, Dutch sitemap stale. This change ships the AI- crawling baseline so we get cited in ChatGPT/Claude/Perplexity answers and our content gets pulled into long-term training corpora — both, on purpose, posture documented in static/robots.txt. Changes - static/robots.txt — explicit allow for OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, GPTBot, ClaudeBot, CCBot and friends. Two Sitemap: lines (en + nl) for locale-suffix-blind crawlers. - static/llms.txt — llmstxt.org-format index pointing at the 15 shipping apps plus company/contact tells. Dev-tool consumers (Cursor, Aider, MCP doc servers) read this; AI-search adoption is thin but the cost is near zero. - docusaurus.config.js — wraps createConfig() return to inject Organization + WebSite JSON-LD via headTags[] on every page, sets themeConfig.image to the new OG card, and adds twitter:site / twitter:card / og:type baselines via themeConfig.metadata. Also adds sitemap options to the classic preset (changefreq, priority, ignore /academy/tags/** in both locales) so the sitemap is consistent across /sitemap.xml and /nl/sitemap.xml. - static/img/og-conduction.png — new 1200x630 brand OG card. Cobalt #4376FC background, point-up hex motif per brand rules, white wordmark, EUPL-1.2 / ISO 27001 / Made in Amsterdam tells. Out of scope (filed separately): SoftwareApplication JSON-LD per /apps/<slug> page, FAQPage schema on /support, fleet rollout to the 17 per-app docs sites and to the @conduction/docusaurus-preset. Verified: local production build emits both JSON-LD scripts in raw HTML on / and /apps/, og:image + twitter:* meta on every page, og:type stays 'article' on blog posts (not overridden), and both sitemap.xml + nl/sitemap.xml carry 67 URLs each with tag pages excluded.

Conduction writing rules ban em-dashes in published copy. robots.txt counts as published copy (served at conduction.nl/robots.txt and read by every AI crawler). Two em-dashes in the comment header replaced with a colon and a period.

@type

CI now blocks any change that drops the AI-crawler baseline. The validation script runs as a postbuild step so every `npm run build` (local dev + CI) re-asserts the contract. Checks (12, all hard-fail) - robots.txt: exists, has Sitemap lines, allows OAI-SearchBot + Claude-SearchBot + PerplexityBot, has the default User-agent: * Allow: / block. - llms.txt: exists, >= 1 KB (catches an accidental placeholder). - sitemap.xml + nl/sitemap.xml: each emit >= 50 URLs. - Homepage: >= 2 JSON-LD blocks, every block valid JSON, includes Organization and WebSite @type entries. - Homepage: og:image, og:type, twitter:site, twitter:card meta tags present. og:image URL resolves to an actual file in the build (so LinkedIn / Slack / AI previews never render a 404). - /apps/openregister: has SoftwareApplication JSON-LD with applicationCategory and operatingSystem populated. This is the regression marker for the DetailHero schema emission that ships from @conduction/docusaurus-preset >= 3.4.0. Wiring - postbuild npm script runs scripts/validate-ai-baseline.mjs. - validate:ai-baseline npm script for ad-hoc runs. - @conduction/docusaurus-preset bumped to ^3.4.0 so the SoftwareApplication check has a guaranteed source. Adding a check: append to the CHECKS list in scripts/validate-ai-baseline.mjs. Keep the failure message specific so a CI annotation tells the reader exactly what regressed.

…locks PR 1 shipped @conduction/docusaurus-preset 3.4.0 with AI-baseline defaults (Organization JSON-LD, SoftwareApplication via DetailHero, FAQPage via FAQ, postBuild robots.txt fallback, og:image, twitter meta, sitemap options). This commit makes conduction.nl actually consume that baseline and adds the citation-friendly content layer on top. Changes - docusaurus.config.js: drop the hand-rolled headTags wrap, the inline Organization / WebSite JSON-LD constants, and the themeConfig.image / themeConfig.metadata overrides. All of it comes from createConfig() now. ~80 lines removed. - src/pages/apps/*.mdx (15 files): Wikipedia-style 1-paragraph lede injected immediately under <DetailHero/> on every shipping app page. Each lede follows the template "<Name> is an open-source <category> app for the Nextcloud workspace. <Capabilities.> <Distinguishing positioning.> Released under EUPL-1.2 and maintained by Conduction since 2019." Sourced from each app's own docs site / repo, then edited for brand voice (no em-dashes, one claim per sentence, concrete verbs). - src/pages/install.mdx: new 7-question FAQ block (admin install, Nextcloud version range, multi-app install, offline install, uninstall + data, missing-from-store, SLA / support). - src/pages/iso.mdx: new 6-question FAQ block (9001 vs 27001 scope, self-host coverage, BIO alignment, DigiD, commonground.nu stack, certificate request flow). - scripts/validate-ai-baseline.mjs: three more checks pin the FAQPage schema to >= 5 questions each on /support, /install, /iso. Hard-fail on regression. Validation now runs 15 checks total (was 12). Notes - /support already shipped a 6-question <FAQ>. Now that preset 3.4.0 emits FAQPage JSON-LD automatically, the existing block validates without any source change. - mydash README declares AGPL-3.0 while the lede template ends with EUPL-1.2. Filed as a follow-up to verify which is canonical; the lede currently says EUPL-1.2 to match brand positioning. - softwarecatalog has two readings in its docs (IT-asset management vs internal user-sync). The lede goes with the conduction.nl positioning (IT-asset management), which matches the existing tagline in apps-catalog.js.

The centralised ConductionNL/.github documentation workflow only builds on push, schedule, and workflow_dispatch. That meant a PR could drop robots.txt, break a sitemap, or remove SoftwareApplication JSON-LD without the validate-ai-baseline.mjs check ever running in CI. The defect lands; nightly cron catches it the next morning; production is already broken. This workflow runs npm run ci (= npm ci --legacy-peer-deps && npm run build) on every pull_request to development or documentation. The build's existing postbuild step runs the 15-check validator, which exits non-zero on regression. A follow-up explicit invocation acts as belt-and-braces in case anyone removes the postbuild hook. GITHUB_TOKEN is passed so the prebuild app-downloads fetch stays under the 60-req/hr unauthenticated rate limit (same pattern the centralised workflow uses).

….txt) (#136) Matches the fleet rollout pattern from ConductionNL/conduction-website#55: - docs/package.json: @conduction/docusaurus-preset bumped from ^2.6.1 to ^3.4.0 (ships Organization + WebSite JSON-LD on every page, SoftwareApplication via <DetailHero>, FAQPage via <FAQ>, default og:image + twitter meta, postBuild robots.txt fallback). - docs/package.json scripts: postbuild npm hook runs the validator on every build, validate:ai-baseline alias for ad-hoc runs. - docs/scripts/validate-ai-baseline.mjs (new): generic 8-check validator. Asserts robots.txt + Sitemap, AI-bot allow lines, sitemap.xml, Organization + WebSite JSON-LD, og:image, twitter meta, og:image file resolves. Exits 1 on regression. - docs/static/llms.txt (new): llmstxt.org-format index. LarpingApp intro + docs link block + Conduction contact. CI gate runs via ConductionNL/.github documentation workflow on every PR (post that workflow's split-build-from-deploy change).

….txt) (#776) Matches the fleet rollout pattern from ConductionNL/conduction-website#55: - docusaurus/package.json: @conduction/docusaurus-preset bumped from ^2.6.1 to ^3.4.0 (Organization + WebSite JSON-LD on every page, SoftwareApplication via <DetailHero>, FAQPage via <FAQ>, default og:image + twitter meta, postBuild robots.txt fallback). - docusaurus/package.json scripts: postbuild npm hook runs the validator on every build, validate:ai-baseline alias for ad-hoc runs. - docusaurus/scripts/validate-ai-baseline.mjs (new): generic 8-check validator. Exits 1 on regression. - docusaurus/static/llms.txt (new): llmstxt.org-format index with OpenConnector intro and docs link block. The Docusaurus site currently lives at `docusaurus/`. PR #690 is migrating it to `docs/`; when that lands these files will move with the rename. If conflicts arise during that migration, replay this commit on top of the new `docs/` location. CI gate runs via ConductionNL/.github documentation workflow on every PR (post that workflow's split-build-from-deploy change).

PR #55 bumped @conduction/docusaurus-preset to ^3.4.0 in package.json without regenerating package-lock.json. CI uses `npm ci --legacy-peer-deps` which is strict about lockfile sync, so every Documentation workflow run on development since #55 merged has failed at the install step. The AI baseline never reached production. This commit: - Bumps the package.json range from ^3.4.0 to ^3.5.0 (the latest preset published — also matches what the rest of the fleet now uses). - Regenerates package-lock.json against the new range. - The cooldown rule in .npmrc (min-release-age=1) blocks fresh @conduction/* releases by default; this regen used --min-release-age=0 per the .npmrc comment. After this lands, the documentation workflow should install cleanly, the postbuild validate-ai-baseline check should pass, and the deploy job should publish the AI baseline (robots.txt, llms.txt, JSON-LD, og-conduction.png, sitemaps) to www.conduction.nl for the first time.

…k override) (#57) Companion to the AI-crawler baseline shipped in #55. Targets Google, Bing, DuckDuckGo, and the AI surfaces those engines feed. Title dedup (7 pages) - about, apps, demo, partners, sidecars, solutions, support: drop the ", Conduction" suffix from frontmatter title:. Docusaurus appends " | Conduction" automatically, so the prior frontmatter produced "Apps, Conduction | Conduction" in browser title + og:title + SERP. After: "Apps | Conduction" (clean). - index: tighten title from "Conduction, Public Tech for Nextcloud" to "Public Tech for Nextcloud" (still gets " | Conduction" appended). Trim meta description from 180 to ~157 chars so Google stops truncating in SERP. - /apps/<slug> pages left as-is: "OpenRegister, Conduction" reads as branding (made-by signal), not a typo, so the appended duplicate is acceptable. Preset 3.6.0 - Bump @conduction/docusaurus-preset to ^3.6.0 (companion preset PR ships sitemap lastmod, drops priority/changefreq, adds searchConsoleVerification opt, ships absolute www.conduction.nl defaults for footer legal links). - package-lock.json regenerated with --min-release-age=0 per .npmrc cooldown override. legalLinks override - conduction-website hosts /privacy, /terms, /iso itself, so we override the preset's new absolute defaults back to relative routes. Per-app subdomain sites inherit the absolute defaults (they don't have /privacy etc.) so those 645 sitewide broken internal links get fixed automatically on their next deploy. searchConsoleVerification scaffold - Commented-out placeholder in docusaurus.config.js. Once ops claims the property in Google Search Console / Bing Webmaster Tools, the tokens go here and the preset emits a meta tag per token.

rubenvdlinde added 4 commits May 18, 2026 14:26

style(seo): drop em-dashes from robots.txt comments

fa27dba

Conduction writing rules ban em-dashes in published copy. robots.txt counts as published copy (served at conduction.nl/robots.txt and read by every AI crawler). Two em-dashes in the comment header replaced with a colon and a period.

rubenvdlinde changed the title ~~feat(seo): AI-crawler-friendly metadata (robots.txt, llms.txt, JSON-LD, OG card)~~ feat(seo): AI-crawler baseline + content layer + validation gates May 18, 2026

rubenvdlinde marked this pull request as ready for review May 18, 2026 17:32

rubenvdlinde merged commit 40ff90b into development May 18, 2026
4 of 5 checks passed

rubenvdlinde deleted the feat/ai-crawler-friendly branch May 18, 2026 18:04

This was referenced May 18, 2026

feat(docs): wire AI-crawler baseline (preset 3.4.0 + validator + llms.txt) ConductionNL/larpingapp#136

Merged

feat(docs): wire AI-crawler baseline (preset 3.4.0 + validator + llms.txt) ConductionNL/openconnector#776

Merged

This was referenced May 18, 2026

feat(docs): per-app OG card + accurate llms.txt links ConductionNL/opencatalogi#588

Merged

feat(docs): per-app OG card + accurate llms.txt links ConductionNL/openregister#1566

Merged

This was referenced May 18, 2026

feat(docs): per-app OG card + accurate llms.txt links ConductionNL/docudesk#183

Merged

feat(docs): per-app OG card + accurate llms.txt links ConductionNL/mydash#184

Merged

fix(deps): sync package-lock.json with preset 3.5.0 (unbreaks deploy) #56

Merged

rubenvdlinde mentioned this pull request May 18, 2026

feat(seo): traditional-search baseline (titles, preset 3.6, legal-link override) #57

Merged

5 tasks

WilcoLouwerse mentioned this pull request May 19, 2026

docs(academy): public Hydra leerlijn — 6 chained Dutch tutorials #37

Merged

4 tasks

This was referenced May 19, 2026

feat: AI-crawler-friendly metadata + cobalt hero across apps/solutions/sidecars #82

Closed

feat(pages): cobalt hero + intro lede across apps, solutions, sidecars #83

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(seo): AI-crawler baseline + content layer + validation gates#55

feat(seo): AI-crawler baseline + content layer + validation gates#55
rubenvdlinde merged 5 commits into
developmentfrom
feat/ai-crawler-friendly

rubenvdlinde commented May 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rubenvdlinde commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's in here

CI gate (15 checks, all hard-fail)

Notes

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rubenvdlinde commented May 18, 2026 •

edited

Loading