Search quality, source prioritization, OutSystems connector, dark mode & indexing perf (#47) * feat(search): per-assistant + global cross-encoder reranking, unbiased candidates Incremental, A/B-comparable reranking instead of a single global switch. Two-level gate: rerank runs only when the global master switch RERANK_ENABLED (a GPU-backed model server is deployed) AND the per-assistant opt-in Persona.rerank_enabled are both on. Default off everywhere, so existing assistants and the GPU-free local setup are unchanged and need no GPU. - Persona.rerank_enabled column (+ migration f6a7b8c9d0e1, server_default false), threaded through upsert/create_update_persona and the persona API models, with a 'Rerank results (beta)' toggle in the assistant editor. - Single resolver _resolve_skip_rerank() in retrieval_preprocessing is now the one place both chat and Slack decide reranking (Slack passes skip_rerank=None to share it). Legacy ENABLE_RERANKING_* flags kept as a fallback. - RERANK_MODEL_NAME makes the cross-encoder env-selectable (prod can pick a stronger model, e.g. BAAI/bge-reranker-v2-m3); model server warms it when RERANK_ENABLED. - Retrieval split: when reranking is on, skip the two-query source-prioritization flow (it normalizes a narrow source-filtered set independently, inflating those scores and polluting the rerank candidate window) and run a single all-sources query; when off, the legacy prioritized flow is unchanged. Driven by prioritize_sources=query.skip_rerank. - Tests: global x per-assistant resolver matrix; single-vs-two-query split. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s: gpu-inference component (co-located rerank on a GPU node) Optional kustomize component that pins the inference-model-server onto a GPU node pool (nodeSelector agentpool=gpupool, toleration sku=gpu:NoSchedule, nvidia.com/gpu: 1) and serves the cross-encoder reranker (RERANK_MODEL_NAME, default BAAI/bge-reranker-v2-m3) alongside the embedding + intent models. Also sets real cpu/mem requests+limits (base leaves them empty -> eviction-prone). Opt in from the prod overlay's components: and set RERANK_ENABLED=true in env.properties. The existing model-server image already bundles CUDA torch, so no rebuild. Local omits the component and runs GPU-free. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: branch design doc for reranking/recency/retrieval quality Captures the verified query-path analysis (chat + Slack), the rerank flow and its corrected mental model, the recency decay math + levers (incl. the dead 'auto' auto-detect finding), the source-prioritization normalization bias and its two-path fix, the incremental per-assistant rollout, the GPU sizing/plan, the implementation map, and how to enable in prod. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(search): one-shot LLM relevance filter + source diversity + TEI reranker client - LLM relevance filter is now a single listwise call on the MAIN llm (llm_eval_chunks_listwise + LISTWISE_CHUNK_FILTER_PROMPT, fails open), gated independently of reranking by LLM_RELEVANCE_FILTER_ENABLED x per-assistant llm_relevance_filter (resolver _resolve_skip_llm_chunk_filter). No GPU. - Reranker served by Hugging Face TEI on CPU when RERANK_SERVER_URL is set (CrossEncoderEnsembleModel /rerank path); model server skips loading the cross-encoder in that case. Replaces the GPU plan. - _query_vespa simplified to a single all-sources query (removed the two-query source-prioritization union and its score-inflation bias). - Source diversity moved to final selection: ensure_source_diversity in doc_pruning reserves up to SOURCE_DIVERSITY_RESERVED_SLOTS slots for PROTECTED_SOURCES, so KB/web aren't crowded out. Always-on, global, no per-assistant knob. - Chat per-conversation toggles (use_reranking / use_relevance_filter) threaded via SearchTool -> SearchRequest. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(chat): per-conversation Rerank / Relevance toggles in the chat input bar Two per-conversation switches (default off) wired through sendMessage -> CreateChatMessageRequest (use_reranking / use_relevance_filter), independent of the assistant's own settings. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s: serve the reranker via CPU TEI (drop the GPU plan) - optional/tei-rerank: Hugging Face TEI on CPU (bge-reranker-v2-m3, fp32), Deployment + Service, health probes, model-cache volume. - prod overlay includes it + sets RERANK_ENABLED / RERANK_SERVER_URL / LLM_RELEVANCE_FILTER_ENABLED. Local sets RERANK_ENABLED (no RERANK_SERVER_URL) so the model server loads the reranker in-process — no GPU anywhere. - Removed the optional/gpu-inference component. Co-Authored-By: Claude Opus 4.8 (1M context) * test+docs: rerank/relevance/diversity tests + design-doc and dev-setup updates - Unit: relevance-filter gating matrix, listwise parser, source-diversity promotion/caps/disable (replaces the removed two-query test). - Integration: TEI rerank transport (mocked), real CPU cross-encoder reordering (MiniLM), filter_chunks with a stub LLM. - Docs updated to the final design (TEI-on-CPU, source diversity at selection, two assistant knobs); CONTRIBUTING documents local in-process reranking. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s: bake bge-reranker-v2-m3 into our own ONNX TEI image The TEI CPU image is ONNX-Runtime-based and bge-reranker-v2-m3 ships no ONNX weights, so the stock image 404s on onnx/model.onnx. Our own image exports the model to ONNX at build time (HF Optimum; pinned optimum 1.23.3 + torch 2.2.2 + numpy<2 for the >2GB external-data path) and bakes it at /model — no runtime download, no re-download on restart, no HF dependency. Deployment uses the ACR image with --model-id /model (dropped the emptyDir that shadowed /data and the unneeded istio annotation). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-148, web vha-78 (rerank/relevance/diversity) Built + deployed from feature/improve-queries; api-server self-migrated persona.rerank_enabled on rollout. Validated live: TEI reranker healthy and scoring correctly. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s: serve the reranker on GPU via the upstream TEI image CPU serving was validated as non-viable (24-98s for 15 chunks on prod; the long blocking inference also starved /health and got the pod liveness-killed). Move reranking to a T4 GPU node pool (Standard_NC4as_T4_v3, tainted gpu=true:NoSchedule) where the same bge-reranker-v2-m3 reranks in ~0.6-3.2s (45ms for short inputs). - Use the UPSTREAM TEI GPU image directly (ghcr turing-1.5) instead of building our own: TEI's GPU backend is Candle+safetensors, which the model ships, so the ONNX-export/custom-image dance (a CPU-runtime-only constraint) is gone. Deleted the custom Dockerfile. - Take the pod out of the istio mesh (sidecar.istio.io/inject: false): leaf service, PERMISSIVE mTLS so the api-server still reaches it, and an injected sidecar isn't up during the init phase (so the prefetch init container couldn't reach the network). - Prefetch the model into the PVC with the Python HF client in an init container: TEI's Rust hf-hub client fails on HF's redirect with 'relative URL without a base'; the Python client handles it. TEI loads offline from /data/model. Cluster-side (not in repo): added a gpu=true:NoSchedule toleration to the nvidia-device-plugin-daemonset so it schedules on the tainted GPU node and advertises nvidia.com/gpu. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): make source-diversity config explicit in env.properties PROTECTED_SOURCES and SOURCE_DIVERSITY_RESERVED_SLOTS were running on code defaults (web,sfkbarticles / 2), so the always-on diversity logic was active but invisible in the configmap. Pin them to the current defaults for visibility and tunability — no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(auth): authorize valid X-API-Key requests under enforced auth (OIDC) These api keys are service credentials for automation and intentionally don't map to a User. Enabling OIDC (AUTH_TYPE=oidc) flipped DISABLE_AUTH off, so current_user started 403'ing api-key requests (no session, and the keys don't resolve to a user) — bouncing automation into the SSO login flow. current_user now authorizes a request carrying a valid X-API-Key as an anonymous service caller (user=None, which endpoints already handle). Browser requests without a session still 403 (SSO gate intact), and a key alone does not grant admin (current_admin_user still requires an admin user). Adds request_has_valid_api_key() mirroring validate_api_key's lookup+cache, plus integration tests locking both the SSO and api-key flows. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-149 (api-key auth under OIDC fix) Deployed + validated live: GET /api/persona with a valid x-api-key returns 200; without a key still 403s (SSO gate intact). Co-Authored-By: Claude Opus 4.8 (1M context) * web: app-wide dark mode (default) + chat UX overhaul + token theming - Make design-system color tokens CSS-variable-driven so they flip under a `.dark` ancestor (light values unchanged via fallbacks). Dark is now the app-wide default via a no-flash init script on ; users opt into light via the sidebar toggle (persisted app-wide to localStorage). - Typography: replace Inter with IBM Plex Sans (body) + Fraunces (display); empty-state headline uses the display font with a staggered entrance. - Chat polish: input bar elevated with an accent focus-glow (fixes the previously-broken focus ring); subtle atmosphere glow on the chat canvas; reusable da-fade-up motion (reduced-motion guarded). - Assistant picker ("Choose Assistant") gains a live search box (name + description) with an empty state. - Consistency sweep: map raw neutral colors to semantic tokens across shared components + search/admin so they flip correctly in dark. Followup judgment-call stragglers + per-page UX land in a separate commit. Co-Authored-By: Claude Opus 4.8 (1M context) * web(dark): tone down bright native controls in dark mode - Add color-scheme: dark to the .dark root so the browser renders native form controls (checkboxes, date pickers, default input backgrounds) and scrollbars in dark — fixes the glaring white checkboxes (assistant form Tools, knowledge- set list) and any browser-default white input backgrounds. - Brand the checkbox/radio accent-color to the app accent. - Give the chat Filters knowledge-set search input an explicit dark surface (bg-background + token text/placeholder) — it had no bg class and rendered a bright white box. Followup to 8c5f4f87 (kept separate so it can be reverted independently). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web vha-79 (app-wide dark mode + chat UX) Deployed + verified live: web pod Running 2/2 on vha-79, build-deploy verify shows live==manifest, and /auth/login serves the dark-mode no-flash init. Build note: the local Mac amd64 web build SIGSEGVs (next build / musl under Rosetta), so vha-79 was built natively via 'az acr build' on darwinacr and copied into sfbrdevhelmweacr (docker pull/tag/push) — same digest d53f076e. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: web deploy runbook (az acr build path, RBAC/SP, local SIGSEGV) Captures the working web-image deploy flow: local Mac next build SIGSEGVs under amd64 emulation (even with Rosetta), so build natively via az acr build on darwinacr, copy into sfbrdevhelmweacr, bump tag, apply, verify. Includes the Contributor/PIM requirement and the optional service-principal path for CI. Co-Authored-By: Claude Opus 4.8 (1M context) * web: add light/dark toggle to the global user menu The toggle previously lived only in the chat sidebar, so admin/assistants and other pages had no way to switch themes. Add a Light/Dark item to UserDropdown (the avatar menu present app-wide), reusing the same darwin-theme localStorage + .dark-on- mechanism. Verified: flips both ways, label reflects state. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(web): move UserDropdown theme hooks above the early return next build's eslint (rules-of-hooks) failed the prod build because the darkMode useState/useEffect sat after the `if (!combinedSettings) return null` early return. Hooks must be unconditional — moved them above it. (Dev mode didn't catch this; only the production build's lint does.) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web vha-80 (global light/dark toggle in user menu) Deployed + verified: web pod Running 2/2 on vha-80, verify shows live==manifest. Built via az acr build on darwinacr -> transfer to sfbrdevhelmweacr (digest a1f3167). Note: Docker Hub anon pull rate-limit on the node:20-alpine base flaked several ACR runs; succeeded once the window freed. Co-Authored-By: Claude Opus 4.8 (1M context) * feat: make rerank + relevance cluster-config-driven; disable + drop GPU Single source of truth = RERANK_ENABLED / LLM_RELEVANCE_FILTER_ENABLED env: - Backend surfaces the effective global flags via /settings (load_settings), mirroring the existing chat_file_max_size_mb env-injection (relevance also respects the DISABLE_LLM_CHUNK_FILTER kill-switch). - Frontend hides the per-conversation (ChatInputBar) and per-assistant (AssistantEditor) rerank/relevance toggles when disabled cluster-wide. - prod overlay: RERANK_ENABLED=false, LLM_RELEVANCE_FILTER_ENABLED=false; drop the tei-rerank component so the GPU node pool can be removed. Code is intact — flip the env (+ re-add the tei-rerank component and a GPU node pool) to re-enable. No code change needed to toggle. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-150 (/settings exposes rerank/relevance flags) Deployed + verified: load_settings() returns rerank_enabled=False, llm_relevance_filter_enabled=False — the cluster-level flags the chat + assistant UIs read to hide the toggles. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web vha-81 (hide rerank/relevance toggles when disabled) Deployed + verified: web Running 2/2 on vha-81; live /api/settings returns rerank_enabled=false / llm_relevance_filter_enabled=false, so the chat + assistant toggles are hidden. GPU pool (gput4) scaled to 0 — no GPU VM running. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s: self-managed Velero overlay for weekly Vespa backups Standalone prod overlay (k8s/overlays/prod-velero), applied separately from the app like prod-vespa. Backs up only the backup=vespa PVCs (Vespa index + configserver state) weekly, ttl 504h (last 3 retained). - reuses SP darwinvelero (155fae3c), which already holds Contributor on the backup/node/darwin RGs -> no Owner/UAA role assignment needed - BSL reuses the darwinaksbackup SA; disk snapshots go to the unlocked node RG - alerts via the existing robusta Prometheus stack: PrometheusRule + ServiceMonitor (release: robusta), incl. a no-recent-successful-backup staleness alert that catches silent failure - notifier CronJob posts per-run success/failure to #darwin-devs via the Slack bot token - SP secret + bot token sourced from gitignored files (templates committed) Replaces the prior self-managed Velero that silently failed for ~a year on an expired SP secret. Co-Authored-By: Claude Opus 4.8 (1M context) * chat: disable LLM relevance filter on the default assistants Sets llm_relevance_filter=false for the 3 seeded personas (Darwin, GPT, Paraphrase) so the on-boot upsert_persona() reseed doesn't re-enable the filter. A/B eval over 120 questions showed the filter yields no measurable answer-quality gain (20/20/80 A/B/tie, p~1.0) and discarded all chunks in 14% of cases. Global LLM_RELEVANCE_FILTER_ENABLED=false already gates it off; this keeps the per-assistant seed config consistent and durable. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: Azure-managed AKS Backup runbook (reference / alternative path) Runbook for the managed Azure Backup-for-AKS approach, kept for reference. The cluster uses self-managed Velero instead (k8s/overlays/prod-velero) because managed AKS Backup needs role assignments / Trusted Access that require Owner/UAA, which isn't available; this documents that path + the [OWNER] hand-off if it's ever pursued. Co-Authored-By: Claude Opus 4.8 (1M context) * docs: local Redis via compose + shared danswer-stack network Document starting Postgres + Redis together under the -p danswer-stack project (shared danswer-stack_default network), run Vespa via the manual docker run on that same network (the compose `index` service is unreliable locally), add Redis ping/stop + no-auth notes, and set REDIS_HOST=localhost for the host-run backend. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-151 (relevance-filter-off personas + settings gating) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(deletion): re-drive connector deletions orphaned by lost broker messages Connector deletion is the event-driven cleanup_connector_credential_pair_task on the non-durable Redis broker. If Redis or the celery worker restarts while it's queued, the broker message is lost but the task_queue_jobs row stays PENDING forever — the connector shows "Deleting" indefinitely and nothing re-runs it (deletion, unlike sync/prune, is never periodically rescheduled), while the delete API's dedup guard blocks resubmission. We had 5 connectors stuck Deleting since Mar–May for exactly this reason. Adds a periodic celery-beat task (check_for_stuck_deletion_tasks, every 30m) that re-enqueues any cleanup task whose latest task_queue_jobs row has been non-terminal past JOB_TIMEOUT (db.tasks.get_stuck_deletion_cc_ids). The cleanup task's per-cc-pair advisory lock makes a re-enqueue a no-op if a deletion is genuinely still running, and the fresh row stays live for JOB_TIMEOUT, so this self-throttles to one re-drive per cc-pair per window. Also recovers the existing stuck connectors on first run. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-152 (orphaned-deletion re-drive) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(web connector): opt-in latest-N version tracking for docs.uipath For docs.uipath.com versioned products (on-prem/standalone like Automation Suite, Robot standalone), index the latest N versions automatically instead of hard-coding each version URL — so new releases (~every 6 months) aren't missed. - get_uipath_docs_version_base_urls(): reads the product page's version selector (already newest-first, which correctly handles the calendar -> MAJOR.YYMM scheme change, e.g. 2.2510 newer than 2023.10), returns the latest N concrete version base URLs; falls back to [base_url] for non-docs.uipath, evergreen (only "latest"), or any fetch error. - OPT-IN via WebConnector(uipath_latest_versions=bool, max_versions=int=3), RECURSIVE only — existing connectors are untouched (auto-applying would make per-version connectors all crawl the same set). Re-evaluated each run, so new versions are picked up and the oldest drops off. - Recursion scope generalized to the N version prefixes (get_internal_links allowed_prefixes; WebConnector.recursive_prefixes). - Frontend: "Track latest versions" checkbox + "Number of latest versions" field on the Web connector form. - Fix: web poll fallback used is_polling=True (recursion off -> landing pages only) when the sitemap is unavailable; the product sitemap URL currently 404s, so this was the common path. Fall back to a full recursive crawl instead. - scripts/test_uipath_version_discovery.py: checks the agreed URL test set. Verified against the test set: robot/standalone/latest -> 3, apps/automation- suite -> 3, activities/other -> 1 (evergreen), automation-cloud -> as-is, learn.microsoft.com -> as-is (domain-gated). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-153 (uipath latest-N web connector) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web vha-82 (uipath latest-N connector form field) Co-Authored-By: Claude Opus 4.8 (1M context) * docs(AGENTS): slow indexing = per-doc Vespa visit; content-hash dedup already exists New "Critical facts that bite" #11: the indexing bottleneck is the per-document Vespa Visit-API existence check (_get_vespa_chunks_by_document_id, selection scan ~11s/doc), not the source crawl, and Vespa isn't CPU-bound. Documents that the Postgres content-hash dedup (Document.indexed_content_hash + get_doc_ids_to_update) already provides should NOT be re-implemented; notes why it can still re-index everything (from_beginning bypass, NULL legacy hashes, web connector not setting doc_updated_at) and that it self-heals as hashes backfill. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(web connector): never fall back to a product-base crawl on version-expand failure get_uipath_docs_version_base_urls previously returned [base_url] on any fetch/parse error. For a version-less product base (e.g. docs.uipath.com/automation-suite/automation-suite/) that means a recursive crawl spans EVERY version — the exact leak this feature prevents. Observed on connector 681: a transient init-time fetch failure made it index all 7 versions instead of the latest 3 (no cross-product leak, but cross-version). Now: retry the selector fetch (3x w/ backoff); on persistent failure RAISE (fail the run, retries cleanly) instead of crawling the product base. When no concrete versions are found, only crawl base_url as-is if it's already scoped to a version/'latest' (evergreen/deep pages); a version-less base RAISES. Test set (robot/standalone, apps/automation-suite, activities/other, automation-cloud deep, learn.microsoft) still passes. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-154 (uipath version-expand fallback hardening) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(web connector): scope POLL sitemap to recursive_prefixes (latest-N) The previous version-expansion only fixed the LOAD path and the 404-sitemap poll fallback. For products whose product sitemap EXISTS (e.g. automation-suite -> /products/automation-suite/sitemaps/en/sitemap.xml, 200, 4270 urls across all 7 versions), poll_source rebuilt to_visit_list from that sitemap filtered only by the product base, re-indexing EVERY version and bypassing the latest-N expansion (observed: connector 681 indexing 2021.10..2.2510). poll_source's RECURSIVE branch now also filters sitemap URLs by self.recursive_prefixes, so a uipath_latest_versions connector polls only the latest-N versions' changed pages (incremental preserved). For normal connectors recursive_prefixes == [base_url], so behavior is unchanged. Verified: 4270 -> 2592 urls, versions {2.2510, 2024.10, 2023.10} only. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-155 (poll sitemap latest-N scoping) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(web connector): default uipath latest-N tracking to 2 versions Change the default from 3 to 2 (max_versions): backend get_uipath_docs_version_ base_urls + WebConnector.__init__ defaults, and the admin form initial value + subtext. New latest-version connectors now track the last 2 versions unless overridden. Existing connectors with an explicit max_versions are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-156 (default latest-N=2); web Dockerfile node-base arg - backend vha-156 carries the default-2 change. - web/Dockerfile: NODE_BASE build-arg (defaults to docker.io node:20-alpine for local builds) so ACR CI can pull the base from our mirror (darwinacr.azurecr.io/library/node:20-alpine) and avoid Docker Hub's unauthenticated pull-rate limit, which 429'd the web build. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web vha-83 (form default latest-N=2, ACR node base) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(indexing): per-source concurrency cap override (uncap web) Add INDEXING_PER_SOURCE_CAP_OVERRIDES on top of the global INDEXING_PER_SOURCE_CAP so a single DocumentSource can run at a different concurrency than the default without affecting others. Web is the motivating case: every web cc-pair has its own dummy credential and mostly distinct domains, so it parallelizes safely, while Slack/Jira/Confluence stay pinned at 1 (rate limits / shared creds). Set web=0 in prod (uncapped; bounded by NUM_INDEXING_WORKERS). Scheduler threads the override through _build_running_view and _evaluate_dispatch_for_attempt via a new cap_for_source() helper; both gained an optional trailing cap_overrides arg (backward compat). +5 unit tests (uncap one source, finite override, non-overridden source unaffected, per-cc-pair lock still holds, env parsing). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(web): doc-set picker crash, EE 404 spam, indexing activity/failed shortcuts - getSourceMetadata: fall back to a generic tile for unmapped sources (e.g. ingestion_api) instead of throwing undefined.displayName — the document-set connector picker crashed the page on unmapped cc-pairs. - useUserGroups: pass a null SWR key when EE is off so it stops spamming /api/manage/admin/user-group 404s on every EE-gated page mount. - web connector page: alignTop on the long-subtext checkbox so it sits by its label instead of floating mid-paragraph. - Indexing status: composite 'Active (running + queued)' filter + read ?status= from the URL (Suspense-wrapped) so links land pre-filtered; new 'Indexing Activity' and 'Failed Indexing' sidebar shortcuts. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-157, web vha-85 backend vha-157: per-source indexing cap override (web uncapped). web vha-85: doc-set crash fix, EE 404 fix, indexing activity/failed shortcuts. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(web): smarter connector search in document-set picker The Pick-your-connectors dropdown matched only option.name with a single contiguous substring, so typing a URL fragment found nothing (web connectors carry their URL in config, not the display name) and pieces typed out of order didn't match. - Dropdown: add optional Option.searchableText (falls back to name) and switch the filter to token-based AND matching (split query on whitespace; every term must appear, any order, case-insensitive). - Document-set form: set searchableText to name + source + config summary (which holds base_url) so URL fragments and source names match. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web vha-86 (document-set connector search fix) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(web): rank connector search results best-match-first + cap rendering Builds on the searchableText/token-AND filter: matches are now scored and sorted so the closest result is at the top — exact name > name prefix > contiguous-in-name > contiguous-in-haystack > scattered tokens; ties break toward shorter names and earlier match position. Results are capped to the top 50 (with a 'showing top N of M' footer) so a broad query no longer renders hundreds of heavy rows. No query = original order, unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web vha-87 (best-match-first connector search ranking) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(db): pool_pre_ping + recycle on async engine The async engine (unlike the sync one) had no pool_pre_ping, so it handed out connections the server/proxy had already closed on idle -> asyncpg.InterfaceError: connection is closed -> intermittent 500s on async endpoints. This surfaced as frequent 'Application error' on /chat (the SSR data fetch hit a 500). Add pool_pre_ping=True + pool_recycle=3600 to match the sync engine and transparently reconnect stale connections. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(api): user-friendly errors — stop leaking raw SQL/exceptions to the UI Add server/utils.py::user_facing_http_exception and route the document-set, folder, tool, and Highspot-connector write handlers through it: ValueError passes through (already friendly), IntegrityError becomes an actionable message, everything else is logged server-side and replaced with a generic message. No raw psycopg2/SQLAlchemy text reaches the browser. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(web): chat landing redesign + resilient SSR + dark-mode markdown fixes - Empty-state landing: centered 'Start your day with Darwin' + input, a focused glow behind the chat box (dark elsewhere), wider layout, and the conversation text aligned with the input; placeholder -> 'How can I help you today?'; removed the assistant/filter/model + knowledge/source tiles; docs sidebar collapsed by default and auto-opens when results return. - Resilient chat SSR: fetchChatData uses Promise.allSettled + guards the results[4] destructure, so one failed fetch degrades gracefully instead of 500-ing the whole /chat render ('object null is not iterable'). - Dark mode: prose /headings inherit the light text color so bold bullet labels are no longer dark-on-dark; fixed a -z-10 backdrop stacking bug via . Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-158, web vha-88 Co-Authored-By: Claude Opus 4.8 (1M context) * perf(vespa): batch chunk-id visits + fix duplicate-append in bulk updates Two fixes to VespaIndex.update (used by doc-set sync, boost/ACL/hidden updates, indexing): 1. _get_vespa_chunks_by_document_id appended every chunk twice (a stray extend after the per-chunk loop), so every bulk update/delete issued 2x the Vespa writes. Removed the duplicate append. 2. Chunk-id lookup was one Vespa visit (a selection scan) PER document, so syncing N docs meant N scans. Added _get_vespa_chunk_ids_by_document_ids which selects up to _VESPA_VISIT_DOC_ID_BATCH (25) document_ids per visit (id1 or id2 or ...) -> one scan per batch. update() now uses it. Verified against live Vespa: identical chunk IDs to the per-doc path. Speeds up document-set sync (was ~285s/100 docs) and bulk Vespa updates. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-159 (vespa bulk-update perf) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(celery): run prune check every 15m, not every 5s check_for_prune_task scans every cc-pair (444+ here) with lazy-loaded connector/credential (N+1, ~8s/run). At a 5s beat cadence the runs overlapped and accumulated until they occupied all 10 worker threads, starving sync_document_set_task — document sets sat in 'syncing' for hours with no thread to run on. Pruning is governed by each connector's prune_freq (~daily), so 15-min checks are more than enough. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-160 (prune-check cadence fix) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(celery): cap concurrent document-set syncs at 2 Each sync_document_set_task fans out _NUM_THREADS (32) concurrent Vespa requests, so with worker concurrency 10 up to ~320 simultaneous Vespa calls could run — uncontrolled load with no Vespa-side throttle. The scheduler now counts in-flight syncs and only kicks off enough new ones to reach _MAX_CONCURRENT_DOCUMENT_SET_SYNCS (2); the rest wait for a later tick. Keeps Vespa load predictable while still draining the backlog. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-161 (cap doc-set sync concurrency at 2) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(vespa): skip documents with no chunks in bulk update (KeyError regression) The batched chunk-id lookup only creates a dict entry for documents that actually returned chunks, but update() iterated all_doc_chunk_ids[document_id] for every requested doc — so an orphaned doc with zero Vespa chunks (e.g. a Slack doc removed from Vespa but still in Postgres) raised KeyError and failed the whole document-set sync in a retry loop. Use .get(document_id, []) to skip chunkless docs, matching the prior per-document behavior. Verified batched == per-doc for real docs against live Vespa. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-162 (vespa bulk-update KeyError fix) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(celery): resumable document-set sync via persisted cursor sync_document_set_task now checkpoints its keyset cursor (last synced doc id) to key_value_store after each batch and resumes from it on the next run, instead of always restarting from cursor=None. Previously a set too large to finish within the 6h soft_time_limit (e.g. the ~216k-doc ServiceCatalog sets) re-did only its first batches every window and never reached the rest — an infinite non-progressing loop. Now it advances across restarts/timeouts and eventually completes. The cursor is cleared on successful completion, and a doc-set edit (membership change) drops it so the next sync re-tags from scratch. No schema migration (reuses key_value_store). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-163 (resumable doc-set sync) Co-Authored-By: Claude Opus 4.8 (1M context) * perf(celery): raise document-set sync concurrency cap 2 -> 3 Vespa measured to have headroom at cap=2 (content nodes ~2-2.5 cores, feed/ query containers idle, no 429s/timeouts, stable ~25s/batch). 3 concurrent (~96 Vespa calls across 3 content nodes) drains the backlog faster while staying well under the old uncapped ~320. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend vha-164 (doc-set sync cap 3) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(backend): chat history pagination, persona display_name, docs-version dedup, Highspot sync, slack blocklist, perf + tests - chat: paginated/date-bucketed sidebar history (get_chat_sessions_by_user limit/offset/time-window) + citation KeyError fix (translate_citations) - personas: display_name field (+migration a8b9c0d1e2f3) and N+1 eager-load fix on the admin list / edit endpoints (get_personas/get_persona_by_id) - retrieval: docs-version dedup in prune_documents (keep newest version per page; DOCS_VERSION_DEDUP_URL_SUBSTR) - highspot: spot->connector sync (connectors/highspot/sync.py) + endpoint + incremental delta-fetch - slackbot: response blocklist (+migration f7a8b9c0d1e2) - unit tests: translate_citations, clean_spot_name, indexing_concurrency, dedupe_doc_versions, persona_eager_load Co-Authored-By: Claude Opus 4.8 (1M context) * feat(web): chat readability + assistants UX + lazy-loaded sidebar history - chat history sidebar: Today loads by default; Previous 7/30 Days & Over 30 collapsed and lazy-load with a "Show more" button (ChatTab/ChatContext/ fetchChatData); removed redundant per-page theme toggle (ChatThemeToggle) - readability: AI response + ChatBanner/ChatPopup use dark:prose-invert; question text uses text-default; retrieved-docs keyword highlight removed - assistants: display_name field + hover tooltip + assistantDisplayName helper; My-Assistants drag fix for id=0; zero-document-set assistants flagged red; document-set picker split into Selected / Available - misc UI: intro heading/glow/logo, send-button states, doc-set creation connector-type filter + select-all, dark default theme, modal/border polish Co-Authored-By: Claude Opus 4.8 (1M context) * ops(prod): Apple-Silicon web cloud-build routing, dedup/Highspot env, tag bumps - build-deploy.sh: auto-route the web image to darwinacr cloud build + import on Apple Silicon (local next build SIGSEGVs under emulation); documented as AGENTS.md "Critical facts" #12 - env.properties: DOCS_VERSION_DEDUP_URL_SUBSTR + highspot in PROTECTED_SOURCES - kustomization: backend vha-170, web vha-100 (deployed) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(outsystems): inside.uipath.com connector with large-doc hardening One-time LoadConnector for the OutSystems-backed UiPath intranet: enumerates pages, fetches via the screenservice (interim session-cookie + x-csrftoken auth), extracts page HTML + attached files (PDF/DOCX/PPTX/XLSX, downloaded from the backing SharePoint via the W_Document metadata action), and indexes them. Auth swaps to a service account later with no change to the fetch/extract logic. Reliability for large/pathological docs (learned the hard way on prod): - HTTP (10,60) timeouts + skip-on-network-error - skip videos/images/oversized files (never download/parse) - long-token sanitizer (keeps tokenization linear) - 30K section-splitting (bounded chunk/tokenize units) - thread-bounded extraction; small batch size - skip list of ~20 giant table docs the CPU embedder can't handle Includes DocumentSource.OUTSYSTEMS + factory registration, the web admin page (cloned from sf-account) + source/credential types, ConnectorForm support for refreshFreq=null (one-time, no schedule), and unit tests for extraction, widget classification, the sanitizer, and section splitting. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(web/admin): responsive indexing tabs + sidebar active state - AdminSidebar: highlight the active item by path+query, so the Existing Connectors / Indexing Activity / Failed Indexing tabs (same route, different ?status=) each light up. Previously every tab looked identical, so a click gave no feedback. Wrapped in Suspense in Layout (useSearchParams). - Indexing status table: sync the status filter with the ?status= deep-link. Navigating between the tabs is client-side and doesn't remount, so the filter (seeded once on mount) never updated -> the click appeared to do nothing. - Document Sets: stop the "Cannot update while syncing" tooltip from being clipped by the cell's overflow-hidden (column width is held by table-fixed, truncation by the inner span, so the cell didn't need overflow-hidden). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(ops): verify image in registry before apply; bump prod tags build-deploy.sh now checks each target image actually exists in the registry (docker manifest inspect) immediately before `kubectl apply`. An interrupted or backgrounded run can bump the manifest tag without the build/push completing; applying that rolls deployments onto an unpullable tag (ImagePullBackOff), and for the dask scheduler it cascades the whole indexing pipeline down. The gate aborts before apply and leaves the cluster untouched. Also bumps prod image tags to the deployed values (backend vha-183). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump web=vha-102 (admin UX fixes: sidebar active-tab, status-filter sync, doc-set tooltip) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(assistants): opt-out visibility — new assistants appear for all users Flip per-user assistant visibility from opt-IN to opt-OUT. Previously chosen_assistants was both the visible allow-list AND the order, so any admin-created assistant was invisible to every user who had ever customized their list (hidden and never-seen were indistinguishable). Now visibility = "every accessible assistant MINUS hidden_assistants"; a new column hidden_assistants holds explicit per-user exclusions, and chosen_assistants is order/default only. A newly created (e.g. admin) assistant is in nobody's hidden list, so it shows up for all users automatically; a user can still disable one for themselves. - models.py: User.hidden_assistants (ARRAY(int), NOT NULL default {}) - migration b9c0d1e2f3a4: additive column, no backfill (chat not rolled out) - PATCH /user/hidden-assistants; UserPreferences + from_model serialize it - web: orderAssistantsForUser filters by hidden then orders by chosen; Manage Assistants + gallery hide/show now write hidden_assistants - tests: orderAssistants (7) + UserInfo.from_model serialization (3) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-184 web=vha-103 (opt-out assistant visibility) Co-Authored-By: Claude Opus 4.8 (1M context) * ops(prod): add outsystems to PROTECTED_SOURCES (source-diversity reserved slots) inside.uipath.com OutSystems index (DocumentSource OUTSYSTEMS -> value 'outsystems') now gets reserved front slots at final doc selection, alongside web/sfkbarticles/highspot, so it isn't crowded out of the LLM prompt. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(search): source-reserved retrieval — guarantee curated sources reach candidates PROTECTED_SOURCES / SOURCE_DIVERSITY_RESERVED_SLOTS only reserves FINAL-prompt slots among docs retrieval already surfaced. It can't help when a chatty source (e.g. the #help-benefits-amer Slack archive) saturates the entire top-N and a relevant curated doc (web/KB/OutSystems) never enters the candidate set — a recall problem, not a prioritisation one. Observed on AMERBenefits: an RSU/vesting question returned 50/50 Slack candidates, 0 OutSystems, while the inside.uipath.com index has highly relevant RSU guides (top score 0.87) that were simply crowded out. Fix: SearchPipeline.retrieved_chunks now runs ONE extra source-scoped retrieval (SOURCE_RESERVED_RETRIEVAL_SLOTS) reusing the SAME filters (ACL + persona doc-set fence) plus a source_type restriction, and merges up to N PROTECTED_SOURCES chunks into the candidate set. The existing diversity reservation then carries them into the prompt. Universal: both chat and Slack flows funnel through SearchPipeline, so it applies to every assistant with no per-persona gating. - chat_configs: SOURCE_RESERVED_RETRIEVAL_SLOTS (default 0 = off) - pipeline: pure protected_source_topup() + _supplement_protected_sources() - prod env: SOURCE_RESERVED_RETRIEVAL_SLOTS=3, SOURCE_DIVERSITY_RESERVED_SLOTS 2->3 - tests: 5 unit tests for the top-up/dedupe/scope logic Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-185 (source-reserved retrieval) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(search): build supplemental query via copy(update=) — SearchQuery is immutable Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-186 (source-reserved retrieval immutability fix) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): cap docs-per-source in prompt so curated sources get cited Source-reserved retrieval + ensure_source_diversity already put curated docs (e.g. OutSystems RSU guides) at prompt positions [1][2][3], yet the LLM still cited only Slack — because the prompt held ~49 Slack docs vs 3 OutSystems, and the model grounds each claim in the dominant/conversational source. Citations are LLM-determined; the lever is prompt composition, not ranking/boosting. cap_docs_per_source (MAX_PROMPT_DOCS_PER_SOURCE) keeps the top-N docs per source (after diversity promotion) and drops the rest before the token cut, so no single source can monopolize grounding. Generic: lives in doc_pruning (universal chokepoint for both chat + Slack flows, all assistants), only binds when a source dominates, no per-persona config. - chat_configs: MAX_PROMPT_DOCS_PER_SOURCE (default 0 = off) - doc_pruning: cap_docs_per_source(), applied after ensure_source_diversity - prod env: MAX_PROMPT_DOCS_PER_SOURCE=8 - tests: 6 unit tests (cap/order/per-source/enum/disabled) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-187 (per-source doc cap) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(prompts): global authoritative-sources citation nudge (from PROTECTED_SOURCES) The per-source cap rebalances the prompt (3 OutSystems + 8 Slack) but the LLM still grounded in the conversational Slack threads. Add a generic, global nudge telling the model which sources are authoritative systems of record — derived from PROTECTED_SOURCES so it covers all of them and stays in sync — and to prefer citing them over chat discussions when they support the answer (and never cite an unsupporting source). Lives in build_task_prompt_reminders (rides the shared CITATION_REMINDER), so it applies to every assistant and both chat + Slack flows with no per-persona setup. Source names rendered via clean_up_source to match the "Source: X" doc labels. - prompt_utils: build_authoritative_sources_reminder(), appended when citations on - tests: 5 unit tests (names match doc labels, dedupe, gating, empty) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-188 (authoritative-sources citation nudge) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(prompts): strengthen authoritative-sources nudge to mandatory citation 'prefer' -> 'you MUST cite the authoritative document when it supports the statement; cite chat discussions only when no authoritative source does.' Aims to flip citation attribution away from near-duplicate Slack threads. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-189 (mandatory authoritative-sources nudge) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(prompts): grouped citations — 'Authoritative sources' vs 'Other sources' Mandatory 'you MUST cite authoritative' nudge failed to flip the LLM's attribution away from near-duplicate Slack threads. Reframe: let the model cite naturally, but require a trailing 'Sources' section with two labelled groups — authoritative systems of record vs the rest. Authoritative docs are promoted to prompt positions [1..3], so the model lists their [n] markers in the authoritative group and danswer parses them into real citations. Works with the model instead of against it; still global + prompt-only, derived from PROTECTED_SOURCES. Omits empty groups; only supporting docs listed. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-190 (grouped authoritative citations) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(prompts): revert to soft authoritative-source suggestion (drop grouped output) The grouped 'Authoritative sources / Other sources' instruction caused the LLM to MISLABEL its Slack citations as authoritative — wrong and trust-damaging. Revert to a soft, honest suggestion: name the authoritative systems of record (from PROTECTED_SOURCES) and ask the model to prefer citing them over chat discussions when they support the point, citing only sources that actually support the answer. Benign and helps on queries where the authoritative doc is the real source; doesn't fabricate or relabel on the hard cases. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-191 (soft authoritative suggestion) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): verify-then-retain authoritative citations (post-generation) Citations are the LLM's output and it inconsistently cites curated sources even when they're promoted to prompt position [1] (confirmed in chat + Slack across runs). Prompt nudging can't reliably fix this. Add a deterministic post-generation step in the shared Answer pipeline (covers chat + Slack, all assistants): - additive: the LLM's own inline citations are untouched - deduped: only authoritative (PROTECTED_SOURCES) docs the LLM did NOT cite, deduped by document_id (same page often appears as multiple chunks) - honest: one batched LLM call verifies each candidate actually SUPPORTS a statement in the answer (topic-match is not enough); fail-closed on any error - bounded: at most ONE extra call, and only when an uncited authoritative doc is in context (no call otherwise) Supporting docs are appended as an "Authoritative sources" markdown footer (renders in chat UI + Slack). Gated by AUTHORITATIVE_CITATION_RETENTION_ENABLED (default off; prod on). In this deployment fast_llm == main llm, so the verify call uses self.llm. - chat_configs: AUTHORITATIVE_CITATION_RETENTION_ENABLED - llm/answering/authoritative_retention.py: select/verify/footer + orchestrator - llm/answering/answer.py: accumulate answer+cited-ids in _process_stream, append verified footer after the citation stream - prod env: enabled - tests: 14 unit tests (selection/dedupe/parse/verify-fail-closed/footer) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-192 (verify-then-retain authoritative citations) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): retry the authoritative-citation verify call once The verify step is a non-streaming gateway completion that occasionally times out (observed transiently; normally 1.6-5.5s). Retry once before failing closed so a single gateway hiccup doesn't drop the authoritative-sources footer. Still fail-closed after retries (never appends on a real error). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-193 (verify-call retry hardening) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): merge retained authoritative citation into Sources + tighten gate Two changes per product feedback: 1. Single Sources section (no footer): inject the verified authoritative doc as a CitationInfo using its context position as citation_num. Since authoritative docs are promoted to positions 1-3 and the UI orders the Sources group by citation number (JS integer-key order / Slack list), it lands at the TOP of the existing single "Sources" section. Drops the separate markdown footer. 2. Tighter gate: only run when the answer cites NO authoritative source at all. If the LLM already cited any PROTECTED_SOURCES doc, do nothing (no verify call) — the answer is already authoritatively grounded. Net: at most one conditional verify call, only on answers missing authoritative citations; result merges into the one Sources section instead of a second block. - authoritative_retention.py: gate in select_authoritative_candidates; retained_authoritative_citations() returns CitationInfo (footer removed) - answer.py: yield the CitationInfo packets after the stream - tests: 18 (gate skip + no-LLM-call, context-position citation_num, verify reject) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-194 (authoritative citation merged into Sources) Co-Authored-By: Claude Opus 4.8 (1M context) * revert(answering): back to authoritative footer (merge-into-Sources collided) Merging the retained authoritative doc into the single Sources section assigned it a citation_num equal to its context position — which collides with the LLM's own citations (it owns the low numbers), and translate_citations de-dupes first-wins, so the injected citation was dropped (validated: 0 OutSystems shown). Putting it at the TOP of the numbered list would require renumbering the LLM's inline [[n]], which breaks the inline links. Revert to the 'Authoritative sources' footer block (reliably surfaces the link), but KEEP the tightened gate from the prior change: only run when the answer cites NO authoritative source at all (no verify call otherwise). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-195 (revert to authoritative footer) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): loosen authoritative-source verify to relevance Strict 'directly supports a statement' verification dropped clearly-relevant authoritative docs whose content didn't literally restate the answer — e.g. the 'Elastic Robot Orchestration Setup For AWS' KB article was at prompt position #1 for an ERO/Automation-Suite question but got filtered out because the answer's key statement was a negative ('not available self-hosted') the setup guide doesn't literally assert. Loosen the verify prompt to 'is a RELEVANT authoritative reference for this answer (same subject/scenario; need not restate it)'. Surfaces the authoritative KB/docs link in these cases; tradeoff is occasionally a topically-related doc, acceptable for the support-engineer use case. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-196 (loosen authoritative verify to relevance) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(answering): surface uncited authoritative docs even if another was cited The tightened gate ('skip if any authoritative source was cited') was too coarse: on an Orchestrator upgrade question the LLM cited a KB article (sfkbarticles) + Slack, which suppressed surfacing the relevant, uncited docs.uipath.com page (/2023.10/.../maintenance-considerations) that was at the front of the prompt — a regression vs the old 2-phase union. Loosen: candidates = any uncited relevant authoritative doc, regardless of whether some other authoritative source was cited. Verify call now fires whenever an uncited authoritative doc is present (still conditional/batched/retry-guarded). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-197 (surface uncited authoritative docs) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(answering): tighten authoritative-relevance verify (same-product != relevant) Relevance verify was too loose: for an 'AI in standalone Orchestrator' question it surfaced 'Does Standalone Orchestrator leverage Azure SignalR Service?' (a messaging/transport doc) as an authoritative reference. Tighten to require the doc be about the SAME SPECIFIC topic/feature, not merely the same product — with an explicit messaging/infra-vs-AI exclusion and 'when unsure, exclude'. Middle ground between strict 'supports-a-statement' (missed relevant docs) and loose 'same-subject' (false positives). Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-198 (tighten authoritative-relevance verify) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(answering): generalize relevance verify (drop AI-specific example) The messaging-vs-AI example was overfit to one question. State the general rule: the doc must be about the SAME SPECIFIC topic/feature, not merely the same product; a different feature/component/service isn't a relevant reference. No domain example. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-199 (generalize relevance verify) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): verify relevance against the matched chunk (reliable signal) The verify was judging relevance from the title + first 600 chars of the doc — usually boilerplate, not the passage that matched — causing oscillation (signalR false-positive when loose; 'Before you upgrade' false-negative when strict). Now pass each candidate's full MATCHED PASSAGE (LlmDoc.content = the retrieved chunk; capped at 4000 chars) and judge from that, with rebalanced wording (same topic as the answer = include; different feature/component = exclude). Candidate list is small (1-3), so the extra tokens are bounded. This makes precision come from the chunk rather than from overfit prompt strictness. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-200 (verify relevance against matched chunk) Co-Authored-By: Claude Opus 4.8 (1M context) * ops(prod): bump SOURCE_RESERVED_RETRIEVAL_SLOTS 3->6 The reserved pass guaranteed only the top-3 protected docs into the candidate set; relevant docs ranked #4 among protected sources (e.g. Orchestrator Maintenance Considerations on an upgrade question) fell off and never reached verify/footer. Bump to 6 so more of the top protected docs are guaranteed in; the per-source prompt cap still bounds how many reach the LLM. Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): query-time rewrite of versioned docs links to latest indexed The version dedup only collapses versions that were retrieved, so when retrieval surfaces a stale version (content is near-identical across versions, so which one ranks is ~arbitrary) the old link is shown even though newer versions are indexed (e.g. /2023.4/ surfaced while /2025.10/ exists). After pruning, rewrite each versioned docs.uipath.com link to the NEWEST version of that same page (same URL with the version segment stripped, slug preserved) found in the index. Fixes both inline docs citations and the authoritative footer (both use final_context_docs links). One PK-indexed prefix-scan per distinct page-prefix; no reindex. - doc_pruning: _versioned_url_parts + rewrite_docs_links_to_latest(db_session) - search_tool: rewrite final_context_documents after prune - tests: 6 (parse, rewrite-to-latest, noop-when-latest, slug-variants, non-docs, new-scheme) Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-201 (rewrite docs links to latest indexed version) Co-Authored-By: Claude Opus 4.8 (1M context) * feat(answering): version-aware docs link rewrite Blindly rewriting docs links to latest is wrong for version-specific questions ("is X supported in 23.10?" must resolve to the 23.10 doc, not latest). Make it version-aware: parse_question_doc_version() extracts a version from the question (23.10 -> 2023.10) only when EXACTLY one is named (multiple, e.g. 'upgrade 23.10 to 25.10', is ambiguous -> None -> latest). rewrite_docs_links() then resolves each docs page to that exact indexed version (even if older than retrieved); falls back to newest indexed when no version is specified. Not confused by '2.9 million'. - doc_pruning: parse_question_doc_version + rewrite_docs_links(target_version) - search_tool: parse from the query, pass through - tests: parse cases (single/multiple/none/2.9M), target-older, target-not-indexed Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-202 (version-aware docs link rewrite) Co-Authored-By: Claude Opus 4.8 (1M context) * fix(answering): anchor authoritative relevance verify to the question Judging relevance only against the ANSWER let cloud-connectivity-adjacent KBs through non-deterministically: for 'does standalone Orchestrator have AI?' the answer says 'AI requires an Automation Cloud connection', so KBs sharing that theme (Azure SignalR, 'Automation Cloud cannot be accessed' Studio error) matched the answer even though they don't address AI capability. Pass the QUESTION to the verify and judge whether the doc helps answer THE QUESTION's specific subject — keyword/product/service overlap is explicitly not enough, and error/troubleshooting docs are excluded unless that's what's asked. When in doubt, exclude. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(answering): simplify verify prompt to relevance-vs-(question AND answer) Replace the carve-out-heavy wording with a clean rule: include a candidate only if its matched passage is genuinely relevant to BOTH the question and the answer. Requiring relevance to the question (not just the answer) is what excludes cloud-connectivity-adjacent KBs (Azure SignalR, 'Automation Cloud cannot be accessed' Studio error) that shared the answer's 'needs cloud connection' theme but don't address the question's AI-capability subject. Co-Authored-By: Claude Opus 4.8 (1M context) * k8s(prod): bump backend=vha-204 (simplified verify prompt) Co-Authored-By: Claude Opus 4.8 (1M context) * docs(search-quality): document the source-prioritization + authoritative-citation pipeline Rewrite §5 from 'source diversity' into the full layered pipeline as built: 5.1 reserved retrieval (recall, SOURCE_RESERVED_RETRIEVAL_SLOTS=6) 5.2 diversity promotion (SOURCE_DIVERSITY_RESERVED_SLOTS=3) 5.3 per-source cap (MAX_PROMPT_DOCS_PER_SOURCE=8) 5.4 authoritative-sources nudge (soft, from PROTECTED_SOURCES) 5.5 verify-then-retain footer (chunk + question/answer relevance; why footer not merged) 5.6 version-aware docs link rewrite Plus the citation-attribution lesson (presence != citation), PROTECTED_SOURCES now web,sfkbarticles,highspot,outsystems, updated §6b/§8/§9, and §10 follow-ups (configmap externalization, inline-card version gap, indexed-content freshness, DB-backed prompts). Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context)

Uh oh!

Connector to scrape data from highspot #4

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions