Skip to content

PROGRESS.md

Latest commit

 

History

History
222 lines (176 loc) · 12.4 KB

PROGRESS.md

File metadata and controls

222 lines (176 loc) · 12.4 KB
Raw

Sharp Docs — Progress Summary

Session dates: 2026-06-07 (scaffold) → 2026-06-07 (Google Gemini migration) → 2026-06-07 (persistence + BM25 + store management) → 2026-06-07 (tests + OTLP + responsive + eval + toasts) Status: ✅ Complete — full RAG pipeline with hybrid search, disk persistence, OTLP tracing, responsive UI, error toasts, 39 tests, and retrieval eval benchmarks. Production build passes.

What Is Built

Project scaffold

  • Next.js 16 + TypeScript + Tailwind v4 in C:\Users\filip\Documents\sharp-docs
  • Dependencies: langchain, @langchain/google-genai, @langchain/community, cheerio, html-to-text, uuid

AI Provider

  • Chat: gemini-2.5-flash via ChatGoogleGenerativeAI (streaming, temp 0.3)
  • Embeddings: gemini-embedding-001 via GoogleGenerativeAIEmbeddings
  • Auth: Google AI Studio free tier API key in .env

Ingestion pipeline (src/lib/ingestion/)

File Purpose
fetcher.ts Fetches HTML from a URL, strips nav/footer/scripts with cheerio, extracts clean text via html-to-text (15s timeout)
chunker.ts RecursiveCharacterTextSplitter — 1000-char chunks with 200-char overlap
embedder.ts Google gemini-embedding-001 via LangChain singleton
pipeline.ts Orchestrates fetch → chunk → embed → store, returns IngestionResult with per-step error isolation

Vector store (src/lib/vectorstore/)

File Purpose
store.ts In-memory Map<string, DocumentChunk> with vectorSearch, hybridSearch (vector + BM25 fusion), removeDocumentsByUrl, getUniqueSources, clearStore
bm25.ts Full BM25 implementation — tokenizer, IDF, term-frequency with k1=1.5, b=0.75, ranked scores normalized to 0-1
persistence.ts Serialize/deserialize to data/store.json via fs/promises, auto-saves on every mutation

RAG chain (src/lib/rag.ts)

  • retrieveContext() — embeds query + runs hybrid search (vector 70% weight + BM25 30% weight), builds context with source citations
  • streamChat() — full pipeline: retrieval → gemini-3.5-flash streaming via SSE, sources sent as first SSE event, token-by-token response
  • Built-in trace spans (retrieve + generate timing), recorded to observability ring buffer

Startup & preload

File Purpose
src/instrumentation.ts Next.js instrumentation hook — loads data/store.json on startup, hydrates in-memory store + BM25 index

API routes (App Router)

Route Method Function
/api/chat POST SSE streaming: sources → token* → done
/api/ingest POST Accepts { urls: string[] } (max 10), returns { results }
/api/store GET Returns { sources, totalChunks } for all ingested sources
/api/store?url=... DELETE Removes all chunks for a source URL, returns { removed, sourceUrl, totalChunks }
/api/traces GET Returns { stats, recentTraces } — avg duration, retrieval time, generation time, sources found

UI (src/app/page.tsx)

  • Sidebar: ingestion form, multi-source list with hover-delete buttons, store chunk count, Gemini model footer
  • Chat: SSE stream consumption with real-time token display, collapsible source citations (relevance %), stop button
  • Auto-fetches /api/store on mount to populate source list

Observability (src/lib/observability.ts)

  • In-memory trace ring buffer (last 100 requests)
  • Stats: avg total duration, avg retrieval time, avg generation time, avg sources found
  • Exposed via /api/traces

Live Test Results (verified 2026-06-07, session 3)

GET /api/store
→ { "sources": [], "totalChunks": 0 }  (empty start)

POST /api/ingest  { "urls": ["https://nextjs.org/docs"] }
→ { "chunksIngested": 4, "sourceTitle": "Next.js Docs | Next.js", "errors": [] }

GET /api/store
→ { "sources": [{ "title": "Next.js Docs | Next.js", "chunkCount": 4 }], "totalChunks": 4 }

POST /api/chat  { "query": "What is the App Router?" }
→ 4 sources (relevance 0.43-0.74), streaming Gemini 3.5 Flash response with citation:
  "The App Router is the newer router in Next.js that supports new React 
   features like Server Components..."

GET /api/traces
→ { "totalTraces": 1, "avgDurationMs": 3449, "avgRetrievalMs": 573, 
    "avgGenerateMs": 2876, "avgSourcesFound": 4 }

DELETE /api/store?url=https://nextjs.org/docs
→ { "removed": 4, "totalChunks": 0 }

GET /api/store  (after delete)
→ { "sources": [], "totalChunks": 0 }

What's Complete (Session 4 — 2026-06-07)

1. Test Suite (Vitest — 39 tests, all passing)

  • src/lib/__tests__/bm25.test.ts — index building, empty/no-term queries, score normalization, relevance ranking (7 tests)
  • src/lib/__tests__/store.test.ts — add/remove/clear, duplicate IDs, source grouping, vector search by embedding similarity, hybrid search fusion, disk hydration (14 tests)
  • src/lib/__tests__/chunker.test.ts — chunk count, metadata propagation, content preservation, custom sizes (4 tests)
  • src/lib/__tests__/rag.test.ts — retrieveContext with empty/populated store, source citation fields, streamChat ReadableStream, no_context path, trace inclusion (6 tests)
  • src/lib/__tests__/observability.test.ts — record/retrieve, reverse chronological, ring buffer cap, limit, getById, empty stats, average computation (8 tests)
  • Run: npm test / npm run test:watch / npm run test:coverage

2. OpenTelemetry OTLP Export

  • src/lib/telemetry.ts — initTelemetry() / shutdownTelemetry(), auto-starts if OTEL_EXPORTER_OTLP_ENDPOINT is set
  • Wired into src/instrumentation.ts for startup
  • src/lib/rag.ts — OpenTelemetry spans wrapping chat-request, retrieve-context, and generate-response with attributes (sources_found, response_length)
  • Both OTLP and in-memory ring buffer run in parallel
  • Config in .env.example

3. Responsive Layout

  • Desktop: static 320px sidebar + chat area
  • Mobile (< lg): sidebar hidden, hamburger menu in top bar, slide-over overlay
  • Chat bubbles: max-w-[90%] on mobile, max-w-[80%] on desktop
  • Input/form padding scales down on mobile

4. RAG Evaluation (src/lib/eval.ts)

  • Precision@1/3/5, Recall@3/5, MRR metrics on retrieval quality
  • loadPresetQueries() auto-generates eval queries from ingested chunk content
  • runEval() computes per-query and aggregate metrics
  • Runnable as: npx tsx src/lib/eval.ts

5. Error Toast Notifications

  • Toast component in bottom-right corner with auto-dismiss after 5 seconds
  • Types: error (red) and success (green)
  • Wired into ingest failures, source deletion failures, and chat failures
  • Dismiss button on each toast

Remaining (Nice-to-Haves)

  • RAGAS integration — generation faithfulness/answer relevancy metrics (needs LLM-as-judge call)
  • E2E tests — browser-based tests for ingestion → chat flow
  • PWA / offline — service worker for cached responses

File Structure

src/
  instrumentation.ts     — startup store preload + OTLP telemetry init
  app/
    layout.tsx           — metadata: "Sharp Docs"
    page.tsx             — full chat UI (client, responsive sidebar, toasts)
    globals.css          — Tailwind v4
    api/
      chat/route.ts      — POST SSE streaming chat
      ingest/route.ts    — POST URL ingestion
      store/route.ts     — GET sources / DELETE source
      traces/route.ts    — GET observability stats
  lib/
    types.ts             — all shared interfaces
    rag.ts               — RAG chain (hybrid search + stream + OTLP spans)
    observability.ts     — trace buffer + stats
    telemetry.ts         — OTLP exporter (init/shutdown)
    eval.ts              — retrieval eval (P@k, Recall@k, MRR)
    ingestion/
      fetcher.ts         — URL → cleaned text
      chunker.ts         — text → chunks via LangChain
      embedder.ts        — chunks → vectors via gemini-embedding-001
      pipeline.ts        — full ingest orchestration
    vectorstore/
      store.ts           — in-memory map + vector/hybrid search + source mgmt
      bm25.ts            — BM25 keyword index
      persistence.ts     — JSON disk persistence
    __tests__/
      bm25.test.ts       — 7 tests
      chunker.test.ts    — 4 tests
      observability.test.ts — 8 tests
      rag.test.ts        — 6 tests
      store.test.ts      — 14 tests
vitest.config.ts         — Vitest config with @/ alias and v8 coverage
.env                     — Google AI Studio key + model config
.env.example             — template with OTLP vars
data/
  store.json             — persisted vector store snapshot

All Five Priorities Delivered

  1. ✅ Tests — 39 Vitest tests across 5 files (bm25, store, chunker, rag, observability)
  2. ✅ OpenTelemetry — OTLP trace export with spans wired into rag.ts
  3. ✅ Responsive layout — mobile sidebar overlay, hamburger menu, responsive sizing
  4. ✅ RAG evaluation — src/lib/eval.ts with P@k, Recall@k, MRR benchmarks
  5. ✅ Error toasts — bottom-right toast notifications for ingest/delete/chat failures

Review & Hardening (Session 5 — 2026-06-10)

A code review pass surfaced several issues that are now fixed:

  • Removed unused exact dependency — a stray package not referenced anywhere in the codebase.
  • Removed dead code in persistence.saveStore (an unused serialized array).
  • Fixed lint errors — resolved react-hooks/set-state-in-effect errors in page.tsx and cleared all unused-import/var warnings across rag.ts, embedder.ts, and eval.ts. npm run lint is now clean.
  • Fixed the test suite — rag.test.ts was hitting the live Google API (failing without a key). The @langchain/google-genai SDK is now mocked, so all 39 tests run fully offline.
  • Added ingest URL validation — /api/ingest now rejects non-http(s) and malformed URLs before fetching.
  • Hardened streaming — mid-stream LLM errors now emit a clean error SSE event (handled client-side) and OpenTelemetry spans are always ended, fixing a span leak on failure.
  • Rewrote the README — replaced the create-next-app boilerplate with real setup, env var, API, and structure docs.

Verified: npm test (39 passing), npm run lint (clean), npm run build (passing).

Review & Hardening (Session 6 — 2026-06-13)

A second review pass focused on retrieval quality, performance, and robustness:

  • BM25 O(N²) → O(N) IDF. idf() previously rescanned every document for every query term while scoring every document. Document frequencies are now precomputed at index-build time, so scoring is linear in the corpus.
  • Asymmetric embedding task types. Queries are now embedded with RETRIEVAL_QUERY and stored passages with RETRIEVAL_DOCUMENT, which is how Gemini embeddings are meant to be used for retrieval. ⚠️ Existing stores must be re-ingested — vectors created before this change are not comparable to the new query embeddings (the old demo store.json was reset to empty).
  • Reciprocal Rank Fusion. Hybrid search now fuses the vector and BM25 result rankings (RRF) instead of blending raw scores. Cosine and BM25 scores live on different scales, so the old weighted sum skewed results; RRF only depends on rank position.
  • Source-labeled context. Retrieved passages are now prefixed with their source title/URL before being handed to the model, so it can actually cite which document an answer came from (the prompt asked for citations the model previously couldn't make).
  • Atomic, serialized persistence. Snapshots are written to a temp file and atomically renamed, and concurrent saves are serialized through a write queue — a crash or overlapping ingest can no longer corrupt store.json.
  • Fetcher guards. Non-HTML responses and pages over 5 MB are rejected before parsing, preventing garbage embeddings and memory blowups.
  • API input validation. Both POST routes reject malformed JSON bodies, and chat history is sanitized to well-formed { role, content } messages before reaching the prompt.
  • Scoring robustness. Cosine similarity guards against embedding dimension mismatches (no more NaN-poisoned rankings), and the generation-eval relevancy score clamps negative cosine to zero instead of taking its absolute value.
  • Dead code removed. Unused similaritySearch / getStoreSnapshot helpers deleted; citation snippets no longer append a stray ellipsis to short content.

Verified: npm test (39 passing), npm run lint (clean), npm run build (passing).