Enterprise Websites

[ngoiyaeric]

In system prompts enable business websites to be indexed crawled, ragged and automated.

[ngoiyaeric]

@coderabbitai plan and research for tools how to index websites

[coderabbitai]

Coding Plan

Summary

• Extend the existing database schema with two new tables for websites and embedded document chunks, leveraging the already-defined vector(1536) custom type for pgvector storage
• Follow the established settings tab pattern to add enterprise website management UI with CRUD operations backed by new server actions
• Implement a crawl-and-embed pipeline using Jina AI for content fetching (matching the existing retrieve tool pattern) and OpenAI for embedding generation
• Create a new enterpriseSearch agent tool following the existing tool factory pattern, performing semantic similarity search via pgvector and returning relevant chunks for RAG

Design Choices

Design Choice 1: Embedding Model Selection

Options Considered:

1. OpenAI text-embedding-ada-002 (1536 dimensions) - matches existing vector(1536) schema definition
2. OpenAI text-embedding-3-small (1536 dimensions) - newer model, same dimensions
3. Google Gemini embeddings - would require different dimension configuration

Chosen Option: 1

Rationale: Option 1: OpenAI text-embedding-ada-002 aligns with the existing vector(1536) column definition already present in the schema and the explicit comment in lib/db/schema.ts referencing this model.

Design Choice 2: Crawl Trigger Mechanism

Options Considered:

1. On-demand crawling when user adds a URL or clicks "refresh" in settings
2. Background worker/cron job for periodic re-crawling
3. Hybrid approach with both on-demand and scheduled refresh

Chosen Option: 1

Rationale: Option 1: On-demand crawling keeps the initial implementation simple, avoids infrastructure complexity (no job queue/workers needed), and follows the existing pattern where tools fetch content at request time via Jina AI.

Design Choice 3: Crawl Scope

Options Considered:

1. Single page per URL - user explicitly adds each important page
2. Full site spider with depth limits - automatically discover and index linked pages
3. Sitemap-based crawling - parse sitemap.xml to discover pages

Chosen Option: 1

Rationale: Option 1: Single page per URL matches the existing retrieve tool pattern, keeps scope manageable, and gives users explicit control over what content enters their knowledge base.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Database Foundation

Establish the data model for storing enterprise websites and their crawled, embedded content. This phase creates the persistence layer that all subsequent features depend on.

Task 1: Define Enterprise Schema Tables

Add two new tables to the Drizzle schema: one for user-configured websites and one for the crawled document chunks with vector embeddings.

• Add enterpriseWebsites table to lib/db/schema.ts with columns: id, userId (FK to users), url, name, status (enum: pending/indexed/failed), lastCrawledAt, createdAt
• Add enterpriseDocuments table to lib/db/schema.ts with columns: id, websiteId (FK to enterpriseWebsites), userId (FK), title, content, url, embedding (using existing vector custom type), chunkIndex, createdAt
• Export both tables and add type inference exports following existing pattern

Task 2: Create Database Migration

Generate and customize the migration SQL to include pgvector extension enabling and appropriate indexes.

• Run drizzle-kit generate to create the base migration SQL in drizzle/migrations/
• Prepend CREATE EXTENSION IF NOT EXISTS vector; statement for non-Supabase environments
• Add HNSW vector index on enterprise_documents.embedding using vector_cosine_ops for efficient similarity search
• Add standard B-tree indexes on user_id and website_id foreign keys
• Follow the statement-breakpoint convention from existing migrations

🤖 Prompt for AI agents

This phase establishes the database schema and migration for enterprise website
indexing. The goal is to add two new Drizzle ORM tables and a corresponding
migration with the necessary indexes.

**Schema changes in `lib/db/schema.ts`:**
- Add `enterpriseWebsites` table with columns: `id`, `userId` (FK to users),
`url`, `name`, `status` (enum: `pending`/`indexed`/`failed`), `lastCrawledAt`,
`createdAt`
- Add `enterpriseDocuments` table with columns: `id`, `websiteId` (FK to
`enterpriseWebsites`), `userId` (FK to users), `title`, `content`, `url`,
`embedding` (use the existing `vector` custom type for 1536 dimensions),
`chunkIndex`, `createdAt`
- Export both tables and add type inference exports following the existing
patterns in the file

**Migration in `drizzle/migrations/`:**
- Run `drizzle-kit generate` to produce the base migration SQL
- Prepend `CREATE EXTENSION IF NOT EXISTS vector;` to the migration for
non-Supabase environments
- Add an HNSW vector index on `enterprise_documents.embedding` using
`vector_cosine_ops` for cosine similarity search
- Add standard B-tree indexes on `user_id` and `website_id` foreign key columns
- Follow the `statement-breakpoint` convention used in existing migrations

Phase 2: Configuration and Content Ingestion

Build the user-facing settings interface for managing enterprise websites and implement the crawling and embedding pipeline that processes website content.

Task 1: Create Enterprise Settings Server Actions

Implement server actions for CRUD operations on enterprise websites and the crawl/embed pipeline.

• Create lib/actions/enterprise.ts with actions: getEnterpriseWebsites(userId), addEnterpriseWebsite(userId, url, name), removeEnterpriseWebsite(websiteId), refreshEnterpriseWebsite(websiteId)
• Implement crawlAndIndexWebsite(websiteId) action that: fetches content via Jina AI (same pattern as retrieve tool), chunks text into ~500 token segments, generates embeddings via OpenAI API, and stores in enterpriseDocuments
• Wire addEnterpriseWebsite to automatically trigger crawlAndIndexWebsite after inserting the website record
• Handle errors by setting website status to "failed" with appropriate error messaging

Task 2: Implement Text Chunking Utility

Create a utility function for splitting crawled content into embedding-appropriate chunks.

• Add lib/utils/chunking.ts with a chunkText(content, options) function
• Implement paragraph-aware splitting that respects sentence boundaries where possible
• Target ~500 tokens per chunk with configurable overlap (~50 tokens) for context continuity
• Return array of chunks with index position for maintaining document order

Task 3: Implement Embedding Generation Utility

Create a utility for generating vector embeddings using OpenAI's API.

• Add lib/utils/embeddings.ts with generateEmbeddings(texts: string[]) function
• Use OpenAI's text-embedding-ada-002 model via their Node SDK
• Support batch processing (OpenAI allows up to 2048 inputs per request)
• Return array of 1536-dimension float arrays matching input order

Task 4: Build Enterprise Settings UI Tab

Add a new "Enterprise" tab to the settings interface for managing business websites.

• Create components/settings/components/enterprise-form.tsx following existing form component patterns
• Display a table of configured websites showing: name, URL, status, last crawled timestamp
• Include "Add Website" form with URL and display name inputs
• Add per-row actions: "Refresh" button to re-crawl, "Remove" button with confirmation
• Show loading/status indicators during crawl operations
• Extend the Zod schema in settings.tsx if form-level validation is needed

Task 5: Integrate Enterprise Tab into Settings

Wire the new enterprise tab component into the existing settings infrastructure.

• Import EnterpriseForm component in components/settings/components/settings.tsx
• Add "Enterprise" tab trigger to the Tabs.List (update grid to accommodate 5 tabs)
• Add corresponding Tabs.Content section rendering EnterpriseForm
• Fetch enterprise websites in the useEffect data loading block alongside existing settings

🤖 Prompt for AI agents

This phase builds the ingestion pipeline and the settings UI for managing
enterprise websites. It involves creating utility files, server actions, and a
new settings tab component.

**Utility: `lib/utils/chunking.ts`**
- Export `chunkText(content, options)` function
- Implement paragraph-aware splitting that respects sentence boundaries
- Target ~500 tokens per chunk with configurable overlap (~50 tokens)
- Return an array of chunks with their index position

**Utility: `lib/utils/embeddings.ts`**
- Export `generateEmbeddings(texts: string[])` function
- Use OpenAI's `text-embedding-ada-002` model via the Node SDK
- Support batch processing (up to 2048 inputs per request)
- Return array of 1536-dimension float arrays in the same order as inputs

**Server actions: `lib/actions/enterprise.ts`**
- Implement `getEnterpriseWebsites(userId)`, `addEnterpriseWebsite(userId, url,
name)`, `removeEnterpriseWebsite(websiteId)`,
`refreshEnterpriseWebsite(websiteId)`
- Implement `crawlAndIndexWebsite(websiteId)` that: fetches content via Jina AI
(matching the `retrieve` tool pattern), chunks text using `chunkText`, generates
embeddings using `generateEmbeddings`, and inserts records into
`enterpriseDocuments`
- `addEnterpriseWebsite` should automatically trigger `crawlAndIndexWebsite`
after inserting the website record
- On failure, update the website's `status` to `"failed"` with appropriate error
messaging

**UI Component: `components/settings/components/enterprise-form.tsx`**
- Follow existing form component patterns in `components/settings/components/`
- Display a table of configured websites: name, URL, status, last crawled
timestamp
- Include an "Add Website" form with URL and display name inputs
- Per-row actions: "Refresh" (re-crawl) and "Remove" (with confirmation)
- Show loading/status indicators during crawl operations
- Extend Zod schema in `settings.tsx` for form-level validation if needed

**Settings integration: `components/settings/components/settings.tsx`**
- Import `EnterpriseForm` and add an "Enterprise" tab trigger to `Tabs.List`
(update grid columns for 5 tabs)
- Add a `Tabs.Content` section rendering `EnterpriseForm`
- Fetch enterprise websites in the `useEffect` data loading block alongside
existing settings data

Phase 3: Agent Integration

Connect the enterprise knowledge base to the AI assistant so it can search and reference enterprise content during conversations.

Task 1: Create Enterprise Search Tool

Implement a new agent tool that performs semantic search against the user's enterprise documents.

• Create lib/schema/enterprise-search.ts with Zod schema accepting query (required) and optional websiteFilter (URL substring to scope search)
• Create lib/agents/tools/enterprise-search.tsx following the existing tool factory pattern
• Implement execute function that: generates embedding for the query, performs pgvector cosine similarity search against enterpriseDocuments for the current user, returns top 5 most relevant chunks with source URL and title
• Render results using a new EnterpriseSearchSection component (similar to RetrieveSection) showing sources with expandable content previews

Task 2: Register Enterprise Search Tool

Add the new tool to the tool registry so the researcher agent can invoke it.

• Import enterpriseSearchTool in lib/agents/tools/index.tsx
• Add to the tools object in getTools(), conditionally enabled when user has enterprise websites configured (or always available with graceful "no results" handling)
• Ensure ToolProps provides access to userId for scoping queries to the user's documents

Task 3: Update Default System Prompt

Extend the researcher agent's default system prompt to include guidance on when to use the enterprise search tool.

• Modify getDefaultSystemPrompt() in lib/agents/researcher.tsx to add a section for enterprise search
• Add directive: prioritize enterpriseSearch for queries about the user's business, internal knowledge, company-specific topics, or when user explicitly references their websites
• Position this guidance alongside existing tool selection rules (after geospatial, before general web search)
• Clarify the tool's purpose: searching the user's pre-indexed business websites, not live web search

🤖 Prompt for AI agents

This phase wires the enterprise knowledge base into the AI agent by creating a
new search tool, registering it, and updating the system prompt.

**Zod schema: `lib/schema/enterprise-search.ts`**
- Define schema with `query` (required string) and optional `websiteFilter` (URL
substring string to scope the search)

**Tool: `lib/agents/tools/enterprise-search.tsx`**
- Follow the existing tool factory pattern used by other tools in
`lib/agents/tools/`
- In the execute function:
  - Generate an embedding for the `query` using `generateEmbeddings`
- Perform a pgvector cosine similarity search against `enterpriseDocuments`
filtered by the current user's `userId`
  - Apply `websiteFilter` as a URL substring filter if provided
  - Return the top 5 most relevant chunks with source URL and title
- Render results using a new `EnterpriseSearchSection` component (modeled after
`RetrieveSection`) that shows sources with expandable content previews

**Tool registry: `lib/agents/tools/index.tsx`**
- Import `enterpriseSearchTool` and add it to the `tools` object inside
`getTools()`
- The tool can be always available with graceful "no results" handling, or
conditionally enabled when the user has enterprise websites configured
- Ensure `ToolProps` passes `userId` so the tool can scope queries to the
current user's documents

**System prompt: `lib/agents/researcher.tsx`**
- Modify `getDefaultSystemPrompt()` to add a section on enterprise search usage
- Add a directive to prioritize `enterpriseSearch` for queries about the user's
business, internal knowledge, company-specific topics, or when the user
explicitly references their websites
- Position this guidance after geospatial tool rules and before general web
search rules
- Clarify the tool searches the user's pre-indexed business websites, not the
live web

Research

The application is a Next.js AI assistant with a modular agent architecture. Tools are registered via a getTools() factory pattern in lib/agents/tools/, each providing a Zod schema, description, and async execute function. User settings are managed through a tabbed UI in components/settings/ backed by server actions in lib/actions/. The database uses Drizzle ORM with PostgreSQL/Supabase, and already has customType definitions for vector(1536) (pgvector) and PostGIS geometry. URL content fetching exists via the retrieve tool using Jina AI's Reader API. The system prompt can be customized per-user and is injected into the researcher agent, though a custom prompt fully replaces the default rather than extending it.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

This phase establishes the database schema and migration for enterprise website
indexing. The goal is to add two new Drizzle ORM tables and a corresponding
migration with the necessary indexes.

**Schema changes in `lib/db/schema.ts`:**
- Add `enterpriseWebsites` table with columns: `id`, `userId` (FK to users),
`url`, `name`, `status` (enum: `pending`/`indexed`/`failed`), `lastCrawledAt`,
`createdAt`
- Add `enterpriseDocuments` table with columns: `id`, `websiteId` (FK to
`enterpriseWebsites`), `userId` (FK to users), `title`, `content`, `url`,
`embedding` (use the existing `vector` custom type for 1536 dimensions),
`chunkIndex`, `createdAt`
- Export both tables and add type inference exports following the existing
patterns in the file

**Migration in `drizzle/migrations/`:**
- Run `drizzle-kit generate` to produce the base migration SQL
- Prepend `CREATE EXTENSION IF NOT EXISTS vector;` to the migration for
non-Supabase environments
- Add an HNSW vector index on `enterprise_documents.embedding` using
`vector_cosine_ops` for cosine similarity search
- Add standard B-tree indexes on `user_id` and `website_id` foreign key columns
- Follow the `statement-breakpoint` convention used in existing migrations
===============================================================================

Task: 2

This phase builds the ingestion pipeline and the settings UI for managing
enterprise websites. It involves creating utility files, server actions, and a
new settings tab component.

**Utility: `lib/utils/chunking.ts`**
- Export `chunkText(content, options)` function
- Implement paragraph-aware splitting that respects sentence boundaries
- Target ~500 tokens per chunk with configurable overlap (~50 tokens)
- Return an array of chunks with their index position

**Utility: `lib/utils/embeddings.ts`**
- Export `generateEmbeddings(texts: string[])` function
- Use OpenAI's `text-embedding-ada-002` model via the Node SDK
- Support batch processing (up to 2048 inputs per request)
- Return array of 1536-dimension float arrays in the same order as inputs

**Server actions: `lib/actions/enterprise.ts`**
- Implement `getEnterpriseWebsites(userId)`, `addEnterpriseWebsite(userId, url,
name)`, `removeEnterpriseWebsite(websiteId)`,
`refreshEnterpriseWebsite(websiteId)`
- Implement `crawlAndIndexWebsite(websiteId)` that: fetches content via Jina AI
(matching the `retrieve` tool pattern), chunks text using `chunkText`, generates
embeddings using `generateEmbeddings`, and inserts records into
`enterpriseDocuments`
- `addEnterpriseWebsite` should automatically trigger `crawlAndIndexWebsite`
after inserting the website record
- On failure, update the website's `status` to `"failed"` with appropriate error
messaging

**UI Component: `components/settings/components/enterprise-form.tsx`**
- Follow existing form component patterns in `components/settings/components/`
- Display a table of configured websites: name, URL, status, last crawled
timestamp
- Include an "Add Website" form with URL and display name inputs
- Per-row actions: "Refresh" (re-crawl) and "Remove" (with confirmation)
- Show loading/status indicators during crawl operations
- Extend Zod schema in `settings.tsx` for form-level validation if needed

**Settings integration: `components/settings/components/settings.tsx`**
- Import `EnterpriseForm` and add an "Enterprise" tab trigger to `Tabs.List`
(update grid columns for 5 tabs)
- Add a `Tabs.Content` section rendering `EnterpriseForm`
- Fetch enterprise websites in the `useEffect` data loading block alongside
existing settings data
===============================================================================

Task: 3

This phase wires the enterprise knowledge base into the AI agent by creating a
new search tool, registering it, and updating the system prompt.

**Zod schema: `lib/schema/enterprise-search.ts`**
- Define schema with `query` (required string) and optional `websiteFilter` (URL
substring string to scope the search)

**Tool: `lib/agents/tools/enterprise-search.tsx`**
- Follow the existing tool factory pattern used by other tools in
`lib/agents/tools/`
- In the execute function:
  - Generate an embedding for the `query` using `generateEmbeddings`
- Perform a pgvector cosine similarity search against `enterpriseDocuments`
filtered by the current user's `userId`
  - Apply `websiteFilter` as a URL substring filter if provided
  - Return the top 5 most relevant chunks with source URL and title
- Render results using a new `EnterpriseSearchSection` component (modeled after
`RetrieveSection`) that shows sources with expandable content previews

**Tool registry: `lib/agents/tools/index.tsx`**
- Import `enterpriseSearchTool` and add it to the `tools` object inside
`getTools()`
- The tool can be always available with graceful "no results" handling, or
conditionally enabled when the user has enterprise websites configured
- Ensure `ToolProps` passes `userId` so the tool can scope queries to the
current user's documents

**System prompt: `lib/agents/researcher.tsx`**
- Modify `getDefaultSystemPrompt()` to add a section on enterprise search usage
- Add a directive to prioritize `enterpriseSearch` for queries about the user's
business, internal knowledge, company-specific topics, or when the user
explicitly references their websites
- Position this guidance after geospatial tool rules and before general web
search rules
- Clarify the tool searches the user's pre-indexed business websites, not the
live web

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!