websense

Agent-friendly web content fetching — fetch a URL and return cleaned, readable text with SSRF guards.

Why

LLM agents frequently need to read the contents of a web page, but raw HTML is noisy and naive fetchers are vulnerable to server-side request forgery (SSRF) against internal services, cloud metadata endpoints, and loopback addresses. websense fetches a URL, validates it (and every redirect hop) against a strict allow/deny model for private and internal addresses, and converts HTML responses into clean, readable plain text — all with zero runtime dependencies.

Install

npm i -g websense

Or run it directly without installing:

npx websense https://example.com

Usage

websense <url> [options]

Options:

Flag	Description	Default
--format <text|html>	Output format: cleaned text or raw HTML	text
--timeout <ms>	Request timeout in milliseconds	15000
--max-bytes <n>	Maximum response bytes to read	2000000
-h, --help	Show usage

Example:

websense https://example.com/docs --format text --timeout 10000

The CLI prints a JSON object (see Output schema) to stdout on success, or {"error": "..."} to stderr with a non-zero exit code on failure.

Library usage

import { fetchUrl } from 'websense';

const result = await fetchUrl('https://example.com');
console.log(result.title, result.text);

export interface FetchResult {
  url: string;
  finalUrl: string;
  status: number;
  contentType: string | null;
  title: string | null;
  text: string;
  bytes: number;
  truncated: boolean;
}

export interface FetchOptions {
  timeoutMs?: number;
  maxBytes?: number;
  userAgent?: string;
  format?: 'text' | 'html';
  allowHosts?: string[];
  denyHosts?: string[];
  maxRedirects?: number;
}

Output schema

See schemas/websense.output.schema.json for the JSON Schema (draft 2020-12) describing the FetchResult object.

Security

websense is designed to be safe to point at arbitrary, untrusted URLs (e.g. URLs supplied by an LLM or an end user):

• Protocol allow-list: only http: and https: URLs are accepted.
• No embedded credentials: URLs containing a username/password (http://user:pass@host) are rejected.
• Blocked hosts: localhost, *.localhost, *.local, and any literal IP address in a private/loopback/link-local/CGNAT/cloud-metadata range (e.g. 127.0.0.1, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.169.254, ::1, fc00::/7, fe80::/10) are rejected before any network request is made.
• DNS-resolution check: hostnames are resolved via DNS, and if any resolved address is private, the request is rejected — this defends against DNS rebinding / DNS-based SSRF.
• Redirect re-validation: each redirect hop is independently re-validated (URL shape + DNS resolution) before being followed, up to a configurable maxRedirects limit.
• Bounded reads: response bodies are streamed and capped at maxBytes, with truncated: true reported if the cap was hit.

You can further restrict allowed destinations with allowHosts / denyHosts in FetchOptions.

License

MIT