diff --git a/website/observability/tracing/concept.md b/website/observability/tracing/concept.md deleted file mode 100644 index 338c1c7f4..000000000 --- a/website/observability/tracing/concept.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: Concept ---- - -In VoltOps, a trace is a record of the entire process from a user's request to the system's response. Each trace captures all steps, decisions, and data flow from the beginning to the end of a single user interaction. - -:::tip Solves the black-box problem -Traces eliminate the "black box" nature of AI applications by providing complete transparency and enabling you to manage your systems with confidence. -::: - -VoltOps provides **n8n-style observability** with a visual canvas interface that makes complex AI workflows intuitive to understand. - -:::note The n8n-inspired canvas visualization offers several key advantages - -- **Visual Workflow Mapping**: See your entire AI workflow as connected nodes, making it easy to understand data flow and decision points -- **Intuitive Debugging**: Quickly identify where issues occur by visually following the execution path through your AI agents and tools -- **Non-Technical Accessibility**: Enable stakeholders without deep technical knowledge to understand and analyze AI system behavior -- **Real-time Monitoring**: Watch your AI workflows execute in real-time with live status updates on the visual canvas -- **Simplified Complexity**: Transform complex, multi-step AI processes into clear, digestible visual representations - ::: - -## Core Trace Features - -
- -```mermaid -%%{init: {'theme':'base', 'themeVariables': {'primaryColor':'#10b981', 'primaryTextColor':'#ffffff', 'primaryBorderColor':'#10b981', 'lineColor':'#10b981', 'sectionBkgColor':'#10b981', 'altSectionBkgColor':'#059669', 'gridColor':'#10b981', 'secondaryColor':'#10b981', 'tertiaryColor':'#ffffff'}}}%% -sequenceDiagram - participant User - participant App as LLM Application - participant VoltOps as VoltOps Observability - participant LLM as LLM/Tools/DB - - User->>App: Send Request - App->>VoltOps: Initialize Trace - - App->>LLM: AI Operations (Prompts, Tools, Memory) - VoltOps->>VoltOps: Track: Tokens, Cost, Latency, Status - LLM->>App: Responses & Data - - App->>User: Send Response - App->>VoltOps: Complete Trace - - Note over VoltOps: Real-time Analytics & Monitoring -``` - -
- -**Unique Session Identity:** -Each trace is identified with a unique session ID and maintains complete context throughout the interaction lifecycle. - -**Flexible Organization:** -Traces support tagging and metadata for easy categorization and filtering across your AI workflows. - -**Real-time Visibility:** -VoltOps captures and visualizes traces in real-time, enabling instant monitoring and issue detection. - -**Hierarchical Structure:** -Agents, tools, memory operations, and retrievers are organized in meaningful hierarchies that reflect your AI application's logic. - -**Complete Lifecycle Tracking:** -From start to finish, traces monitor status changes, performance metrics, and outcomes. - -
- - **Information Captured in Traces** - - -### Session and Context Data - -- **User Identity**: User ID, profile information, and preferences -- **Conversation Context**: Conversation ID, history references, and session state -- **Input/Output Flow**: Original requests, system responses, and data transformations -- **Custom Metadata**: Priority levels, source channels, departments, and domain-specific information - -### Performance and Cost Metrics - -- **Token Usage**: Prompt tokens, completion tokens, and total consumption -- **Timing Data**: Processing times, response latency, and execution duration -- **Cost Analysis**: API call costs and resource consumption - -### Error and Status Information - -- **Status Tracking**: Running, completed, or error states with timestamps -- **Error Details**: Messages, codes, and failure stages -- **Success Metrics**: Completion rates, confidence scores, and quality indicators -- **Audit Trail**: Decision points and state transitions - -
- -## VoltOps vs General Observability Concepts - -VoltOps tracing builds upon established observability concepts but adapts them specifically for AI agent workflows. Understanding how VoltOps concepts map to general observability terms helps you leverage existing knowledge while working with AI-native structures. - -### Concept Mapping - -**VoltOps Trace ≡ General Trace** -Both represent a complete user interaction session from request to response, maintaining unique identifiers, timestamps, and hierarchical organization. - -**VoltOps Agent ≈ Event + Trace** -An agent functions as both a span (tracking execution and hierarchy) and a generation (capturing AI model interactions with parameters, token usage, and responses). - -**VoltOps Tools/Memory/Retrievers ≈ Events** -These operations behave like traditional spans, tracking execution time, success/error states, input/output data, and nested hierarchies. - -**VoltOps Success/Error Events ≈ Discrete Events** -Status changes function as timestamped events marking state transitions and providing audit trails. diff --git a/website/observability/tracing/feedback.md b/website/observability/tracing/feedback.md new file mode 100644 index 000000000..17f8bd793 --- /dev/null +++ b/website/observability/tracing/feedback.md @@ -0,0 +1,23 @@ +--- +title: Trace Feedback +--- + +import MediaPanel from "@site/src/components/docs-widgets/MediaPanel"; + +# Trace Feedback + +Use Feedback to tie human judgment to a specific trace so you can validate fixes and catch regressions early. It turns "this felt wrong" into a trackable signal you can compare across runs. + +What to look at: + +- **History and source**: confirm the trace was rated and where it came from (app, API, model). +- **Key and score**: keep signals consistent (for example, satisfaction) so you can compare runs. +- **Comments**: capture short context that explains why the output was good or bad. + + + +
diff --git a/website/observability/tracing/logs.md b/website/observability/tracing/logs.md new file mode 100644 index 000000000..5d217a227 --- /dev/null +++ b/website/observability/tracing/logs.md @@ -0,0 +1,18 @@ +--- +title: Trace Logs +--- + +# Trace Logs + +Use Logs to review structured events tied to the trace and span context. This is the fastest way to inspect errors and metadata without opening each span. + + + +
+
+ +Logs are best when you need a fast answer to "what happened and why." Start with errors, then use search or level filters to narrow down, and follow the trace/span context to the exact step. diff --git a/website/observability/tracing/node-based.md b/website/observability/tracing/node-based.md new file mode 100644 index 000000000..03b4bd8ed --- /dev/null +++ b/website/observability/tracing/node-based.md @@ -0,0 +1,15 @@ +--- +title: Node-Based View +--- + +# Node-Based View + +The node-based view shows the same trace as a visual flow. It helps you understand the overall path at a glance and see how agents, tools, memory, and model calls connect. + +Use this view when you want the big picture, then drill into a specific node for details. + + diff --git a/website/observability/tracing/overview.md b/website/observability/tracing/overview.md index 358e1c022..484e1fb64 100644 --- a/website/observability/tracing/overview.md +++ b/website/observability/tracing/overview.md @@ -4,7 +4,11 @@ title: Overview # VoltOps Tracing -Tracing is the process of recording and visualizing the execution path of your AI applications in real-time. It shows you exactly what your AI agents are doing, which tools they're using, and how data flows through your system from start to finish. +Tracing shows how a single user request runs from start to finish. As a developer, you use it to answer: + +- Why did the agent respond this way? +- Which step failed? +- Where did latency or cost spike? -## Why Tracing Matters for LLM Applications - -
- -Understanding Complex AI Workflows - - -Large Language Model applications often involve complex, multi-step processes that can be difficult to debug and optimize. Unlike traditional applications with predictable execution paths, LLM apps feature: - -- **Dynamic decision-making**: AI agents make context-dependent choices that vary between runs -- **Multi-step reasoning**: Complex tasks are broken down into multiple sequential or parallel operations -- **Tool integration**: AI agents interact with external APIs, databases, and services -- **Non-deterministic behavior**: The same input can produce different execution paths - -
- -
- -Key Benefits of Tracing - - -**Debug with Confidence** - -- Identify exactly where errors occur in your AI workflow -- Understand why certain decisions were made by your AI agents -- Track the flow of data through complex processing chains -- Pinpoint performance bottlenecks in real-time - -**Monitor Performance** - -- Track response times for each component of your AI system -- Monitor token usage and costs across different LLM calls -- Identify slow operations that impact user experience +
+
-**Optimize Your AI Applications** +When you select a trace from the list, you can open the [Waterfall view](https://voltagent.dev/observability-docs/tracing/waterfall/) or the [Node-Based view](https://voltagent.dev/observability-docs/tracing/node-based/) and then dig into [Logs](https://voltagent.dev/observability-docs/tracing/logs/) and [Feedback](https://voltagent.dev/observability-docs/tracing/feedback/). -- Analyze which tools and prompts perform best -- Identify redundant or inefficient processing steps -- Optimize prompt engineering based on actual execution data -- Fine-tune your AI workflows for better performance +## Trace Filters -**Collaborate Effectively** +Filters help you find the right trace fast. -- Share detailed traces with team members for debugging -- Document AI behavior patterns for future reference -- Enable non-technical stakeholders to understand AI decision-making -- Create reproducible test cases from real execution traces +:::tip +For example use: -
+- Status + Duration to see where a failure started, +- Token usage + Cost to catch expensive runs, and +- User ID + Conversation ID to inspect a specific user flow. +::: -### Common Use Cases +![Trace filters overview](https://cdn.voltagent.dev/docs/observability/tracing/overview-trace-filters.gif) -- **Agent Debugging**: When your AI agent produces unexpected results, tracing shows exactly what happened -- **Performance Optimization**: Identify which LLM calls or tool executions are taking too long -- **Cost Analysis**: Track token usage and API costs across your entire application -- **Quality Assurance**: Verify that your AI workflows behave consistently across different scenarios -- **Compliance**: Maintain audit trails of AI decision-making for regulatory requirements +
-Tracing transforms the black box of AI applications into a transparent, observable system that you can understand, debug, and optimize with confidence. +- **Status**: isolate failed or retry-heavy runs. +- **Agent ID / Entity type**: narrow down which agent or step is involved. +- **Token usage / Cost**: find expensive or abnormal runs. +- **Duration**: spot slow traces and latency outliers. +- **Feedback source / key**: review feedback by source. +- **User ID / Conversation ID**: drill into one user journey. diff --git a/website/observability/tracing/users.md b/website/observability/tracing/users.md new file mode 100644 index 000000000..1b9245bd2 --- /dev/null +++ b/website/observability/tracing/users.md @@ -0,0 +1,50 @@ +--- +title: User Analytics +--- + +# User Analytics + +The Users page groups traces by user so you can see who is impacted and where issues concentrate. It helps you go from "something feels off" to a specific user, session, and trace. + + + +
+
+ +## Key Metrics + +Quick read of volume, success rate, latency, and cost over time. Use it to decide if the problem is quality, performance, or spend, and to link regressions to model switches. + + + +
+ +## User Feedback + +Feedback turns user sentiment into signals you can track. Use it to confirm whether fixes improved the experience and to see which issues users feel most. + + + +
+ +Use the charts to see feedback volume, sentiment trends, score clusters, dominant feedback keys, and when users leave comments. + +## Conversations + +Conversations group traces by session so you can follow a user flow end to end. + +Use the cards (status, time range, model, cost) to pick the most suspicious session, then filter traces to that path. + +![conversation](https://cdn.voltagent.dev/docs/observability/tracing/tracing-conversation-2.png) diff --git a/website/observability/tracing/waterfall.md b/website/observability/tracing/waterfall.md new file mode 100644 index 000000000..04642eb82 --- /dev/null +++ b/website/observability/tracing/waterfall.md @@ -0,0 +1,25 @@ +--- +title: Waterfall View +--- + +# Waterfall View + +Use Waterfall to understand timing and execution order. It shows spans as a hierarchy so you can see where time is spent, how retries fan out, and which step failed first. + + + +
+
+ +Use Waterfall when you need the root cause of a failure or a latency spike. The details panel gives the context you need to confirm tool usage, memory behavior, and LLM settings without digging through logs. + +Key signals to check: + +- **Failure path**: error badges + child spans to see the first break and its causes. +- **Latency hotspots**: duration bars and timing to find the slowest steps. +- **Model context**: input/output and UI vs model messages to verify prompts and responses. +- **Configuration**: LLM config, memory config, and knowledge base usage for this span. diff --git a/website/sidebarsObservability.ts b/website/sidebarsObservability.ts index 455b9202a..44e7b7ee2 100644 --- a/website/sidebarsObservability.ts +++ b/website/sidebarsObservability.ts @@ -28,7 +28,14 @@ const sidebars: SidebarsConfig = { { type: "category", label: "Tracing", - items: ["tracing/overview", "tracing/concept"], + items: [ + "tracing/overview", + "tracing/waterfall", + "tracing/node-based", + "tracing/logs", + "tracing/feedback", + "tracing/users", + ], }, { type: "doc", diff --git a/website/src/components/docs-widgets/MediaPanel.tsx b/website/src/components/docs-widgets/MediaPanel.tsx new file mode 100644 index 000000000..d3d43eb81 --- /dev/null +++ b/website/src/components/docs-widgets/MediaPanel.tsx @@ -0,0 +1,29 @@ +import React from "react"; + +type MediaPanelProps = { + src: string; + alt: string; + width?: number | string; +}; + +const MediaPanel = ({ src, alt, width = "100%" }: MediaPanelProps) => ( +
+
+ {alt} +
+
+); + +export default MediaPanel;