docs(bench): OpenClaw + Dream 500d Executive Assistant results - #4
Open
spqian wants to merge 1 commit into
Open
Conversation
Add a Recall Bench results writeup for the OpenClaw agent loop backed by the Dreamweave nightly consolidation engine, on the Executive Assistant persona over a 500-day corpus (50 checkpoints, 3,164 evals). Headline vs the published plain-vector 500d run: overall composite 5.32 (last-quartile 5.15 vs 4.98), hallucination roughly halved (7.7% avg, 16.2% peak vs 13.6% floor / 27.1% peak), and every category improved. Temporal reasoning remains the weakest category (4.60). Includes the heatmap, per-category quartile tables, hallucination trajectory, an operational note on tier2=50000 ingest cost, and the raw run artifacts under bench-results/openclaw/ea-500d-dream/. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenClaw + Dream — 500-day Executive Assistant results
Per the /dream skill discussion — @Stevenic offered to publish this run in the recall bench. This PR adds the writeup + evidence following the existing
docs/bench/results/convention (mirrorsopenclaw-ea.md).What this is: the same
openclaw[vector:text-embedding-3-small+agent]answer loop and judges, but with the memory backend pointed at the Dreamweave nightly consolidation engine (report→judge→apply merges, supersede/sequence edges, graph recall). It's the natural A/B against the published plain-vector 500d run.Headline (vs plain-vector 500d baseline)
Consolidation roughly halves the hallucination rate and lifts every category; temporal reasoning stays the weakest and is called out as the frontier.
Contents
docs/bench/results/openclaw-dream-ea.md— full writeup (TL;DR, per-category quartiles, hallucination trajectory, findings, reproduce)docs/bench/results/openclaw-dream-ea-500d-heatmap.pngdocs/bench/results/index.md— new rowbench-results/openclaw/ea-500d-dream/— rawresult.json,progress.jsonl,failures.jsonl,heatmap.png(per-question log omitted for size, ~23 MB)Includes an honest fairness note (engine-level comparison, not a single isolated knob) and an operational note on the
tier2=50000ingest-cost artifact.