Conversation
…extraction Related to #157
P0 fixes: - Replace unbounded daemon threads with ThreadPoolExecutor(max_workers=2) for embedding writes; atexit + close() ensure graceful shutdown so in-flight embeddings are not lost on process exit - Restore apprise>=1.9.0 and feedgen>=1.0.0 removed in epic branch, which would have broken Epic #179 push/RSS features on merge P1 fixes: - _escape_fts: replace double-quote-only escaping with a whitelist regex [A-Za-z0-9_+-] so FTS5 operators (*, NEAR, NOT, ^) cannot alter query semantics; empty queries now short-circuit to [] - _hybrid_merge: skip items with None/invalid id instead of defaulting to id=0, preventing silent score collisions across unrelated records - ReproExperienceStore: add application-level dedup check + UNIQUE constraint (paper_id, pattern_type, content) + IntegrityError fallback to prevent duplicate experiences from accumulating across retries Migration: - 0022_repro_experience_dedup: adds uq_repro_exp_paper_type_content unique constraint to existing repro_code_experience table Tests: 47 unit tests pass (+3 new tests covering the fixes above)
…o pipeline Add user-scoped isolation for repro_code_experience and enforce dedup semantics with migration 0022_repro_experience_dedup. Inject ReproExperienceStore through ReproAgent/Orchestrator/CodingAgent/GenerationNode and propagate user_id/pack_id in generation, verification, and debugging persistence paths. Update /api/gen-code to accept user_id and extend unit tests for user isolation and persistence behavior.
Add decay-aware scoring that combines relevance (confidence), recency (exponential decay with 90-day half-life), and usage frequency to re-rank search results. New memories default to expires_at = created_at + 365 days. search_memories() now auto-touches usage on hits. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add search_memories_batch() that queries multiple scope_ids in a single SQL call, eliminating the N+1 loop in build_context_pack(). Engine now uses this batch method for cross-track memory retrieval. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Refactor build_context_pack() into 4 layer methods: - Layer 0: user profile (cached with 5-min TTL, ~200 tokens) - Layer 1: track context — tasks/milestones (~500 tokens) - Layer 2: query-relevant + cross-track memories (~1000 tokens) - Layer 3: paper-scoped memories (on-demand) Return value adds context_layers metadata while remaining fully backward-compatible. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use standard ln(2)/halfLifeDays lambda formula (matches OpenClaw temporal-decay.ts:toDecayLambda) - Default half-life lowered to 30 days (was 90, now matches OpenClaw) - Evergreen memories (global scope, preference kind) are immune to recency decay (inspired by OpenClaw isEvergreenMemoryPath) - Add _to_decay_lambda() and _is_evergreen_memory() helpers - Expand tests for lambda math, half-life precision, and evergreen logic Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
# Conflicts: # .github/workflows/ci.yml
…uite # Conflicts: # .github/workflows/ci.yml # evals/memory/__init__.py
# Conflicts: # evals/memory/__init__.py # src/paperbot/memory/eval/__init__.py
* docs: update README with macOS python3 tips * docs: update README with more badges * docs: update README with deleting extra badges * docs: update README with new god name --------- Co-authored-by: 林杰 <linjie@linjiedeMacBook-Air.local> Co-authored-by: 林杰 <linjie@v8d1ef64c.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef43e.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef6c5.dip.tu-dresden.de>
* docs: update README with macOS python3 tips * docs: update README with more badges * docs: update README with deleting extra badges * docs: update README with new god name --------- Co-authored-by: 林杰 <linjie@linjiedeMacBook-Air.local> Co-authored-by: 林杰 <linjie@v8d1ef64c.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef43e.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef6c5.dip.tu-dresden.de>
Add comprehensive memory module evaluation aligned with LongMemEval (ICLR 2025), LoCoMo (ACL 2024), Mem0, and Letta benchmarks. - Retrieval Bench v2: IR metrics (Recall@5=0.873, MRR@10=0.731, nDCG@10=0.747) with 40 annotated queries across 5 question types and 5 memory dimensions - Scope Isolation + CRUD: zero-leak verification across user x scope matrix, Mem0-aligned CRUD lifecycle (add/update/delete/dedup) - Context Extraction: L0-L3 layer completeness, precision, token budget guard, TrackRouter accuracy (100%), graceful degradation - Injection Robustness L1: offline pattern detection (0% pollution, 0% FP) - Fixture dataset: 45 memories (2 users), 12 injection patterns - Testing documentation with methodology, validity analysis, and results Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add evaluation results section with metrics from 4 bench suites: - Retrieval quality (Recall@5=0.873, MRR@10=0.731, nDCG@10=0.747) - Scope isolation + CRUD lifecycle (zero leaks, all CRUD pass) - Context extraction (100% precision, 100% router accuracy) - Injection robustness (0% pollution, 0% false positive) Includes LoCoMo question-type breakdown and run instructions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Caution Review failedPull request was closed or merged during review 📝 WalkthroughWalkthroughThis pull request introduces the Paper-to-Context (P2C) module for extracting reproducible context from research papers, adds comprehensive MemoryBench evaluation infrastructure with retrieval/scope/injection robustness testing, integrates vector search and FTS5 into memory storage, and expands repro pipeline with runtime verification and LLM-backed agents. Changes
Sequence Diagram(s)sequenceDiagram
participant Paper as 📄 Paper Input
participant P2C as 🔄 P2C Orchestrator
participant StageA as Stage A:<br/>Literature Distill
participant StageB as Stage B-D:<br/>Blueprint/Env/Params
participant StageE as Stage E:<br/>Roadmap Planning
participant StageF as Stage F:<br/>Success Criteria
participant CtxEngine as Context Engine<br/>(Memory+Research)
participant Assembly as 🏗️ Context Assembly<br/>(Validation+Evidence)
participant Storage as 💾 Local Storage<br/>(Pack+Evidence)
Paper->>P2C: paper_context (title, abstract, sections)
activate P2C
P2C->>CtxEngine: build_context_pack(user_id, paper_id)
activate CtxEngine
CtxEngine->>CtxEngine: L0: user_profile (TTL cache)<br/>L1: track_context<br/>L2: cross_track_memories<br/>L3: paper_memories
CtxEngine-->>P2C: context_pack (user_memory, project_context)
deactivate CtxEngine
P2C->>StageA: literature_distill(input + user_memory)
activate StageA
StageA-->>P2C: objective, key_insights
deactivate StageA
P2C->>StageB: blueprint + environment + hyperparams
activate StageB
StageB-->>P2C: extracted_blueprint, env_setup, methods
deactivate StageB
P2C->>StageE: roadmap_planning(project_context)
activate StageE
StageE-->>P2C: task_roadmap, milestones
deactivate StageE
P2C->>StageF: success_criteria extraction
activate StageF
StageF-->>P2C: results, verification_steps
deactivate StageF
P2C->>Assembly: assemble_pack(all_stages)
activate Assembly
Assembly->>Assembly: JSON schema validation<br/>Evidence linking<br/>Confidence scoring
Assembly-->>P2C: validated_pack
deactivate Assembly
P2C->>Storage: persist_pack + evidence
activate Storage
Storage->>Storage: Store repro_context_pack<br/>Store stage_results<br/>Store evidence_links
Storage-->>P2C: pack_id, session_id
deactivate Storage
P2C-->>Paper: ReproContextPack<br/>(ready for Studio)
deactivate P2C
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested reviewers
Poem
✨ Finishing Touches
🧪 Generate unit tests (beta)
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the project's memory and code reproduction capabilities by integrating a comprehensive benchmarking suite, introducing advanced search and context management features, and enabling persistence of code generation experiences. These changes aim to improve the quality, reliability, and efficiency of the AI-powered research workflow, making the system more robust and intelligent in handling complex tasks. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Ignored Files
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Pull request overview
This PR significantly expands PaperBot’s offline evaluation/benchmarking tooling (MemoryBench + retrieval/context benchmarks) and adds supporting runtime, persistence, and UI error-handling changes across the repro + memory stack, along with extensive documentation updates.
Changes:
- Adds offline benchmark suites (retrieval, context engine, injection robustness, memory performance, memory effectiveness, ROI) with fixtures, scripts, and CI smoke gates.
- Introduces repro experience persistence (DB model + store + CodeMemory integration), verification runtime preparation, and P2C context-bridge enrichment.
- Improves frontend error reporting by mapping upstream backend failures to friendlier UI messages.
Reviewed changes
Copilot reviewed 97 out of 98 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| web/src/components/research/ResearchPageNew.tsx | Adds friendly error parsing/mapping and safer error string handling. |
| web/src/components/research/ResearchDiscoveryPage.tsx | Adds friendly error parsing/mapping and safer error string handling. |
| web/src/components/research/ResearchDashboard.tsx | Adds friendly error parsing/mapping and safer error string handling. |
| tests/unit/test_verification_runtime.py | Unit tests for cached verification runtime preparation + requirements normalization. |
| tests/unit/test_verification_error_classification.py | Verifies “cannot import name …” is classified as LOGIC errors. |
| tests/unit/test_roi_benchmark.py | Unit tests for ROI benchmark seeding, reporting, and significance behavior. |
| tests/unit/test_retrieval_benchmark.py | Unit tests for retrieval benchmark case parsing and metrics aggregation. |
| tests/unit/test_repro_code_experience.py | Tests for ReproExperienceStore CRUD + CodeMemory persistence hooks. |
| tests/unit/test_openai_provider_fallback.py | Tests for OpenAI provider retry behavior and system-role fallback. |
| tests/unit/test_memory_performance_benchmark.py | Smoke tests for memory performance benchmark runner. |
| tests/unit/test_memory_metric_collector.py | Adds unit coverage for new scope isolation metrics recording. |
| tests/unit/test_memory_fts5.py | Adds FTS5 behavior tests (setup/sync/search/fallback paths). |
| tests/unit/test_memory_embedding.py | Adds embedding packing + hybrid search merge tests (vec + FTS5). |
| tests/unit/test_memory_effectiveness_benchmark.py | Tests multi-session effectiveness benchmark + heuristic answer scoring. |
| tests/unit/test_memory_decay.py | Tests memory decay scoring, ranking, expiry, and auto-touch usage behavior. |
| tests/unit/test_memory_batch_search.py | Tests cross-scope batch search behavior and parameter plumbing. |
| tests/unit/test_llm_service.py | Extends cost estimation tests for openrouter-prefixed OpenAI models. |
| tests/unit/test_injection_guard.py | Unit tests for offline injection pattern detection/normalization. |
| tests/unit/test_generation_node_llm.py | Ensures generation node uses injected memory context and template behavior. |
| tests/unit/test_embeddings_fallback.py | Tests embedding provider chain fallback (openai→hash→none). |
| tests/unit/test_context_layers.py | Tests ContextEngine layer caching, TTL behavior, and layer assembly. |
| tests/unit/test_context_engine_benchmark.py | Tests context-engine benchmark fixture parsing, scoring, and runner. |
| src/paperbot/repro/orchestrator.py | Adds config knobs + wiring for experience store, LLM client, and verification runtime settings. |
| src/paperbot/repro/nodes/planning_node.py | Routes planning through injected llm_client with Claude SDK fallback. |
| src/paperbot/repro/memory/code_memory.py | Adds DB-backed experience load/record hooks and injects prior experience into context. |
| src/paperbot/repro/agents/verification_agent.py | Adds cached runtime preparation + runs verification using prepared python executable. |
| src/paperbot/repro/agents/planning_agent.py | Passes llm_client into PlanningNode. |
| src/paperbot/repro/agents/coding_agent.py | Passes llm_client/experience_store into nodes and propagates user_id/pack_id. |
| src/paperbot/memory/eval/performance_benchmark.py | Adds offline memory performance benchmark runner and seeding helpers. |
| src/paperbot/memory/eval/injection_guard.py | Adds offline injection detection rules + normalization and safe-marker allowlist. |
| src/paperbot/memory/eval/collector.py | Adds cross-user/cross-scope leak metrics and target evaluation. |
| src/paperbot/memory/eval/init.py | Re-exports MemoryBench evaluation/benchmark APIs. |
| src/paperbot/infrastructure/stores/repro_experience_store.py | Adds SQLAlchemy store for persisted repro code experiences with dedup behavior. |
| src/paperbot/infrastructure/stores/models.py | Adds embedding blob column + ReproCodeExperienceModel with unique index. |
| src/paperbot/infrastructure/llm/providers/openai_provider.py | Adds retries/backoff and fallback when system-role is rejected by provider. |
| src/paperbot/context_engine/embeddings.py | Adds HashEmbeddingProvider + env-configurable provider chain. |
| src/paperbot/application/services/p2c/stages.py | Adds user_memory/project_context plumbing + narrows JSON decode exceptions. |
| src/paperbot/application/services/p2c/prompts.py | Adds tag-delimited context blocks with sanitization to mitigate tag-escape injection. |
| src/paperbot/application/services/p2c/orchestrator.py | Enriches normalized input via ContextEngineBridge; improves LLMService import handling. |
| src/paperbot/application/services/p2c/models.py | Extends NormalizedInput with user_memory/project_context strings. |
| src/paperbot/application/services/p2c/context_bridge.py | Adds ContextEngineBridge to inject memory/project context into P2C prompts. |
| src/paperbot/application/services/llm_service.py | Updates model cost estimation to handle provider-prefixed model names. |
| src/paperbot/api/routes/repro_context.py | Persists P2C observations into paper-scoped memory for reuse. |
| src/paperbot/api/routes/gen_code.py | Adds user_id to gen-code request and passes through to agent. |
| scripts/eval_search.py | CLI for offline retrieval benchmark with thresholds/output. |
| scripts/eval_context_engine.py | CLI for offline context-engine benchmark with thresholds/output. |
| scripts/benchmark_memory_performance.py | CLI for memory performance benchmark baseline runs. |
| requirements.txt | Adds optional sqlite-vec dependency for vector search. |
| evals/runners/run_retrieval_benchmark_smoke.py | Adds CI smoke gate for retrieval benchmark thresholds. |
| evals/runners/run_context_engine_benchmark_smoke.py | Adds CI smoke gate for context engine benchmark thresholds. |
| evals/memory/test_injection_robustness.py | Adds offline injection robustness test + metrics recording. |
| evals/memory/fixtures/roi_cases.json | Fixture for ROI benchmark paper cases. |
| evals/memory/fixtures/repro_experiences.json | Fixture for seeded repro experiences used in ROI benchmark. |
| evals/memory/fixtures/injection_patterns.json | Fixture for malicious/benign injection detection patterns. |
| evals/memory/fixtures/bench_v2/bench_memories.json | MemoryBench dataset fixture (multi-user/multi-scope memories). |
| evals/memory/bench_roi.py | Manual ROI benchmark runner CLI. |
| evals/memory/bench_effectiveness.py | Manual effectiveness benchmark runner CLI. |
| evals/memory/init.py | Updates evals module docstring to include new suites. |
| evals/fixtures/retrieval/bench_v2.jsonl | Deterministic offline retrieval benchmark fixture cases. |
| evals/README.md | Notes retrieval regression coverage in evals workflow. |
| docs/search_eval.md | Documentation for offline retrieval benchmark schema and usage. |
| docs/p2c/P2C_ROI_BENCHMARK.md | Documentation for ROI benchmark purpose, fixtures, and usage. |
| docs/p2c/P2C_MODULE_1_CORE_ENGINE.md | Updates docs to reflect user_memory/project_context now formatted as strings. |
| docs/memory_performance_eval.md | Documentation for memory performance baseline harness. |
| docs/memory_effectiveness_eval.md | Documentation for multi-session memory effectiveness benchmark. |
| docs/context_engine_eval.md | Documentation for context engine benchmark fixture + metrics. |
| docs/benchmark/MEMORYBENCH_EPIC_283_COMPLETION.md | Epic completion report for MemoryBench suite rollout. |
| alembic/versions/0022_repro_experience_dedup.py | Migration adding user_id + unique constraint for repro experiences. |
| alembic/versions/0021_repro_code_experience.py | Migration creating repro_code_experience table. |
| alembic/versions/0020_memory_embedding.py | Migration adding embedding column + sqlite-vec virtual table/triggers (best-effort). |
| alembic/versions/0019_memory_fts5.py | Migration creating FTS5 virtual table + triggers (SQLite-only). |
| README.md | Adds MemoryBench results section and updates header/badges. |
| Makefile | Adds bench-roi target with API key guard and default output path. |
| .github/workflows/ci.yml | Adds retrieval benchmark smoke and memory safety evals to CI. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| <img src="https://img.shields.io/badge/Python-3.10+-blue?logo=python&logoColor=white" alt="Python"> | ||
| <img src="https://img.shields.io/badge/Next.js-16-black?logo=nextdotjs" alt="Next.js"> | ||
| <img src="https://img.shields.io/badge/Platform-Win%20|%20Mac%20|%20Linux-lightgrey" alt="Platform"> | ||
| <img src="https://img.shields.io/badge/Downloads-xx-lightgrey?style=social&logo=icloud" alt="Downloads"> |
There was a problem hiding this comment.
The Downloads badge currently uses a placeholder value ("xx"), which will render misleading information in the public README. Either wire this to a real metric (e.g., GitHub releases/downloads) or remove the badge until a real value is available.
Suggested change
| <img src="https://img.shields.io/badge/Downloads-xx-lightgrey?style=social&logo=icloud" alt="Downloads"> | |
| <img src="https://img.shields.io/github/downloads/jerry609/PaperBot/total?style=social&logo=icloud" alt="GitHub all releases"> |
Comment on lines
+200
to
+206
| ## MemoryBench Evaluation | ||
|
|
||
| > Aligned with [LongMemEval](https://arxiv.org/abs/2407.15460) (ICLR 2025), [LoCoMo](https://arxiv.org/abs/2402.09281) (ACL 2024), Mem0, Letta. Full methodology: [`evals/memory/README.md`](evals/memory/README.md) · Epic [#283](https://github.com/jerry609/PaperBot/issues/283) | ||
|
|
||
| <details open> | ||
| <summary><b>Retrieval Quality</b> — 40 queries, 45 memories, 2 users (FTS5 + BM25)</summary> | ||
|
|
There was a problem hiding this comment.
PR metadata says this is “docs only”, but this PR includes substantial code, test, dependency, and CI changes (e.g., MemoryBench runner additions, new stores/models, workflow updates). Please update the PR title/description to match the actual scope so reviewers can triage appropriately.
Comment on lines
+49
to
+77
| type UpstreamErrorBody = { | ||
| detail?: string | ||
| error?: string | ||
| } | ||
|
|
||
| function toFriendlyErrorMessage(status: number, rawText: string): string | null { | ||
| if (!rawText) return null | ||
|
|
||
| let parsed: UpstreamErrorBody | null = null | ||
| try { | ||
| parsed = JSON.parse(rawText) as UpstreamErrorBody | ||
| } catch { | ||
| parsed = null | ||
| } | ||
|
|
||
| const detail = parsed && typeof parsed.detail === "string" ? parsed.detail : undefined | ||
|
|
||
| if (detail && (detail.includes("Upstream API unreachable") || detail.includes("Upstream API timed out"))) { | ||
| if (detail.includes("timed out")) { | ||
| return "Unable to connect to service (request timed out). Please ensure the backend is running. Please try again." | ||
| } | ||
| return "Unable to connect to service. Please ensure the backend is running." | ||
| } | ||
|
|
||
| // Fallback: surface backend-provided detail when available | ||
| if (detail) return detail | ||
|
|
||
| return null | ||
| } |
There was a problem hiding this comment.
UpstreamErrorBody / toFriendlyErrorMessage() are duplicated here and in other research pages. This duplication increases the chance of the error-mapping logic drifting over time. Consider extracting a shared helper (e.g., web/src/lib/httpErrors.ts) and importing it from each page.
Comment on lines
+17
to
+45
| type UpstreamErrorBody = { | ||
| detail?: string | ||
| error?: string | ||
| } | ||
|
|
||
| function toFriendlyErrorMessage(status: number, rawText: string): string | null { | ||
| if (!rawText) return null | ||
|
|
||
| let parsed: UpstreamErrorBody | null = null | ||
| try { | ||
| parsed = JSON.parse(rawText) as UpstreamErrorBody | ||
| } catch { | ||
| parsed = null | ||
| } | ||
|
|
||
| const detail = parsed && typeof parsed.detail === "string" ? parsed.detail : undefined | ||
|
|
||
| if (detail && (detail.includes("Upstream API unreachable") || detail.includes("Upstream API timed out"))) { | ||
| if (detail.includes("timed out")) { | ||
| return "Unable to connect to service (request timed out). Please ensure the backend is running. Please try again." | ||
| } | ||
| return "Unable to connect to service. Please ensure the backend is running." | ||
| } | ||
|
|
||
| // Fallback: surface backend-provided detail when available | ||
| if (detail) return detail | ||
|
|
||
| return null | ||
| } |
There was a problem hiding this comment.
UpstreamErrorBody / toFriendlyErrorMessage() are duplicated here and in other research pages. This duplication increases maintenance cost and makes it easy for behavior to diverge. Consider centralizing this mapping in a shared utility and reusing it across pages.
Comment on lines
+93
to
+121
| type UpstreamErrorBody = { | ||
| detail?: string | ||
| error?: string | ||
| } | ||
|
|
||
| function toFriendlyErrorMessage(status: number, rawText: string): string | null { | ||
| if (!rawText) return null | ||
|
|
||
| let parsed: UpstreamErrorBody | null = null | ||
| try { | ||
| parsed = JSON.parse(rawText) as UpstreamErrorBody | ||
| } catch { | ||
| parsed = null | ||
| } | ||
|
|
||
| const detail = parsed && typeof parsed.detail === "string" ? parsed.detail : undefined | ||
|
|
||
| if (detail && (detail.includes("Upstream API unreachable") || detail.includes("Upstream API timed out"))) { | ||
| if (detail.includes("timed out")) { | ||
| return "Unable to connect to service (request timed out). Please ensure the backend is running. Please try again." | ||
| } | ||
| return "Unable to connect to service. Please ensure the backend is running." | ||
| } | ||
|
|
||
| // Fallback: surface backend-provided detail when available | ||
| if (detail) return detail | ||
|
|
||
| return null | ||
| } |
There was a problem hiding this comment.
UpstreamErrorBody / toFriendlyErrorMessage() are duplicated here and in other research components. To avoid divergence and reduce copy/paste updates, consider extracting this into a shared helper module and importing it where needed.
Comment on lines
+125
to
+127
| assert all("alice" not in r.get("user_id", "alice") or True for r in results) | ||
| # Strictly: no result should belong to bob | ||
| assert all(r.get("user_id") == "alice" for r in results) |
There was a problem hiding this comment.
This assertion is effectively a no-op because ... or True always evaluates to True, so it won’t catch cross-user leakage regressions. It should be removed or replaced with a strict check (the following assertion already covers the intended behavior).
Comment on lines
+3
to
7
| <h1 align="center">Oh, God! My idea comes true.</h1> | ||
|
|
||
| <h3 align="center"> | ||
| AI-powered research workflow: paper discovery → LLM analysis → scholar tracking → Paper2Code → multi-agent studio | ||
| </h3> |
There was a problem hiding this comment.
The README’s primary heading was changed from the project name to a slogan. This makes the repo harder to identify/search and conflicts with the repository name/badges. Consider keeping the H1 as “PaperBot” and moving the slogan to a subtitle/tagline instead.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This is a substantial and high-quality pull request that introduces a comprehensive benchmarking suite (MemoryBench) and significantly enhances the memory and code generation systems. The changes include new features like vector search, memory decay, and persistent code experiences, along with major improvements to robustness and error handling. The code is well-structured and the new documentation is thorough. I have a few suggestions for improving readability and future configurability.
Note: Security Review did not run due to the size of the PR.
| .PHONY: bench-roi | ||
|
|
||
| bench-roi: | ||
| @if [ -z "$$OPENAI_API_KEY$$ANTHROPIC_API_KEY$$OPENROUTER_API_KEY$$NVIDIA_MINIMAX_API_KEY$$NVIDIA_GLM_API_KEY" ]; then \ |
There was a problem hiding this comment.
The method of concatenating all API key variables to check if at least one is set is functional but can be difficult to read and debug. For better clarity and maintainability, I suggest checking each variable individually. This makes the condition's intent more explicit.
@if [ -z "$$OPENAI_API_KEY" ] && [ -z "$$ANTHROPIC_API_KEY" ] && [ -z "$$OPENROUTER_API_KEY" ] && [ -z "$$NVIDIA_MINIMAX_API_KEY" ] && [ -z "$$NVIDIA_GLM_API_KEY" ]; then \
| <!-- markdownlint-disable MD001 MD041 --> | ||
|
|
||
| <h1 align="center">PaperBot</h1> | ||
| <h1 align="center">Oh, God! My idea comes true.</h1> |
There was a problem hiding this comment.
The new project name 'Oh, God! My idea comes true.' is unconventional and may not convey a professional image for the project. While this might be a deliberate creative choice, consider reverting to 'PaperBot' or choosing another name that is more descriptive of the project's purpose to aid in its branding and adoption.
Comment on lines
+37
to
+38
| vec_weight: float = 0.6, | ||
| bm25_weight: float = 0.4, |
There was a problem hiding this comment.
The weights for vector search and BM25 search in the hybrid merge are hardcoded. While vec_weight=0.6 and bm25_weight=0.4 are reasonable defaults, making these configurable (e.g., through SqlAlchemyMemoryStore's configuration) would provide more flexibility for tuning retrieval performance in the future.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Database