feat: harden search and connector infrastructure#339
Merged
Conversation
Implements issue #262 by moving Arxiv/OpenAlex/PapersCool connectors onto the shared async transport with retry support, updating async call sites, and adding focused tests.
Implements issue #267 by replacing per-ID OpenAlex work fetches with batched filter queries and adds focused coverage for the batched path.
Implements issue #266 by routing both the shared mixin and the scholar-tracking agent through the same SemanticScholarClient surface, removing the duplicated legacy API client path, and adding focused unification tests.
Implements issue #263 by moving heartbeat, timeout, cancellation cleanup, and X-Accel-Buffering handling into the shared SSE wrapper and switching the streaming routes onto the common response helper with regression coverage.
Implements issue #268 by adding timeout/max_tries metadata to worker functions, reusing the SqlAlchemyEventLog singleton inside the worker module, and covering both behaviors with focused tests.
Replaces the identity-key-only dedup in PaperSearchService._fuse_with_rrf() with a dedicated PaperDeduplicator that matches across DOI, arxiv_id (version- stripped), and fuzzy title similarity via rapidfuzz. Merged papers accumulate the best metadata (highest citations, longest abstract, union of identities). Closes #317 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extends HashEmbeddingProvider regex to include CJK Unified Ideographs (U+4E00–U+9FFF) and Extension A (U+3400–U+4DBF), enabling proper embedding of Chinese text. Closes #276 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly strengthens the application's backend infrastructure, particularly focusing on external API interactions and data processing. It introduces a robust asynchronous request layer to manage external connector calls more reliably, enhances the stability of Server-Sent Events for real-time communication, and implements a sophisticated paper deduplication system to maintain data integrity. Additionally, it improves embedding capabilities for CJK languages and refines background job handling, leading to a more resilient and accurate system overall. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Pull request overview
This PR hardens PaperBot’s search/connectors and SSE streaming infrastructure by unifying async request handling, improving SSE transport behavior (heartbeats/timeouts/headers), and adding a new three-tier paper deduplication layer used during multi-source result fusion.
Changes:
- Introduces a shared SSE response helper with heartbeats/timeouts and standardized headers, updating all SSE routes to use it.
- Adds a shared async request layer and migrates key connectors/adapters (OpenAlex/arXiv/papers.cool) to async + batched OpenAlex fetching.
- Adds three-tier paper deduplication (DOI/arXiv/fuzzy title) integrated into RRF fusion, plus expanded CJK tokenization for hash embeddings and new regression tests.
Reviewed changes
Copilot reviewed 39 out of 40 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_streaming_envelope.py | Adds heartbeat + timeout + header coverage for SSE wrapper/response. |
| tests/unit/test_semantic_scholar_unification.py | Verifies Semantic Scholar mixin/agent reuse unified client + close behavior. |
| tests/unit/test_retry_helper_async.py | Adds regression tests for new async retry decorator. |
| tests/unit/test_research_discovery_collections_routes.py | Updates OpenAlex connector fakes to async + close. |
| tests/unit/test_request_layer.py | Adds tests for request-layer retry behavior (transport/429/budget). |
| tests/unit/test_paper_dedup.py | Adds broad unit coverage for new three-tier deduplicator. |
| tests/unit/test_openalex_connector.py | Updates OpenAlex tests for async request-layer usage + batched ID fetch. |
| tests/unit/test_embeddings_fallback.py | Adds CJK/Hangul tokenization regression tests for hash embeddings. |
| tests/unit/test_arq_worker_settings.py | Validates bounded ARQ function settings + singleton event-log helper. |
| tests/integration/test_repro_context_routes.py | Ensures SSE responses include X-Accel-Buffering: no. |
| src/paperbot/utils/retry_helper.py | Adds async_retry and extends default retry exception set. |
| src/paperbot/utils/init.py | Re-exports async_retry. |
| src/paperbot/infrastructure/services/api_client.py | Replaces old APIClient implementation with compatibility re-export. |
| src/paperbot/infrastructure/queue/arq_worker.py | Adds singleton event-log helper + bounded ARQ function configs. |
| src/paperbot/infrastructure/crawling/request_layer.py | Adds shared AsyncRequestLayer with retry/backoff/throttle + reusable client. |
| src/paperbot/infrastructure/connectors/paperscool_connector.py | Migrates papers.cool connector to async + shared request layer + close. |
| src/paperbot/infrastructure/connectors/openalex_connector.py | Migrates OpenAlex connector to async + shared request layer + batched work fetch + close. |
| src/paperbot/infrastructure/connectors/arxiv_connector.py | Migrates arXiv connector to async + shared request layer + close. |
| src/paperbot/infrastructure/adapters/paperscool_adapter.py | Removes thread offload; awaits async connector + implements close. |
| src/paperbot/infrastructure/adapters/arxiv_search_adapter.py | Removes thread offload; awaits async connector + implements close. |
| src/paperbot/context_engine/embeddings.py | Extends hash embedding tokenization for broader CJK/Hangul support. |
| src/paperbot/application/services/paper_search_service.py | Integrates deduplication into RRF fusion/provenance computation. |
| src/paperbot/application/services/paper_dedup.py | Introduces new three-tier deduplication implementation. |
| src/paperbot/api/streaming.py | Adds SSE heartbeats/timeouts, standardized SSE headers, and sse_response. |
| src/paperbot/api/routes/track.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/studio_chat.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/sandbox.py | Switches SSE construction to sse_response and disables timeout for long streams. |
| src/paperbot/api/routes/review.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/research.py | Awaits new async OpenAlex connector methods + closes connector. |
| src/paperbot/api/routes/repro_context.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/paperscool.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/harvest.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/gen_code.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/chat.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/analyze.py | Switches SSE construction to sse_response. |
| src/paperbot/api/routes/agent_board.py | Switches SSE construction to sse_response. |
| src/paperbot/agents/scholar_tracking/semantic_scholar_agent.py | Migrates agent to use unified SemanticScholarClient API. |
| src/paperbot/agents/mixins/semantic_scholar.py | Migrates mixin to unified SemanticScholarClient + adds close helper. |
| requirements.txt | Adds rapidfuzz dependency for fuzzy title dedup. |
| pyproject.toml | Adds rapidfuzz dependency for fuzzy title dedup. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
| return resp.content | ||
| except Exception as e: | ||
| return resp | ||
| except (httpx.TimeoutException, httpx.TransportError, httpx.HTTPStatusError) as e: |
There was a problem hiding this comment.
_request retries httpx.HTTPStatusError for all non-2xx responses, which means 4xx like 400/401/403/404 will be retried even though they’re typically non-transient and can waste retry budget / add latency. Consider only retrying HTTPStatusError when the status is 429 or >= 500 (and immediately re-raising for other 4xx).
Suggested change
| except (httpx.TimeoutException, httpx.TransportError, httpx.HTTPStatusError) as e: | |
| except (httpx.TimeoutException, httpx.TransportError, httpx.HTTPStatusError) as e: | |
| if isinstance(e, httpx.HTTPStatusError): | |
| response = e.response | |
| status = response.status_code if response is not None else None | |
| # Only retry HTTPStatusError for 429 or >= 500; re-raise other statuses. | |
| if status is None or (status != 429 and status < 500): | |
| raise |
Comment on lines
+180
to
+186
| if attempt == config.max_retries: | ||
| logger.error( | ||
| f"异步函数 {func.__name__} 在 {config.max_retries + 1} 次尝试后仍然失败" | ||
| ) | ||
| logger.error(f"最终错误: {str(e)}") | ||
| raise e | ||
|
|
There was a problem hiding this comment.
async_retry re-raises with raise e, which loses the original traceback/context. Use a bare raise when rethrowing the current exception so debugging retains the original stack trace.
| logger.warning(f"异步函数 {func.__name__} 第 {attempt + 1} 次尝试失败: {str(e)}") | ||
| logger.info(f"将在 {delay:.1f} 秒后进行第 {attempt + 2} 次尝试...") | ||
| await asyncio.sleep(delay) | ||
|
|
There was a problem hiding this comment.
async_retry’s broad except Exception will catch asyncio.CancelledError on Python 3.8–3.10 (this repo supports >=3.8) and log it as a non-retryable error before re-raising, creating noisy logs and potentially interfering with cancellation semantics. Add an explicit except asyncio.CancelledError: raise before the generic handler.
Suggested change
| except asyncio.CancelledError: | |
| # 直接传播协程取消，不进行日志记录或重试 | |
| raise |
Comment on lines
+234
to
+237
| if pending_next is not None and not pending_next.done(): | ||
| pending_next.cancel() | ||
| with contextlib.suppress(asyncio.CancelledError, Exception): | ||
| await pending_next |
There was a problem hiding this comment.
In the finally block, a completed pending_next task is not awaited or otherwise observed. If it completed with an exception in a path where it was never awaited (e.g., cancellation happens after the task fails but before the loop awaits it), asyncio can emit “Task exception was never retrieved”. Consider always draining the task in finally (await it if pending, otherwise call pending_next.exception() / pending_next.result() under suppress) before closing the generator.
Suggested change
| if pending_next is not None and not pending_next.done(): | |
| pending_next.cancel() | |
| with contextlib.suppress(asyncio.CancelledError, Exception): | |
| await pending_next | |
| if pending_next is not None: | |
| if not pending_next.done(): | |
| pending_next.cancel() | |
| with contextlib.suppress(asyncio.CancelledError, Exception): | |
| await pending_next | |
| else: | |
| # Drain the completed task to avoid "Task exception was never retrieved" | |
| with contextlib.suppress(Exception): | |
| pending_next.result() |
Comment on lines
+43
to
+49
| doi = self._extract_doi(paper) | ||
| if doi: | ||
| existing = self._doi_index.get(doi) | ||
| if existing is not None: | ||
| self._merge_into(existing, paper) | ||
| self._canonical[id(paper)] = existing | ||
| return False |
There was a problem hiding this comment.
When a DOI duplicate is detected (existing found in _doi_index), the donor paper’s arXiv identity (and any other newly merged identities) are not added to the corresponding indexes. This can cause later papers that only have the arXiv ID to miss Tier-2 dedup and appear as duplicates. Consider refreshing both DOI/arXiv indexes after merging (or at least indexing the donor’s extracted arXiv ID(s)) so the canonical winner can be found by any identity type.
Comment on lines
+214
to
+217
| for ident in paper.identities: | ||
| if ident.source == "doi" and ident.external_id: | ||
| values.add(ident.external_id.strip().lower()) | ||
| return values |
There was a problem hiding this comment.
DOI normalization here only lowercases/strips whitespace. The codebase already has a DOI normalizer that handles common URL/prefix forms (e.g. https://doi.org/...); using it here would prevent false negatives in Tier-1 dedup when connectors provide DOI URLs.
Comment on lines
+223
to
+232
| if ident.source != "arxiv" or not ident.external_id: | ||
| continue | ||
| raw = ident.external_id.strip().lower().removeprefix("arxiv:") | ||
| # Strip version suffix (e.g. 2301.12345v2 -> 2301.12345) | ||
| if "v" in raw: | ||
| head, tail = raw.rsplit("v", 1) | ||
| if head and tail.isdigit(): | ||
| values.add(head) | ||
| continue | ||
| values.add(raw) |
There was a problem hiding this comment.
arXiv ID extraction strips only an "arxiv:" prefix and version suffix, but won’t match identities stored as arxiv.org/abs/... or arxiv.org/pdf/... which are normalized elsewhere in the codebase. Consider reusing the shared arXiv normalizer so Tier-2 dedup is consistent across sources.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces significant improvements to the infrastructure, including unifying async request handling with a robust AsyncRequestLayer, hardening SSE streaming with timeouts and heartbeats, and implementing new three-tier paper deduplication logic, along with commendable added test coverage. However, a potential SSRF vulnerability was identified in the OpenAlexConnector where an external URL is used for a network request without validation; it is recommended to validate that all externally-supplied URLs belong to expected domains before fetching them. Additionally, there are a few medium-severity suggestions for further improvement regarding exception handling in the SSE wrapper and unifying the HTTP client implementations.
Comment on lines
+80
to
84
| payload = await self._request.get_json( | ||
| cited_by_api_url, | ||
| headers=self._headers, | ||
| params={"per-page": max(1, min(int(limit), 200))}, | ||
| timeout=self.timeout_s, | ||
| ) |
There was a problem hiding this comment.
The get_citing_works method uses the cited_by_api_url value directly from an external API response (OpenAlex) to make a subsequent network request. An attacker who can influence the OpenAlex API response (e.g., by contributing a malicious paper entry with a crafted cited_by_api_url) could potentially cause the server to make requests to internal services or unintended external endpoints. This is a Server-Side Request Forgery (SSRF) vulnerability. It is recommended to validate that the URL starts with the expected base URL (e.g., https://api.openalex.org/) before making the request.
Comment on lines
+33
to
+40
| def _get_s2_client(self) -> SemanticScholarClient: | ||
| if self._s2_client is None: | ||
| request_interval = 0.3 if self._s2_api_key else 3.0 | ||
| self._s2_client = SemanticScholarClient( | ||
| api_key=self._s2_api_key, | ||
| request_interval=request_interval, | ||
| ) | ||
| return self._s2_client |
There was a problem hiding this comment.
While many connectors have been migrated to use the new httpx-based AsyncRequestLayer, the SemanticScholarClient used here is still based on a different aiohttp-based client (paperbot.infrastructure.api_clients.base.APIClient). This creates an inconsistency in the codebase and doesn't fully achieve the PR's goal of unifying async request handling. To improve maintainability and consistency, consider refactoring SemanticScholarClient and its base APIClient to use the more robust AsyncRequestLayer as well.
| if timeout_seconds is not None and (time.monotonic() - started_at) >= timeout_seconds: | ||
| if pending_next is not None: | ||
| pending_next.cancel() | ||
| with contextlib.suppress(asyncio.CancelledError, Exception): |
There was a problem hiding this comment.
The use of contextlib.suppress(asyncio.CancelledError, Exception) is overly broad and can mask underlying bugs. While CancelledError is expected when a task is cancelled, suppressing all other Exception types during the cleanup await could hide issues in the generator's cleanup logic or other unexpected problems. It's safer to only suppress the expected CancelledError. This same issue is present on line 236.
Suggested change
| with contextlib.suppress(asyncio.CancelledError, Exception): | |
| with contextlib.suppress(asyncio.CancelledError): |
| pending_next.cancel() | ||
| with contextlib.suppress(asyncio.CancelledError, Exception): | ||
| await pending_next | ||
| with contextlib.suppress(Exception): |
There was a problem hiding this comment.
Suppressing all Exception types during generator.aclose() can hide important errors from the generator's cleanup logic. It's better to let unexpected exceptions propagate or at least log them, rather than silently ignoring them. For example, you could log the exception while still suppressing it: with contextlib.suppress(Exception) as e: if e: log.warning(...)
jerry609
added a commit
that referenced
this pull request
Mar 13, 2026
* fix: validate studio output dirs and add p2c module design docs :wq * feat(PaperToContext): M3 #140 Closes #140 * feat(p2c): implement ContextEngineBridge to inject user context into extraction Related to #157 * activate paper-scope memory read/write path. Raletd to #158 * fix(p2c): address Gemini code review issues from PR. Related to #157 * fix(p2c): address Gemini code review issues. Related to #158 * fix(p2c): sanitize XML tag content to prevent tag-escape prompt injection * feat(memory): add FTS5 full-text search and sqlite-vec hybrid search. Related to #161 * feat(p2c): persist CodeMemory experiences to SQLite. Related to #162 * address Gemini code review issues from PR #224 and #225 * fix(memory): address Gemini code review issues from #153 epic audit P0 fixes: - Replace unbounded daemon threads with ThreadPoolExecutor(max_workers=2) for embedding writes; atexit + close() ensure graceful shutdown so in-flight embeddings are not lost on process exit - Restore apprise>=1.9.0 and feedgen>=1.0.0 removed in epic branch, which would have broken Epic #179 push/RSS features on merge P1 fixes: - _escape_fts: replace double-quote-only escaping with a whitelist regex [A-Za-z0-9_+-] so FTS5 operators (*, NEAR, NOT, ^) cannot alter query semantics; empty queries now short-circuit to [] - _hybrid_merge: skip items with None/invalid id instead of defaulting to id=0, preventing silent score collisions across unrelated records - ReproExperienceStore: add application-level dedup check + UNIQUE constraint (paper_id, pattern_type, content) + IntegrityError fallback to prevent duplicate experiences from accumulating across retries Migration: - 0022_repro_experience_dedup: adds uq_repro_exp_paper_type_content unique constraint to existing repro_code_experience table Tests: 47 unit tests pass (+3 new tests covering the fixes above) * fix: harden repro experience isolation and wire persistence into repro pipeline Add user-scoped isolation for repro_code_experience and enforce dedup semantics with migration 0022_repro_experience_dedup. Inject ReproExperienceStore through ReproAgent/Orchestrator/CodingAgent/GenerationNode and propagate user_id/pack_id in generation, verification, and debugging persistence paths. Update /api/gen-code to accept user_id and extend unit tests for user isolation and persistence behavior. * feat(memory): introduce memory decay mechanism (#163) Add decay-aware scoring that combines relevance (confidence), recency (exponential decay with 90-day half-life), and usage frequency to re-rank search results. New memories default to expires_at = created_at + 365 days. search_memories() now auto-touches usage on hits. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(memory): add cross-track batch search (#164) Add search_memories_batch() that queries multiple scope_ids in a single SQL call, eliminating the N+1 loop in build_context_pack(). Engine now uses this batch method for cross-track memory retrieval. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(context): implement layered context loading (#165) Refactor build_context_pack() into 4 layer methods: - Layer 0: user profile (cached with 5-min TTL, ~200 tokens) - Layer 1: track context — tasks/milestones (~500 tokens) - Layer 2: query-relevant + cross-track memories (~1000 tokens) - Layer 3: paper-scoped memories (on-demand) Return value adds context_layers metadata while remaining fully backward-compatible. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(memory): align decay with OpenClaw patterns - Use standard ln(2)/halfLifeDays lambda formula (matches OpenClaw temporal-decay.ts:toDecayLambda) - Default half-life lowered to 30 days (was 90, now matches OpenClaw) - Evergreen memories (global scope, preference kind) are immune to recency decay (inspired by OpenClaw isEvergreenMemoryPath) - Add _to_decay_lambda() and _is_evergreen_memory() helpers - Expand tests for lambda math, half-life precision, and evergreen logic Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(context): isolate layer0 cache and touch batch hits (#236) * feat(memory): upgrade batch retrieval with hybrid and MMR options (#237) * feat(context): add embedding fallback chain and token guard config (#238) * fix(memory): bound batch hybrid candidates by scope_ids * docs: update README with macOS python3 tips * docs: update README with more badges * docs: update README with deleting extra badges * feat(search): add offline retrieval benchmark harness Add a deterministic retrieval benchmark fixture, scorer, CLI, docs, and CI smoke gate for PaperSearchService.\n\nCloses #284\nRefs #283 * feat(context): add offline context-engine benchmark Add deterministic fixtures, scoring, CLI, and smoke coverage for layered assembly, token guard, and advisory routing in ContextEngine.\n\nCloses #286\nRefs #283 * feat(memory): add scope isolation acceptance bench Add an offline scope-isolation benchmark for memory retrieval paths, extend the metric collector with cross-user and cross-scope leak rates, and wire the new check into CI.\n\nCloses #285\nRefs #283 * feat(memory): add offline injection robustness detector Add a deterministic prompt-injection pattern detector, labeled offline fixtures, an acceptance benchmark, and CI coverage for Injection Robustness L1.\n\nCloses #287\nRefs #283 * feat(memory): add offline performance benchmark harness Add a deterministic synthetic benchmark for memory search latency baselines across 10k/100k/1M scales, plus docs and smokeable unit coverage.\n\nCloses #288\nRefs #283 * feat: add ROI benchmark for repro memory Closes #289 Refs #283 * docs: add MemoryBench epic completion report Refs #283 * docs: add runtime memory benchmark report Refs #283 * Docs: update README with our new name (#290) * docs: update README with macOS python3 tips * docs: update README with more badges * docs: update README with deleting extra badges * docs: update README with new god name --------- Co-authored-by: 林杰 <linjie@linjiedeMacBook-Air.local> Co-authored-by: 林杰 <linjie@v8d1ef64c.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef43e.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef6c5.dip.tu-dresden.de> * Fix: show friendly error when backend is unreachable (#291) * docs: update README with macOS python3 tips * docs: update README with more badges * docs: update README with deleting extra badges * docs: update README with new god name --------- Co-authored-by: 林杰 <linjie@linjiedeMacBook-Air.local> Co-authored-by: 林杰 <linjie@v8d1ef64c.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef43e.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef6c5.dip.tu-dresden.de> * feat: improve memory ROI and effectiveness benchmarks * fix: avoid importing missing data template in main * fix: stabilize live memory roi benchmark * feat: expand multi-session memory effectiveness benchmark * feat: implement MemoryBench evaluation suite with 4 bench suites Add comprehensive memory module evaluation aligned with LongMemEval (ICLR 2025), LoCoMo (ACL 2024), Mem0, and Letta benchmarks. - Retrieval Bench v2: IR metrics (Recall@5=0.873, MRR@10=0.731, nDCG@10=0.747) with 40 annotated queries across 5 question types and 5 memory dimensions - Scope Isolation + CRUD: zero-leak verification across user x scope matrix, Mem0-aligned CRUD lifecycle (add/update/delete/dedup) - Context Extraction: L0-L3 layer completeness, precision, token budget guard, TrackRouter accuracy (100%), graceful degradation - Injection Robustness L1: offline pattern detection (0% pollution, 0% FP) - Fixture dataset: 45 memories (2 users), 12 injection patterns - Testing documentation with methodology, validity analysis, and results Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add MemoryBench quantitative results to README Add evaluation results section with metrics from 4 bench suites: - Retrieval quality (Recall@5=0.873, MRR@10=0.731, nDCG@10=0.747) - Scope isolation + CRUD lifecycle (zero leaks, all CRUD pass) - Context extraction (100% precision, 100% router accuracy) - Injection robustness (0% pollution, 0% false positive) Includes LoCoMo question-type breakdown and run instructions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(AgentSwarm): add Claude/Codex workflow, workspace persistence, and human review UX (#296) * feat(AgentSwarm): add Claude/Codex workflow, workspace persistence, and human review UX - 添加 /api/agent-board 路由和 Swarm Commander/Dispatcher 基础架构 - 将生成的任务文件持久化到已配置的工作区 - 在工作区中为每个任务生成用户审核文档（reviews/<任务>-user-review.md） - 添加代理看板 UI，包含看板、任务详细日志和人工审核操作 - 要求“全部运行”任务设置工作区，并添加 VS Code 打开操作 - 实现 Studio 存储和 SSE 任务的更新/插入/同步行为 - 为路由流程、持久化和超时处理添加单元/功能测试 - 更新 Studio 布局、PostCSS 配置、后端 URL 助手和锁定文件 Closes #197 * (fix)修改AI review问题 * refactor: share research fetch helpers (#295) Co-authored-by: 林杰 <linjie@v8d1ef6c5.dip.tu-dresden.de> * feat: add research/yearCombobox. Realted to #297 * refactor(research): dedupe fetch helpers across Research pages; fix Radix popover import; add @radix-ui/react-popover dep * fix(web): move @radix-ui/react-popover to dependencies * fix(research): keep track list order stable across activation * refactor(research): share stable track merge helper * fix(research): make paper feedback togglable * feat(research): hide 'Open Discovery Workspace' behind feature flag * feat(research): gate track memory button behind feature flag (#333) * feat(research): gate track memory button behind feature flag * fix(research): avoid reactivating already active track * ci(vercel): add auto deploy workflow for dev branch * feat: calm dashboard workspace (#334) * feat: harden search and connector infrastructure (#339) * refactor(infra): unify async request layer for connectors Implements issue #262 by moving Arxiv/OpenAlex/PapersCool connectors onto the shared async transport with retry support, updating async call sites, and adding focused tests. * perf(infra): batch OpenAlex ID lookups Implements issue #267 by replacing per-ID OpenAlex work fetches with batched filter queries and adds focused coverage for the batched path. * refactor(infra): unify Semantic Scholar client stack Implements issue #266 by routing both the shared mixin and the scholar-tracking agent through the same SemanticScholarClient surface, removing the duplicated legacy API client path, and adding focused unification tests. * feat(infra): harden SSE transport handling Implements issue #263 by moving heartbeat, timeout, cancellation cleanup, and X-Accel-Buffering handling into the shared SSE wrapper and switching the streaming routes onto the common response helper with regression coverage. * fix(infra): bound ARQ jobs and reuse event log Implements issue #268 by adding timeout/max_tries metadata to worker functions, reusing the SqlAlchemyEventLog singleton inside the worker module, and covering both behaviors with focused tests. * feat(search): add three-tier paper deduplicator (DOI/arxiv_id/rapidfuzz) Replaces the identity-key-only dedup in PaperSearchService._fuse_with_rrf() with a dedicated PaperDeduplicator that matches across DOI, arxiv_id (version- stripped), and fuzzy title similarity via rapidfuzz. Merged papers accumulate the best metadata (highest citations, longest abstract, union of identities). Closes #317 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(search): prevent conflicting identity dedup merges Refs #317 * fix(embeddings): add CJK character support to hash embedding tokenizer Extends HashEmbeddingProvider regex to include CJK Unified Ideographs (U+4E00–U+9FFF) and Extension A (U+3400–U+4DBF), enabling proper embedding of Chinese text. Closes #276 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(embeddings): extend hash tokenizer beyond Han-only CJK Refs #276 --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(research): restore feedback track ux on current dev (#341) * refactor(research): dedupe fetch helpers across Research pages; fix Radix popover import; add @radix-ui/react-popover dep * fix(web): preserve item order in research tracks selection Closes #304 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(web): remove unused components Closes #305 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(web): keep active research track visible Refs #304 * fix(research): align paper feedback toggles with persisted state Refs #324 * fix(web): add missing @radix-ui/react-popover dependency Research page crashed with "Module not found: Can't resolve '@radix-ui/react-popover'" because SearchBox.tsx imports popover.tsx which depends on this package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(research): restore feedback track ux on current dev --------- Co-authored-by: 林杰 <linjie@v8d1ef6a5.dip.tu-dresden.de> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(web): resolve DeepCode Studio page issues * fix(repro): improve memory and workflow correctness * feat: add intelligence radar backend * fix(api): block cross-user intelligence feed access * fix(intelligence): avoid thread-unsafe refresh races * refactor: enforce alembic-only store initialization * fix(db): bootstrap sqlite schemas for lazy stores * fix(stores): align ports with store interfaces * fix(stores): align remaining port contracts * fix(api): harden request boundaries and settings handling * fix(api): stop trusting spoofable forwarded hosts * fix(api): gate studio code execution by default * fix(api): stream request-size checks safely * feat(export): add Obsidian filesystem exporter * feat(cli): add Obsidian export command * feat(obsidian): finish configurable vault export workflow (#348) * refactor(web): move workflow workbench out of dashboard * ci(vercel): use --preview flag for dev deploy * fix(ci): remove invalid Vercel preview flag * fix(ci): keep Vercel secrets free of inline comments * refactor: share candidate search boundary and simplify dashboard * refactor(api): split paperscool search curate and ingest * refactor(web): flatten dashboard brief snapshot * refactor(web): reduce dashboard brief visual noise * refactor(web): feature top dashboard signals * fix(obsidian): harden export follow-ups * fix(obsidian): tighten review follow-ups * feat(web): add Obsidian handoff to Research workspace (#352) * feat(web): add Obsidian handoff to Research workspace * fix(web): address obsidian workspace review feedback * feat: ship track-centric research context stack (#356) * feat(api): add track context read model service * feat(api): expose consolidated track context endpoint * refactor(web): migrate research page to track context endpoint * refactor(memory): wrap track-scoped memory access * chore: document track-centric research model and stabilize tests * docs: refresh readme screenshots * fix(deps): align package manifests with runtime imports * docs: refresh email push screenshot * feat(research): close citation graph and obsidian export gaps (#360) * feat(obsidian): add bidirectional vault sync * fix(paper): improve saved papers table layout * feat(openclaw): add paperbot plugin bridge * fix(obsidian): address sync review feedback * feat(web): flatten dashboard action bands * refactor(web): condense dashboard next-up panel * refactor(web): simplify dashboard around recommendations * refactor(web): simplify dashboard surface and workflow copy * feat(web): add dashboard queue actions * fix(web): harden dashboard queue links and brief parsing * fix(paper): unify unsave with feedback API * fix(paper): scope saved papers per track * fix: unblock dashboard build and e2e * ci: add vercel pr preview automation * fix: repair vercel preview workflow setup * fix: run vercel build from repo root * fix: skip preview smoke without bypass secret * feat(auth): add multi-user authentication foundation. Related to #151 - Add User domain model and SQLAlchemy UserModel with soft-delete support - Add SqlAlchemyUserStore with email/GitHub user CRUD, password auth, reset tokens - Add JWT signing/verification (python-jose), bcrypt password hashing - Add FastAPI auth dependencies: get_user_id (optional fallback) and get_current_user (strict) - Add /api/auth routes: register, login, github/exchange, me, forgot/reset-password - Add Alembic migrations for users and password_reset_tokens tables - Add AUTH_OPTIONAL env var for gradual migration from legacy 'default' user - Fix account lifecycle bugs: reactivate on OAuth re-login, reject inactive on password login - Add auth API tests * fix(auth): address code review feedback on PR #365 * feat(document): add explicit evidence indexing pipeline * docs(benchmark): define document evidence eval contract * feat(benchmark): add document evidence eval scaffold * fix(api): restore py39 auth compatibility after dev rebase * chore(logging): surface cleanup and FTS5 failures * fix(paper): refine saved filters and cleanup code * fix(paper): refine paper context cleanup and year filter * feat(eval): support dedicated embedding benchmark providers * feat(settings): add embedding endpoint configuration * refactor(settings): align embedding endpoint ux with cc-switch * refactor(settings): simplify embedding endpoint panel * refactor(settings): tighten embedding layout on wide screens * feat(studio): refine paper gallery icon animation and context workspace layout * ci: disable native vercel git deploys * docs: restore master demo gallery assets * fix: unblock ci for merge-dev-into-master * fix: address codeql alerts * fix: harden agent board workspace path validation --------- Co-authored-by: boyu <oor2020@163.com> Co-authored-by: WenjingWang <jingnvx@outlook.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: 林杰 <linjie@linjiedeMacBook-Air.local> Co-authored-by: 林杰 <linjie@v8d1ef64c.dip.tu-dresden.de> Co-authored-by: Linjie-top <linjie666z@gmail.com> Co-authored-by: 林杰 <linjie@v8d1ef43e.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef6c5.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef6a5.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef4ea.dip.tu-dresden.de> Co-authored-by: 林杰 <linjie@v8d1ef41f.dip.tu-dresden.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation