merge: dev into master#373
Conversation
…extraction Related to #157
P0 fixes: - Replace unbounded daemon threads with ThreadPoolExecutor(max_workers=2) for embedding writes; atexit + close() ensure graceful shutdown so in-flight embeddings are not lost on process exit - Restore apprise>=1.9.0 and feedgen>=1.0.0 removed in epic branch, which would have broken Epic #179 push/RSS features on merge P1 fixes: - _escape_fts: replace double-quote-only escaping with a whitelist regex [A-Za-z0-9_+-] so FTS5 operators (*, NEAR, NOT, ^) cannot alter query semantics; empty queries now short-circuit to [] - _hybrid_merge: skip items with None/invalid id instead of defaulting to id=0, preventing silent score collisions across unrelated records - ReproExperienceStore: add application-level dedup check + UNIQUE constraint (paper_id, pattern_type, content) + IntegrityError fallback to prevent duplicate experiences from accumulating across retries Migration: - 0022_repro_experience_dedup: adds uq_repro_exp_paper_type_content unique constraint to existing repro_code_experience table Tests: 47 unit tests pass (+3 new tests covering the fixes above)
…o pipeline Add user-scoped isolation for repro_code_experience and enforce dedup semantics with migration 0022_repro_experience_dedup. Inject ReproExperienceStore through ReproAgent/Orchestrator/CodingAgent/GenerationNode and propagate user_id/pack_id in generation, verification, and debugging persistence paths. Update /api/gen-code to accept user_id and extend unit tests for user isolation and persistence behavior.
Add decay-aware scoring that combines relevance (confidence), recency (exponential decay with 90-day half-life), and usage frequency to re-rank search results. New memories default to expires_at = created_at + 365 days. search_memories() now auto-touches usage on hits. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add search_memories_batch() that queries multiple scope_ids in a single SQL call, eliminating the N+1 loop in build_context_pack(). Engine now uses this batch method for cross-track memory retrieval. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Refactor build_context_pack() into 4 layer methods: - Layer 0: user profile (cached with 5-min TTL, ~200 tokens) - Layer 1: track context — tasks/milestones (~500 tokens) - Layer 2: query-relevant + cross-track memories (~1000 tokens) - Layer 3: paper-scoped memories (on-demand) Return value adds context_layers metadata while remaining fully backward-compatible. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use standard ln(2)/halfLifeDays lambda formula (matches OpenClaw temporal-decay.ts:toDecayLambda) - Default half-life lowered to 30 days (was 90, now matches OpenClaw) - Evergreen memories (global scope, preference kind) are immune to recency decay (inspired by OpenClaw isEvergreenMemoryPath) - Add _to_decay_lambda() and _is_evergreen_memory() helpers - Expand tests for lambda math, half-life precision, and evergreen logic Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix(paper): improve saved table layout and track-scoped filtering
workspace layout
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
📝 Coding Plan
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment Tip You can disable the changed files summary in the walkthrough.Disable the |
There was a problem hiding this comment.
Pull request overview
Merge dev into master, bringing across infrastructure/application refactors (ports + store implementations), new FastAPI middleware/streaming helpers, and feature additions including Obsidian export/sync, embedding endpoint settings, and user authentication.
Changes:
- Replace direct
Base.metadata.create_all(...)usage across stores withSessionProvider.ensure_tables(...)and align store classes with application-layer Ports. - Add/extend FastAPI capabilities: SSE helper (
sse_response), centralized error handling, API-key middleware + rate limiting, auth endpoints (JWT + password reset), embedding settings endpoints. - Expand integrations: Obsidian parsing/watching/export, async crawling via shared
AsyncRequestLayer, additional API clients (GitHub radar, X recent search), and OpenClaw plugin shim.
Reviewed changes
Copilot reviewed 136 out of 301 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| src/paperbot/infrastructure/stores/repro_experience_store.py | Switch schema init to ensure_tables during store bootstrap. |
| src/paperbot/infrastructure/stores/repro_context_store.py | Make store implement ReproContextPort. |
| src/paperbot/infrastructure/stores/pipeline_session_store.py | Implement PipelineSessionPort + use ensure_tables. |
| src/paperbot/infrastructure/stores/paper_store.py | Implement RegistryPort + use ensure_tables. |
| src/paperbot/infrastructure/stores/model_endpoint_store.py | Implement ModelEndpointPort + use ensure_tables. |
| src/paperbot/infrastructure/stores/memory_store.py | Memory scoring tweaks, schema bootstrap changes, log FTS5 failures, touch usage after batch search. |
| src/paperbot/infrastructure/stores/llm_usage_store.py | Implement LLMUsagePort + use ensure_tables. |
| src/paperbot/infrastructure/stores/identity_store.py | Use ensure_tables for schema init. |
| src/paperbot/infrastructure/stores/embedding_endpoint_store.py | New store for embedding endpoint settings with keychain support. |
| src/paperbot/infrastructure/stores/author_store.py | Implement AuthorPort, use ensure_tables, add get/list APIs. |
| src/paperbot/infrastructure/stores/_serialization.py | New shared JSON dump/load helpers. |
| src/paperbot/infrastructure/services/api_client.py | Replace legacy implementation with compatibility import of shared APIClient. |
| src/paperbot/infrastructure/queue/arq_worker.py | Cache event log + annotate arq functions with timeouts/max_tries. |
| src/paperbot/infrastructure/obsidian/watcher.py | New Obsidian vault watcher with debounce queue (watchdog optional). |
| src/paperbot/infrastructure/obsidian/parser.py | New Obsidian markdown parsing utilities (frontmatter/sections/tags/wikilinks). |
| src/paperbot/infrastructure/obsidian/conflict.py | Detect managed-content hash mismatch conflicts for Obsidian notes. |
| src/paperbot/infrastructure/obsidian/init.py | Export Obsidian subsystem public API. |
| src/paperbot/infrastructure/monitoring/resource_monitor.py | Use ensure_tables for schema init. |
| src/paperbot/infrastructure/logging/execution_logger.py | Use ensure_tables for schema init. |
| src/paperbot/infrastructure/exporters/obsidian_sync.py | New helper to export track snapshots to an Obsidian vault. |
| src/paperbot/infrastructure/exporters/obsidian_report_exporter.py | New Obsidian report note exporter. |
| src/paperbot/infrastructure/exporters/init.py | Export filesystem exporters + snapshot helpers. |
| src/paperbot/infrastructure/event_log/sqlalchemy_event_log.py | Use ensure_tables for schema init. |
| src/paperbot/infrastructure/crawling/request_layer.py | Add shared async request layer with retries/backoff/jitter and connection reuse. |
| src/paperbot/infrastructure/connectors/paperscool_connector.py | Convert to async + use AsyncRequestLayer. |
| src/paperbot/infrastructure/connectors/openalex_connector.py | Convert to async + batch work fetch + use AsyncRequestLayer. |
| src/paperbot/infrastructure/connectors/arxiv_connector.py | Convert to async + use AsyncRequestLayer params. |
| src/paperbot/infrastructure/api_clients/x_client.py | New X recent-search client (requests). |
| src/paperbot/infrastructure/api_clients/github_client.py | New GitHub issues “radar” client (requests). |
| src/paperbot/infrastructure/api_clients/init.py | Export additional api client types. |
| src/paperbot/infrastructure/adapters/paperscool_adapter.py | Stop using asyncio.to_thread now that connector is async; close connector. |
| src/paperbot/infrastructure/adapters/arxiv_search_adapter.py | Stop using asyncio.to_thread now that connector is async; close connector. |
| src/paperbot/domain/user.py | New User domain model. |
| src/paperbot/core/workflow_coordinator.py | Publish quality/influence scores; integrate fail-fast early exit + stage status updates. |
| src/paperbot/core/fail_fast.py | Expand early-exit skipped stages list. |
| src/paperbot/context_engine/embeddings.py | Add registry-backed embedding config resolution (+ env fallbacks) and improve hash tokenizer. |
| src/paperbot/application/workflows/unified_topic_search.py | Centralize candidate search/id resolution via application service helpers. |
| src/paperbot/application/workflows/dailypaper.py | Use explicit ingest helper (ingest_candidate_papers). |
| src/paperbot/application/services/track_memory_service.py | New track-scoped memory service facade. |
| src/paperbot/application/services/paper_search_service.py | Fuse search results using dedup canonicalization with RRF scoring. |
| src/paperbot/application/services/identity_resolver.py | Allow injecting SessionProvider for reuse/testing. |
| src/paperbot/application/services/candidate_search.py | New shared candidate-search + explicit-ingest helpers. |
| src/paperbot/application/services/author_backfill_service.py | Allow injecting SessionProvider for reuse/testing. |
| src/paperbot/application/services/anchor_service.py | Ensure schema + normalize/collapse effective feedback actions for scoring. |
| src/paperbot/application/services/init.py | Export new services. |
| src/paperbot/application/registries/default_sources.py | Add GitHub/HuggingFace sources and update X source descriptor. |
| src/paperbot/application/ports/workflow_metric_port.py | New workflow metrics persistence port. |
| src/paperbot/application/ports/vault_exporter_port.py | New vault exporter port. |
| src/paperbot/application/ports/track_memory_store_port.py | New track memory store port. |
| src/paperbot/application/ports/subscriber_port.py | New subscriber management port. |
| src/paperbot/application/ports/research_track_read_port.py | New track read-model port. |
| src/paperbot/application/ports/pipeline_session_port.py | New pipeline session port. |
| src/paperbot/application/ports/paper_registry_port.py | Extend registry port (seen_at). |
| src/paperbot/application/ports/model_endpoint_port.py | New model endpoint management port. |
| src/paperbot/application/ports/memory_port.py | New memory port. |
| src/paperbot/application/ports/llm_usage_port.py | New LLM usage tracking port. |
| src/paperbot/application/ports/document_intelligence_port.py | New document intelligence ports + value objects. |
| src/paperbot/application/ports/author_port.py | New author port. |
| src/paperbot/application/ports/init.py | Export expanded port surface. |
| src/paperbot/api/streaming.py | Add heartbeat/timeout SSE wrapping + sse_response helper. |
| src/paperbot/api/routes/track.py | Use shared sse_response wrapper. |
| src/paperbot/api/routes/studio_chat.py | Gate code mode by env + migrate to sse_response. |
| src/paperbot/api/routes/sandbox.py | Centralize error responses + migrate streams to sse_response. |
| src/paperbot/api/routes/runbook.py | Add cross-platform file locking + lazy provider init with ensure_tables. |
| src/paperbot/api/routes/review.py | Use sse_response. |
| src/paperbot/api/routes/repro_context.py | Cache store instance + use sse_response. |
| src/paperbot/api/routes/model_endpoints.py | Cache stores + route through _get_store() helpers. |
| src/paperbot/api/routes/memory.py | Cache store instance in-module via _get_store(). |
| src/paperbot/api/routes/jobs.py | Return structured JSON error responses with trace ids. |
| src/paperbot/api/routes/harvest.py | Use sse_response. |
| src/paperbot/api/routes/gen_code.py | Use sse_response. |
| src/paperbot/api/routes/embedding_settings.py | New routes for embedding endpoint configuration and connectivity testing. |
| src/paperbot/api/routes/chat.py | Limit history via validator + stream errors with trace id + use sse_response. |
| src/paperbot/api/routes/auth.py | New auth endpoints (register/login/me/GitHub exchange/reset password flows). |
| src/paperbot/api/routes/analyze.py | Use sse_response. |
| src/paperbot/api/routes/init.py | Register new routers (obsidian/embedding/intelligence/agent_board). |
| src/paperbot/api/middleware/rate_limit.py | New in-memory rate limiting middleware. |
| src/paperbot/api/middleware/auth.py | New API-key auth + CORS install helpers. |
| src/paperbot/api/middleware/init.py | Export middleware installers. |
| src/paperbot/api/main.py | Deterministic .env load + install error handling/auth/rate limit/CORS + add routers + obsidian runtime hooks. |
| src/paperbot/api/error_handling.py | New centralized trace-id + request-size guard + exception handlers. |
| src/paperbot/api/auth/password.py | New bcrypt password hashing helpers. |
| src/paperbot/api/auth/jwt.py | New JWT helpers for access tokens. |
| src/paperbot/api/auth/email.py | New email sender (log mode + optional Resend). |
| src/paperbot/api/auth/dependencies.py | New JWT auth dependencies with optional fallback mode. |
| src/paperbot/agents/scholar_tracking/semantic_scholar_agent.py | Switch to SemanticScholarClient calls. |
| src/paperbot/agents/mixins/semantic_scholar.py | Replace aiohttp usage with shared SemanticScholarClient. |
| src/paperbot/agents/base.py | Lazily import Anthropic client; avoid hard dependency import at module load. |
| scripts/eval_document_evidence.py | New CLI for document evidence benchmark evaluation. |
| requirements-ci.txt | Add crypto/keyring/watchdog deps for CI. |
| pyproject.toml | Expand and pin runtime deps (FastAPI stack, auth, crawling, exporters, etc.). |
| openclaw/paperbot-openclaw/tsconfig.json | New TS build config for OpenClaw shim package. |
| openclaw/paperbot-openclaw/src/types.ts | Define OpenClaw plugin config/types; normalize config helper. |
| openclaw/paperbot-openclaw/src/paperbot-client.ts | HTTP/SSE client wrapper for PaperBot FastAPI endpoints. |
| openclaw/paperbot-openclaw/src/intents.ts | Intent detection + latest user message helper. |
| openclaw/paperbot-openclaw/src/cron.ts | Default cron descriptors + config-aware input resolution. |
| openclaw/paperbot-openclaw/package.json | New OpenClaw shim package definition and build script. |
| openclaw/paperbot-openclaw/README.md | Package documentation for OpenClaw integration. |
| openclaw/paperbot-openclaw/.gitignore | Ignore dist/node_modules for OpenClaw shim package. |
| evals/runners/run_document_evidence_benchmark_smoke.py | New smoke runner with recall/hit-rate thresholds. |
| evals/fixtures/document_evidence/bench_v1.json | New seed retrieval benchmark fixture dataset. |
| evals/README.md | Document evidence retrieval coverage item added. |
| env.example | Add env vars for embeddings, intelligence, Obsidian. |
| docs/design/research-track-context-model.md | New design doc for track context model & memory ownership. |
| config/config.yaml | Add Obsidian settings block. |
| config/validated_settings.py | Remove legacy pydantic validated settings loader. |
| alembic/versions/0026_embedding_endpoint_settings.py | Migration for embedding endpoint settings table. |
| alembic/versions/0025_password_reset_tokens.py | Migration for password reset tokens table. |
| alembic/versions/0024_users.py | Migration for users table. |
| alembic/versions/0023_intelligence_events.py | Migration for intelligence events table. |
| .github/workflows/vercel-auto-deploy.yml | New Vercel preview deploy workflow + optional smoke. |
| .github/workflows/ci.yml | Expand test list to include repro suite and additional tests. |
Files not reviewed (1)
- openclaw/paperbot-openclaw/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
|
|
||
| _LOCAL_HOSTS = {"127.0.0.1", "::1", "localhost", "testclient", "testserver"} | ||
| _PUBLIC_PATHS = {"/health", "/openapi.json"} | ||
| _PUBLIC_PATH_PREFIXES = ("/docs", "/redoc") |
There was a problem hiding this comment.
Installing APIKeyAuthMiddleware will also protect /api/auth/* endpoints (register/login/reset), because is_public_path() only whitelists /health, /openapi.json, and docs. This will effectively break auth flows for non-local callers unless they also provide PAPERBOT_API_KEY. Fix by whitelisting /api/auth paths (e.g., add a public prefix) or by conditionally skipping API-key auth when JWT auth routes are being used.
| _PUBLIC_PATH_PREFIXES = ("/docs", "/redoc") | |
| _PUBLIC_PATH_PREFIXES = ("/docs", "/redoc", "/api/auth") |
| if is_public_path(request.url.path) or is_local_request(request): | ||
| await self.app(scope, receive, send) | ||
| return | ||
|
|
||
| expected_api_key = resolve_api_key() | ||
| bearer_token = _extract_bearer_token(request) | ||
| if not expected_api_key or bearer_token != expected_api_key: |
There was a problem hiding this comment.
Installing APIKeyAuthMiddleware will also protect /api/auth/* endpoints (register/login/reset), because is_public_path() only whitelists /health, /openapi.json, and docs. This will effectively break auth flows for non-local callers unless they also provide PAPERBOT_API_KEY. Fix by whitelisting /api/auth paths (e.g., add a public prefix) or by conditionally skipping API-key auth when JWT auth routes are being used.
|
|
||
| SECRET_KEY = os.environ.get("PAPERBOT_JWT_SECRET", "change-me-in-production") |
There was a problem hiding this comment.
Using a hard-coded fallback JWT secret (\"change-me-in-production\") is unsafe: if this env var is missing in any deployed environment, tokens become forgeable. Prefer failing fast when PAPERBOT_JWT_SECRET is not set (or generating it at install-time and storing it securely), and consider logging a startup error when the secret is default/weak.
| SECRET_KEY = os.environ.get("PAPERBOT_JWT_SECRET", "change-me-in-production") | |
| import logging | |
| logger = logging.getLogger(__name__) | |
| try: | |
| SECRET_KEY = os.environ["PAPERBOT_JWT_SECRET"] | |
| except KeyError as exc: | |
| logger.error("PAPERBOT_JWT_SECRET environment variable is not set; refusing to start with an insecure JWT secret.") | |
| raise RuntimeError( | |
| "PAPERBOT_JWT_SECRET environment variable must be set to a strong, unpredictable value." | |
| ) from exc | |
| if not SECRET_KEY or SECRET_KEY == "change-me-in-production": | |
| logger.error("Insecure PAPERBOT_JWT_SECRET value detected; refusing to start with a default or empty JWT secret.") | |
| raise RuntimeError( | |
| "Insecure PAPERBOT_JWT_SECRET value; please configure a strong, unpredictable secret." | |
| ) |
| logger.debug("[auth] Received token (first 20 chars): %s...", token[:20]) | ||
| try: | ||
| user_id = decode_token(token) | ||
| logger.debug("[auth] Token valid, user_id=%s", user_id) | ||
| except JWTError as e: | ||
| logger.warning("[auth] Invalid token: %s | token prefix: %s...", e, token[:20]) |
There was a problem hiding this comment.
Logging any part of an access token (even a prefix) is sensitive and can increase the blast radius of log exposure. Remove token prefix logging from both debug and warning logs; if needed for correlation, log the request trace id and/or a one-way hash of the token instead.
| logger.debug("[auth] Received token (first 20 chars): %s...", token[:20]) | |
| try: | |
| user_id = decode_token(token) | |
| logger.debug("[auth] Token valid, user_id=%s", user_id) | |
| except JWTError as e: | |
| logger.warning("[auth] Invalid token: %s | token prefix: %s...", e, token[:20]) | |
| logger.debug("[auth] Received bearer token") | |
| try: | |
| user_id = decode_token(token) | |
| logger.debug("[auth] Token valid, user_id=%s", user_id) | |
| except JWTError as e: | |
| logger.warning("[auth] Invalid token: %s", e) |
| resp = await client.request(method, url, headers=hdrs, params=params) | ||
| if resp.status_code == 429 or resp.status_code >= 500: | ||
| if attempt >= self.policy.max_retries: | ||
| resp.raise_for_status() | ||
| retry_after = resp.headers.get("Retry-After") | ||
| await resp.aread() | ||
| delay = self._retry_delay(attempt, retry_after) | ||
| await asyncio.sleep(delay) | ||
| continue | ||
|
|
||
| resp.raise_for_status() | ||
| return resp.content | ||
| except Exception as e: | ||
| return resp | ||
| except (httpx.TimeoutException, httpx.TransportError, httpx.HTTPStatusError) as e: | ||
| last_err = e | ||
| if attempt >= self.policy.max_retries: | ||
| break | ||
| backoff = min(self.policy.max_backoff_s, self.policy.base_backoff_s * (2**attempt)) | ||
| await asyncio.sleep(backoff) | ||
| await asyncio.sleep(self._retry_delay(attempt)) |
There was a problem hiding this comment.
The retry loop will also retry on non-retryable 4xx responses (e.g., 400/401/403/404), because resp.raise_for_status() raises httpx.HTTPStatusError and that exception is treated as retryable until max_retries. Concretely, a 401 will backoff/retry instead of failing immediately. Fix by only retrying HTTPStatusError for 429 and 5xx (e.g., inspect exc.response.status_code) and re-raise immediately for other 4xx codes.
| def _merge_embedding_config( | ||
| base: Optional[EmbeddingConfig], | ||
| override: Optional[EmbeddingConfig], | ||
| ) -> EmbeddingConfig: | ||
| base_config = base or EmbeddingConfig() | ||
| override_config = override or EmbeddingConfig() | ||
| return EmbeddingConfig( | ||
| model=base_config.model if str(base_config.model or "").strip() else override_config.model, | ||
| api_key=( | ||
| base_config.api_key | ||
| if str(base_config.api_key or "").strip() | ||
| else override_config.api_key | ||
| ), | ||
| base_url=( | ||
| base_config.base_url | ||
| if str(base_config.base_url or "").strip() | ||
| else override_config.base_url | ||
| ), | ||
| model_env=str( | ||
| base_config.model_env or override_config.model_env or "PAPERBOT_EMBEDDING_MODEL" | ||
| ), | ||
| api_key_env=str( | ||
| base_config.api_key_env or override_config.api_key_env or "PAPERBOT_EMBEDDING_API_KEY" | ||
| ), | ||
| base_url_env=str( | ||
| base_config.base_url_env | ||
| or override_config.base_url_env | ||
| or "PAPERBOT_EMBEDDING_BASE_URL" | ||
| ), | ||
| timeout_seconds=float( | ||
| base_config.timeout_seconds or override_config.timeout_seconds or 30.0 | ||
| ), | ||
| ) |
There was a problem hiding this comment.
_merge_embedding_config() unintentionally prevents registry-provided api_key_env / base_url_env / model_env (and timeout_seconds) from taking effect when base is a default EmbeddingConfig(), because the base default strings are truthy and always win. This breaks the intent of _load_registry_embedding_config() setting api_key_env from DB. Fix by treating default values as 'unset' (e.g., compare to EmbeddingConfig().api_key_env / etc.), or by making the override config take precedence whenever it provides a non-empty value.
| existing = _user_store.get_by_email(req.email) | ||
| if not existing: | ||
| raise HTTPException(status_code=401, detail="Email not registered.") | ||
| if not existing.is_active: | ||
| raise HTTPException(status_code=401, detail="Account has been deleted. Please register again.") | ||
| user = _user_store.authenticate(req.email, req.password) | ||
| if not user: | ||
| raise HTTPException(status_code=401, detail="Incorrect password.") |
There was a problem hiding this comment.
The login endpoint returns distinct error messages for unknown email vs deleted account vs wrong password, which enables account/email enumeration. Consider returning a single generic 401 detail for all authentication failures (and logging the specific reason server-side with a trace id) while keeping /forgot-password as the user-facing discovery path.
|
Warning Gemini is experiencing higher than usual traffic and was unable to create the summary. Please try again in a few hours by commenting |
Summary
devbranch intomasterdevcontent taking precedencemasterwhere those screenshots are preferredNotes
masterwas blocked by repository rules (GH013) because CodeQL must evaluate the large merge commit firstValidation
git merge -X theirs --no-edit origin/devorigin/master:README.mdasset/ui/dashboard.pngasset/ui/research.pngasset/ui/deepcode.jpgasset/ui/9-5.pngFollow-up
masterdashboard image instead.