Feat/daily push epic 179#218
Conversation
Multi-channel push infrastructure with structured digest cards: - #187: Integrate HF upvotes into Judge scoring prompt - #180: Add digest_card extraction (highlight/method/finding/tags) with new LLM prompt, email template rendering, markdown output - #181: MinerU Cloud API client for PDF figure extraction - #182: Apprise multi-channel push layer with YAML config - #183-185: Channel formatters (Telegram MarkdownV2, Discord Rich Embed, WeCom markdown+news, Feishu/Lark interactive card) - #186: RSS 2.0 + Atom feed endpoints (/api/feed/daily.xml, .atom) Also creates follow-up issues #190 (Push Channel Settings UI) and #191 (Dashboard Digest Card Enhancement). 56 new tests, all passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…y-paper workflow Refs #179
- Add docs/proposals/agentswarm-design.md covering multi-agent orchestration architecture, adapter pattern, session management, skills ecosystem, and OpenCode/OpenClaw/oMo integration (refs #197, #214) - Add Playwright e2e test scaffolding for web dashboard (smoke, papers, research specs) - Update web/.gitignore for Playwright artifacts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! 此拉取请求引入了名为 AgentSwarm 的多 Agent 协作平台的重大架构转变，旨在简化将研究论文转化为可复现代码的过程。它通过集成 Hugging Face Daily Papers 并提供强大的缓存和趋势功能，增强了每日论文推送系统，并增加了用于管理订阅的 Telegram 命令 API。此外，推送通知系统通过重试机制和幂等性变得更具弹性，项目文档也已重组以反映这些变化并提高可维护性。Web 前端现在包含端到端测试以确保稳定性。 Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request significantly enhances and refactors the 'Daily Push' feature (Epic 179), improving system robustness and scalability by introducing new features such as Hugging Face Daily Papers as a data source and Telegram command hooks for daily digest management. A security audit, however, identified a high-severity Broken Access Control (IDOR) vulnerability in the Telegram command endpoint and a medium-severity CORS misconfiguration in the main API setup. These critical security issues must be addressed before merging to prevent unauthorized access to user data and potential cross-origin attacks. Additionally, the code review noted overall high quality, with improvements in apprise push reliability, paper deduplication logic, and extensive testing, alongside comprehensive documentation. Minor suggestions were made for refining exception handling and addressing hardcoded configurations to further enhance robustness and flexibility.
| async def telegram_command(body: TelegramCommandRequest): | ||
| store = TelegramSubscriptionStore() | ||
| command, scope, value = _parse_telegram_command(body.text) | ||
| subscriptions = store.list_subscriptions(chat_id=body.chat_id) |
There was a problem hiding this comment.
The /api/push/telegram/command endpoint is vulnerable to Insecure Direct Object Reference (IDOR). It accepts a chat_id directly from the request body and uses it to list, subscribe, or unsubscribe users without any authentication or verification that the request actually comes from Telegram or the owner of the chat_id. An attacker could exploit this to disclose sensitive subscription data or modify subscriptions for any user by guessing or obtaining their chat_id.
To remediate this, implement a verification mechanism for the webhook. For Telegram, you should use a secret token in the webhook URL and verify it in the handler, or validate that the request originates from Telegram's official IP ranges.
| async def dispatch(self, task: SwarmTask) -> AgentResult: | ||
| """Dispatch a task to the best agent.""" | ||
| adapter = self._select_adapter(task) | ||
| session = await self.session_mgr.get_or_create(adapter, task.workspace) |
There was a problem hiding this comment.
在 dispatch 方法中，使用了 self.session_mgr，但它并未在 __init__ 方法中被初始化。为了保持设计文档的严谨性，建议在 __init__ 中添加 session_mgr 的初始化逻辑，例如：
def __init__(self, adapters: Dict[str, AgentAdapter], session_mgr: SessionManager, config: RouterConfig):
self.adapters = adapters
self.session_mgr = session_mgr
self.config = config| try: | ||
| payload = json.loads(file.read_text(encoding="utf-8")) | ||
| except Exception: | ||
| continue |
There was a problem hiding this comment.
这里的异常捕获过于宽泛（except Exception），可能会掩盖一些意料之外的错误。建议捕获更具体的异常，例如 json.JSONDecodeError 和 IOError，并考虑在捕获异常时打印一条警告日志，以便于调试。
except (json.JSONDecodeError, IOError) as e:
print(f"Warning: Could not load or parse {file.name}: {e}")
continue|
|
||
| router = APIRouter() | ||
|
|
||
| _REPORTS_DIR = Path("./reports/dailypaper") |
There was a problem hiding this comment.
报告目录 _REPORTS_DIR 被硬编码为 ./reports/dailypaper。这会降低代码的灵活性和可测试性。建议通过配置（例如，从环境变量或统一的配置对象）来获取此路径，以便在不同环境中轻松修改。同时需要 import os。
| _REPORTS_DIR = Path("./reports/dailypaper") | |
| _REPORTS_DIR = Path(os.getenv("PAPERBOT_REPORTS_DIR", "./reports/dailypaper")) |
| try: | ||
| payload = json.loads(self._path.read_text(encoding="utf-8")) | ||
| if isinstance(payload, dict): | ||
| return payload | ||
| except Exception: | ||
| return {} |
There was a problem hiding this comment.
这里的异常捕获过于宽泛（except Exception），可能会掩盖一些意料之外的错误，例如权限问题。建议捕获更具体的异常，如 json.JSONDecodeError 和 IOError，并考虑在捕获后记录警告，以便于问题排查。
| try: | |
| payload = json.loads(self._path.read_text(encoding="utf-8")) | |
| if isinstance(payload, dict): | |
| return payload | |
| except Exception: | |
| return {} | |
| try: | |
| payload = json.loads(self._path.read_text(encoding="utf-8")) | |
| if isinstance(payload, dict): | |
| return payload | |
| except (json.JSONDecodeError, IOError): | |
| # logger.warning("Failed to load subscription store at %s", self._path) | |
| return {} |
There was a problem hiding this comment.
Pull request overview
This PR extends PaperBot’s “daily push” ecosystem by adding push-channel resiliency (retries/idempotency/error codes), a Telegram command webhook + subscription persistence, and a Hugging Face Daily Papers source upgrade (cache + trending + identity-based dedup), alongside Playwright E2E scaffolding and supporting docs/tests.
Changes:
- Add Playwright E2E setup for the Next.js web app (config, scripts, smoke/spec tests, ignored artifacts).
- Implement Telegram
/subscribe|/unsubscribe|/list|/todaycommand endpoint with a lightweight JSON subscription store. - Improve push delivery reliability (retry/backoff, idempotency, normalized error codes) and strengthen HF Daily ingestion (cache, trending ranking, arXiv identity inference, RRF dedup update).
Reviewed changes
Copilot reviewed 30 out of 34 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| web/playwright.config.ts | Adds Playwright test runner configuration + optional Next.js dev server startup. |
| web/package.json | Adds E2E test scripts and Playwright dev dependency. |
| web/package-lock.json | Locks @playwright/test / playwright dependencies. |
| web/e2e/smoke.spec.ts | Adds basic route smoke tests for core pages. |
| web/e2e/research.spec.ts | Adds minimal E2E assertions for the research page. |
| web/e2e/papers.spec.ts | Adds minimal E2E assertions for the papers page. |
| web/README.md | Updates product feature naming (“AgentSwarm Studio”). |
| web/.gitignore | Ignores Playwright output folders (reports, traces, blob-report). |
| tests/unit/test_push_commands_route.py | Adds unit tests for Telegram command endpoint behaviors. |
| tests/unit/test_push_channel_e2e.py | Adds mocked “all channels” push E2E tests with retry/error mapping. |
| tests/unit/test_paper_search_service_rrf.py | Adds RRF dedup test preferring shared arXiv identity across sources. |
| tests/unit/test_hf_daily_papers_connector.py | Adds connector tests for cache, trending modes, and graceful degradation. |
| tests/unit/test_hf_daily_adapter.py | Adds adapter tests for daily/trending methods and arXiv identity inference. |
| tests/unit/test_audit_daily_push_reports.py | Adds tests for new audit script helpers (markdown + CSV outputs). |
| tests/unit/test_apprise_notifier.py | Extends notifier tests for retry behavior, error-code mapping, idempotency. |
| src/paperbot/infrastructure/push/telegram_subscription_store.py | Implements file-backed Telegram subscription persistence. |
| src/paperbot/infrastructure/push/apprise_notifier.py | Adds retry/backoff, idempotency fingerprinting, and error-code mapping. |
| src/paperbot/infrastructure/push/init.py | Exports TelegramSubscriptionStore from push package. |
| src/paperbot/infrastructure/connectors/hf_daily_papers_connector.py | Adds cache/metrics, trending ranking, and degraded-mode handling. |
| src/paperbot/infrastructure/adapters/hf_daily_adapter.py | Adds daily/trending adapter methods + arXiv identity extraction from external URLs. |
| src/paperbot/application/services/paper_search_service.py | Makes dedup key prefer stable identities (DOI/arXiv/etc.) over title hash. |
| src/paperbot/api/routes/push_commands.py | Adds Telegram command webhook endpoint for subscriptions + /today titles. |
| src/paperbot/api/routes/init.py | Registers new push_commands route module export. |
| src/paperbot/api/main.py | Includes the new Push router under /api. |
| scripts/audit_daily_push_reports.py | Adds a CLI tool to sample and audit digest-card coverage across reports. |
| docs/runbooks/daily_push_ops.md | Adds ops runbook for channel config, Telegram webhook, audit, and triage. |
| docs/proposals/agentswarm-design.md | Adds AgentSwarm design proposal document. |
| docs/archive/issues/191-dashboard-digest-card-enhancement.md | Archives historical issue draft. |
| docs/archive/issues/190-settings-push-channel-ui.md | Archives historical issue draft. |
| docs/archive/README.md | Adds documentation archive index and conventions. |
| docs/archive/PROJECT_ISSUE_BACKLOG_legacy.md | Archives legacy backlog as historical reference. |
| docs/HF_DAILY_SOURCE.md | Documents HF Daily connector/adapter APIs, cache, and metrics. |
| docs/AGENTSWARM_TODO.md | Renames/updates TODO doc title to AgentSwarm Studio. |
| README.md | Updates top-level README to reflect AgentSwarm + repro context endpoints and docs links. |
Files not reviewed (1)
- web/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| score, record = candidate | ||
| date_key = _record_date(record) | ||
| if mode == "new": | ||
| return (date_key.timestamp(), date_key, record.upvotes) |
There was a problem hiding this comment.
In _candidate_sort_key, the mode == "new" branch calls date_key.timestamp(). When _record_date() falls back to datetime.min (e.g., missing/invalid submitted_on_daily_at and published_at), datetime.min.timestamp() can raise on some platforms/Python builds (out-of-range timestamp). Consider sorting directly on the datetime value (or guarding timestamp() with a safe fallback) to avoid hard failures when dates are missing.
| return (date_key.timestamp(), date_key, record.upvotes) | |
| try: | |
| date_ts = date_key.timestamp() | |
| except (OverflowError, OSError, ValueError): | |
| # Fallback for out-of-range or otherwise invalid datetimes | |
| # (e.g., datetime.min on platforms where .timestamp() fails). | |
| date_ts = float("-inf") | |
| return (date_ts, date_key, record.upvotes) |
| ); | ||
| const count = await importBtn.count(); | ||
| // At least some action button should be present | ||
| expect(count).toBeGreaterThanOrEqual(0); |
There was a problem hiding this comment.
expect(count).toBeGreaterThanOrEqual(0) is always true (locator counts are never negative), so this test will never fail even if no import/export actions exist. If the intent is to assert at least one relevant button/link is present, use a > 0 assertion (or assert on a specific expected control).
| expect(count).toBeGreaterThanOrEqual(0); | |
| expect(count).toBeGreaterThan(0); |
| details.append( | ||
| { | ||
| "url": channel.url, | ||
| "ok": False, |
There was a problem hiding this comment.
In the except path for per-channel digest pushes, the details row omits the attempts field, while the success and retry-handled failure paths always include it. To keep the result schema consistent (and avoid downstream consumers having to special-case), include attempts (e.g., 1 or 0 depending on where it failed) and consider also including skipped: False for symmetry with the idempotency path.
| "ok": False, | |
| "ok": False, | |
| "attempts": 0, | |
| "skipped": False, |
| Inspect per-channel payload from push result: | ||
| - `ok`: final status | ||
| - `attempts`: retry attempts used | ||
| - `error_code`: normalized code (`rate_limited`, `timeout`, `auth_failed`, `downstream_unavailable`) |
There was a problem hiding this comment.
The runbook lists a limited set of error_code values, but AppriseNotifier._map_error_code() and the idempotency path can also emit network_error, delivery_failed, and idempotent_replay (and successful sends currently set error_code to an empty string). Please update this section to reflect the full set of codes that can appear in push_daily_digest() results so operators can triage consistently.
| - `error_code`: normalized code (`rate_limited`, `timeout`, `auth_failed`, `downstream_unavailable`) | |
| - `error_code`: normalized code; one of `""` (success), `rate_limited`, `timeout`, `auth_failed`, `downstream_unavailable`, `network_error`, `delivery_failed`, `idempotent_replay` |
* Feat/daily push epic 179 (#218) * feat(push): implement Epic #179 — daily push optimization Multi-channel push infrastructure with structured digest cards: - #187: Integrate HF upvotes into Judge scoring prompt - #180: Add digest_card extraction (highlight/method/finding/tags) with new LLM prompt, email template rendering, markdown output - #181: MinerU Cloud API client for PDF figure extraction - #182: Apprise multi-channel push layer with YAML config - #183-185: Channel formatters (Telegram MarkdownV2, Discord Rich Embed, WeCom markdown+news, Feishu/Lark interactive card) - #186: RSS 2.0 + Atom feed endpoints (/api/feed/daily.xml, .atom) Also creates follow-up issues #190 (Push Channel Settings UI) and #191 (Dashboard Digest Card Enhancement). 56 new tests, all passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: integrate MinerU figure extraction into CLI and ARQ worker daily-paper workflow Refs #179 * fix(daily-push): enable MinerU flags in ARQ cron payload (refs #181) * feat(push): wire channel formatters into apprise digest delivery (refs #182 #183 #184 #185) * feat(feed): add keyword RSS, feed cache, and figure enclosure support (refs #186) * feat(mineru): add extraction result cache for daily push figure pipeline (refs #181) * feat(push): add telegram bot command subscription endpoint (refs #191) * fix(push): add retry/backoff for channel delivery failures (refs #191) * feat(hf): add trending modes, cache, and arxiv identity dedupe (refs #193) * feat(push): add idempotency and error-code mapping for channel delivery (refs #191) * test(push): add mock channel e2e and failure-injection coverage (refs #191) * docs(push): add audit script and ops runbook for acceptance closure (refs #192) * docs: update README for AgentSwarm execution transition * docs: archive stale docs and rename AgentSwarm todo * docs: add AgentSwarm design proposal + web Playwright e2e setup - Add docs/proposals/agentswarm-design.md covering multi-agent orchestration architecture, adapter pattern, session management, skills ecosystem, and OpenCode/OpenClaw/oMo integration (refs #197, #214) - Add Playwright e2e test scaffolding for web dashboard (smoke, papers, research specs) - Update web/.gitignore for Playwright artifacts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors (#223) * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors - 支持内联标题/摘要，以绕过 PaperInputRouter 获取 studio 论文 ID - 使上下文包生成 SSE 流期间的数据库持久化操作不再致命 - 从库中导入重复论文时显示错误消息 - 将左侧文件面板宽度缩小至约 18%（默认宽度）；移除默认模板文件 - 为 Claude CLI 添加 --verbose 标志，以兼容 stream-json 输出 Related to #222 * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors - 修改Gemini AI review代码问题 * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors - 修改Gemini AI review代码问题 --------- Co-authored-by: Jerry Zhang <1772030600@qq.com> * fix(web): correct package.json scripts JSON syntax * chore(gitignore): block nested .env secrets and keep examples * docs(env): add MinerU key and figure extraction env templates * refactor(mineru): standardize on v4 task API for daily push (#229) * fix(mineru): support v4 async task extraction flow * refactor(mineru): standardize on v4 task API for daily push (#179) * feat(push): render MinerU main figure in email via inline fallback * docs(readme): update email push demo screenshot * docs(readme): refresh Phase 6 daily push descriptions * fix(security): harden runbook allowed-dir path validation * fix(review): resolve security and reliability findings from AI review * fix(codeql): harden studio chat project_dir normalization --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Boyu liu <oor2020@163.com>
* Feat/daily push epic 179 (#218) * feat(push): implement Epic #179 — daily push optimization Multi-channel push infrastructure with structured digest cards: - #187: Integrate HF upvotes into Judge scoring prompt - #180: Add digest_card extraction (highlight/method/finding/tags) with new LLM prompt, email template rendering, markdown output - #181: MinerU Cloud API client for PDF figure extraction - #182: Apprise multi-channel push layer with YAML config - #183-185: Channel formatters (Telegram MarkdownV2, Discord Rich Embed, WeCom markdown+news, Feishu/Lark interactive card) - #186: RSS 2.0 + Atom feed endpoints (/api/feed/daily.xml, .atom) Also creates follow-up issues #190 (Push Channel Settings UI) and #191 (Dashboard Digest Card Enhancement). 56 new tests, all passing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: integrate MinerU figure extraction into CLI and ARQ worker daily-paper workflow Refs #179 * fix(daily-push): enable MinerU flags in ARQ cron payload (refs #181) * feat(push): wire channel formatters into apprise digest delivery (refs #182 #183 #184 #185) * feat(feed): add keyword RSS, feed cache, and figure enclosure support (refs #186) * feat(mineru): add extraction result cache for daily push figure pipeline (refs #181) * feat(push): add telegram bot command subscription endpoint (refs #191) * fix(push): add retry/backoff for channel delivery failures (refs #191) * feat(hf): add trending modes, cache, and arxiv identity dedupe (refs #193) * feat(push): add idempotency and error-code mapping for channel delivery (refs #191) * test(push): add mock channel e2e and failure-injection coverage (refs #191) * docs(push): add audit script and ops runbook for acceptance closure (refs #192) * docs: update README for AgentSwarm execution transition * docs: archive stale docs and rename AgentSwarm todo * docs: add AgentSwarm design proposal + web Playwright e2e setup - Add docs/proposals/agentswarm-design.md covering multi-agent orchestration architecture, adapter pattern, session management, skills ecosystem, and OpenCode/OpenClaw/oMo integration (refs #197, #214) - Add Playwright e2e test scaffolding for web dashboard (smoke, papers, research specs) - Update web/.gitignore for Playwright artifacts Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors (#223) * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors - 支持内联标题/摘要，以绕过 PaperInputRouter 获取 studio 论文 ID - 使上下文包生成 SSE 流期间的数据库持久化操作不再致命 - 从库中导入重复论文时显示错误消息 - 将左侧文件面板宽度缩小至约 18%（默认宽度）；移除默认模板文件 - 为 Claude CLI 添加 --verbose 标志，以兼容 stream-json 输出 Related to #222 * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors - 修改Gemini AI review代码问题 * fix(studio): fix context pack generation, duplicate import UX, and CLI chat errors - 修改Gemini AI review代码问题 --------- Co-authored-by: Jerry Zhang <1772030600@qq.com> * fix(web): correct package.json scripts JSON syntax * chore(gitignore): block nested .env secrets and keep examples * docs(env): add MinerU key and figure extraction env templates * refactor(mineru): standardize on v4 task API for daily push (#229) * fix(mineru): support v4 async task extraction flow * refactor(mineru): standardize on v4 task API for daily push (#179) * feat(push): render MinerU main figure in email via inline fallback * docs(readme): update email push demo screenshot * docs(readme): refresh Phase 6 daily push descriptions * fix(security): harden runbook allowed-dir path validation * fix(review): resolve security and reliability findings from AI review * fix(codeql): harden studio chat project_dir normalization * docs(readme): restructure README for professional presentation - Add centered hero header with badges (CI, Roadmap, License, Python, Next.js) - Replace verbose 18-row feature table with categorized bullet lists - Move Roadmap Phase 1-6 content to pinned issue #232 - Move module maturity matrix to Roadmap #232 - Consolidate screenshots into collapsible sections - Remove inline API endpoint table (40+ rows) and directory tree - Add Contributing section with Roadmap link - Streamline Getting Started with collapsible config details - Add repo topics for GitHub discoverability References: vLLM, Docling, PaperQA2, R2R README patterns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Boyu liu <oor2020@163.com>
No description provided.