Skip to content

refactor(mineru): standardize on v4 task API for daily push #229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jerry609 merged 2 commits into from
Mar 4, 2026
Merged

refactor(mineru): standardize on v4 task API for daily push #229

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fd7e1c9
fix(mineru): support v4 async task extraction flow
jerry609 Mar 4, 2026
70efd76
refactor(mineru): standardize on v4 task API for daily push (#179)
jerry609 Mar 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
| **文献卡片** | Structured Card（LLM 提取 method/dataset/conclusion/limitations），懒加载 + DB 缓存 |
| **导出增强** | BibTeX/RIS/Markdown/CSL-JSON（Zotero 原生导入），Next.js proxy route 修复 |
| **写作辅助** | Related Work 草稿生成（基于 saved papers + topic），[AuthorYear] 引用格式，一键复制 |
| **每日推送** | DailyPaper 生成后自动推送摘要到 Email/Slack/钉钉，支持 API 手动触发和 ARQ Cron 定时推送；MinerU 图表提取与 Apprise 多渠道（Telegram/Discord/企业微信/飞书/RSS）待集成 |
| **每日推送** | DailyPaper 生成后自动推送摘要到 Email/Slack/钉钉，支持 API 手动触发和 ARQ Cron 定时推送；已集成 MinerU 图表提取与 Apprise 多渠道（Telegram/Discord/企业微信/飞书/RSS） |
| **Model Provider** | 多 LLM 提供商管理（OpenAI/Anthropic/OpenRouter/Ollama），API Key Keychain 安全存储，任务级路由，连接测试 |
| **Deadline Radar** | 会议截止日期追踪，CCF 分级过滤，Research Track 关键词匹配 |
| **论文发现** | 种子论文扩展（引用/被引/共作者），Discovery Graph 可视化，论文集合（Collections）管理 |
Expand All @@ -35,7 +35,7 @@
| DailyPaper | ✅ 可用 | `/research/paperscool/daily` | `daily-paper` | 报告生成 + LLM 增强 + Judge + 保存，完整可用 |
| LLM-as-Judge | ✅ 可用 | `/research/paperscool/analyze` | `--with-judge` | 5 维评分 + 多轮校准 + 推荐分级 + Token Budget，SSE 增量推送 |
| Analyze SSE | ✅ 可用 | `/research/paperscool/analyze` | — | Judge / Trend / Insight 三通道 SSE 流式，前端逐卡片渲染 |
| Push/Notify | 🟡 基本可用 | `/research/paperscool/daily` | `--notify` | Email/Slack/钉钉 已落地；Apprise 多渠道（Telegram/Discord/企业微信/飞书/RSS）+ MinerU 图表提取待集成 |
| Push/Notify | ✅ 可用 | `/research/paperscool/daily` | `--notify` | Email/Slack/钉钉 + Apprise 多渠道（Telegram/Discord/企业微信/飞书/RSS）+ MinerU 图表提取已落地 |
| 学者追踪 | 🟡 基本可用 | `/track` | `track` | 多 Agent 管线 + PIS 评分完整；依赖 Semantic Scholar API Key |
| 深度评审 | 🟡 基本可用 | `/review` | `review` | 模拟同行评审流程完整；输出质量取决于 LLM 后端配置 |
| Paper2Code | 🟡 基本可用 | `/gen-code`（兼容） + `/research/repro/context/*` | `gen-code` | 编排 + RAG + CodeMemory 完整；执行层计划迁移为 AgentSwarm/Codex 专业执行器 |
Expand Down Expand Up @@ -463,7 +463,7 @@ DB 持久化（统一主数据模型 Paper/Scholar/Event/Run）、任务队列/

### Phase 6 — 每日推送优化

MinerU PDF 图表提取（主方法图自动识别）、推送内容增强（一句话总结 + 结构化摘要）、Apprise 多渠道统一推送层（Telegram/Discord/企业微信/飞书/RSS）、HuggingFace Daily Papers API 数据源接入。
MinerU PDF 图表提取（主方法图自动识别）、推送内容增强（一句话总结 + 结构化摘要）、Apprise 多渠道统一推送层（Telegram/Discord/企业微信/飞书/RSS）、HuggingFace Daily Papers API 数据源接入已完成。

## 文档索引

Expand Down
7 changes: 7 additions & 0 deletions env.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@ OPENAI_BASE_URL=
SEMANTIC_SCHOLAR_API_KEY=
GITHUB_TOKEN=
MINERU_API_KEY=
# MinerU Cloud API v4 async task endpoint
MINERU_API_BASE_URL=https://mineru.net/api/v4
# MinerU model_version, e.g. vlm / pipeline
MINERU_MODEL_VERSION=vlm
# Max wait for task polling in seconds
MINERU_MAX_WAIT_SECONDS=180
# MinerU Cloud limits: URL input only, <=200MB, <=600 pages, github/aws URL may timeout

# CCS download (optional, ACM access URL)
ACM_LIBRARY_URL=
Expand Down
23 changes: 21 additions & 2 deletions src/paperbot/application/workflows/dailypaper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,23 @@
SUPPORTED_LLM_FEATURES = ("summary", "trends", "insight", "relevance", "digest_card")


def _is_publishable_figure_url(url: str) -> bool:
u = (url or "").strip().lower()
if not u.startswith(("http://", "https://")):
return False
if u.endswith(".zip") or ".zip#" in u or ".zip?" in u:
return False
return True


def extract_figures_for_report(
report: Dict[str, Any],
*,
api_key: str = "",
max_items: int = 5,
base_url: str = "",
model_version: str = "vlm",
max_wait_seconds: float = 180.0,
) -> Dict[str, Any]:
"""Optionally extract figures from top papers using MinerU Cloud API.

Expand All @@ -32,7 +44,14 @@ def extract_figures_for_report(

from paperbot.infrastructure.extractors.mineru_client import MineruClient

client = MineruClient(api_key=api_key)
client_kwargs: Dict[str, Any] = {
"api_key": api_key,
"model_version": model_version,
"max_wait_seconds": max_wait_seconds,
}
if (base_url or "").strip():
client_kwargs["base_url"] = base_url.strip()
client = MineruClient(**client_kwargs)
enriched = copy.deepcopy(report)
count = 0

Expand All @@ -49,7 +68,7 @@ def extract_figures_for_report(
item["figures"] = [
{"url": f.url, "caption": f.caption, "page": f.page} for f in figures[:5]
]
if main:
if main and _is_publishable_figure_url(main.url):
item["main_figure"] = {"url": main.url, "caption": main.caption}
count += 1

Expand Down
Loading
Loading