feat(pipeline): 多模态识别管线(实验性),传统/多模态模式与 omni 凭据隔离 (#902) - #924
Open
H-Chris233 wants to merge 1 commit into
Open
Conversation
新增实验性「多模态识别管线」:高级设置开启后,服务页出现传统/多模态模式切换。多模态模式用单个模型(OpenAI 兼容 input_audio 或 Gemini 原生 generateContent)一步完成「提示词 + 音频 → 最终文本」,覆盖主听写、划词问答(OpenAI 兼容流式 / Gemini 一次性)、选区润色(omni 当纯文本 LLM)、Less Computer 转写。两套配置在凭据库中完全隔离(新增 omni 命名空间),切换不删数据、不回退传统配置。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
实现 issue #902:两段式「ASR 转写 + LLM 润色」无法解决专有名词误识别(ASR 不知道词典词,后处理模型无法从同音错词还原)。新增实验性「多模态识别管线」:用单个多模态模型一步完成「提示词(风格包 + 词典热词 + 工作语言)+ 音频 → 最终文本」。
改动
omni命名空间(apiKey/baseURL/model/temperature/extraHeaders),与传统 ASR/LLM 槽位互不读写;切换模式不删除另一套配置,切回即恢复;运行时只读当前模式,缺 omni 配置明确报错、不回退传统配置。input_audiobase64 WAV,SSE 流式)+ Gemini 原生 generateContent(inlineData audio/wav);复用现有 thinking 控制、SSRF 校验、SSE 解析与重试。pipelineMode字段,历史页显示「多模态」徽标并隐藏「重新转录」。验证
cargo check通过;cargo test --lib全绿(新增 omni 请求体、凭据隔离、模式门控测试);backend-tests 170 项通过。tsc、frontend-test-runner、npm run build(tsc + vite)全部通过。net::system_proxy_toggle...(并行下全局状态抖动,单独跑通过)、backend-tests Windows symlink PIN 测试(权限环境)。备注
实现期间
cargo fmt --all曾格式化一批未改动的 Rust 文件,已还原且未纳入本 PR(备份位于本机临时目录,可恢复)。