feat(asr): 百炼 ASR 支持 fun-asr-flash 并统一三协议为单一入口 by bigsongeth · Pull Request #793 · Open-Less/openless

bigsongeth · 2026-07-13T04:27:22Z

User description

摘要

让阿里云百炼(DashScope)三种 ASR 协议都能用、并收成一个用户入口。

百炼同一厂商、同一把 key 下有三种互不兼容的线协议:

模型	协议	网关	形态
`fun-asr-realtime`	经典实时 WS(run-task/finish-task 双工 + PCM 帧)	…/api-ws/v1/inference/	边说边出
`qwen3-asr-flash-realtime`	OpenAI-Realtime 风格 WS	…/api-ws/v1/realtime	边说边出
`fun-asr-flash`	multimodal-generation HTTP 批量(POST JSON,URL/base64)	…/multimodal-generation/generation	说完整段转

此前只有前两个,且各是独立 provider、独立 key;fun-asr-flash 是唯一的 HTTP 批量协议,现有两条 WS 通道都接不了 → 用不了。本 PR 补齐第三条通道,并把三条收成一个「阿里云百炼」入口、按模型名自动路由协议。

修复 / 新增 / 改进

新增 asr/dashscope_multimodal.rs:multimodal-generation 批量 ASR 客户端(攒 PCM → WAV → base64 → POST → 多路径兜底解析响应),复用 mimo 的分片/CJK 拼接。
分发重构:ASR provider 分类收敛为 ActiveAsrProviderKind 单一来源,preflight_credential / configured_fields 等改为穷尽 match(去掉静默 else 兜底),新增通道时编译器强制补齐各分发点,消除「装完才发现某处漏了」。
统一入口:UI 三条百炼收成一个「阿里云百炼」+ 模型框(可「拉取模型」或手填任意 DashScope ASR 模型),后端 resolve_effective_asr_provider(model) 按模型名把 build / 验证路由到对应协议,endpoint 按协议自动推导;模型框下加实时 / 录音文件协议提示。

兼容

不包含:移动端 / Android 相关改动;不改版本号。
对现有用户:
- bailian(fun-asr-realtime)用户无感,默认仍走经典实时;
- bailian-qwen3-realtime 保留为隐藏别名,老配置照常工作(下拉仅对已在用者显示);
- bailian-fun-asr-flash 为本 PR 新增(此前无用户);
- 行为变化(需知会):统一入口下 qwen3 / fun-asr-flash 的 endpoint 一律用协议默认地址、忽略共用存储值——业务空间专属域名的高级用户在统一入口暂不能自定义 endpoint(仍可经隐藏别名 id 路径覆盖)。
构建 / 本地环境:mimo 的 split_pcm_by_duration / join_transcript_chunks 提为 pub(crate) 供复用(零行为改动);无新增依赖。

测试计划

命令:cargo test --lib → 结果:610 passed(对 beta 基线);新增 resolve_effective_asr_provider / preflight_credential / configured_fields / dashscope 客户端等单测。
命令:cargo check(macOS native)+ 前端 tsc(npm run build)→ 均通过(对 beta 基线);跨平台 Android/Win/Linux 未加 cfg 分支,交 CI 确认。
真机 API:fun-asr-flash-2026-06-15 multimodal-generation(URL 与 base64 两种输入)→ HTTP 200、转写正确、~600ms。
in-app 三协议听写冒烟(实时经典 / qwen3 / fun-asr-flash 各验证 + 听写)→ macOS 本地重装后手测通过。

PR Type

Enhancement, Bug fix

Description

Add DashScopeMultimodalASR for fun-asr-flash batch ASR
Refactor provider classification into exhaustive ActiveAsrProviderKind enum
Unify three DashScope protocols into single UI entry with model-based routing
Consolidate credential preflight and configured fields checks using new enum
Update i18n and UI for unified Bailian provider with protocol hints

Diagram Walkthrough

flowchart LR
  UI["Unified Bailian Provider"] --> Routing["Model-Based Router"]
  Routing --> Classic["fun-asr-realtime (WS)"]
  Routing --> Qwen3["qwen3-asr-flash-realtime (WS)"]
  Routing --> Flash["fun-asr-flash (HTTP)"]

File Walkthrough

Relevant files

Enhancement

7 files

dashscope_multimodal.rs `New batch ASR client for fun-asr-flash`	+412/-0
providers.rs `Update validation for unified Bailian`	+145/-10
asr_wiring.rs `Refactor preflight and build with effective provider`	+40/-20
dictation.rs `Route dictation sessions by effective provider`	+81/-4
qa_session.rs `Handle DashScopeMultimodal during QA`	+32/-0
Overview.tsx `Derive ASR name keys from shared presets`	+6/-13
ProvidersSection.tsx `Unified Bailian UI with model protocol hints`	+82/-42

Refactoring

4 files

mimo.rs `Reuse split/join functions for multimodal`	+6/-2
credentials.rs `Use ActiveAsrProviderKind for credential checks`	+16/-13
coordinator.rs `Add enum routing and endpoint derivation`	+355/-17
shared.tsx `Move ASR presets to single source of truth`	+42/-16

Configuration changes

1 files

mod.rs `Register new dashscope_multimodal module`	+2/-0

Miscellaneous

1 files

resources.rs `Cancel for DashScopeMultimodal`	+1/-0

Documentation

5 files

en.ts `Add UI labels for unified Bailian`	+3/-0
ja.ts `Add UI labels for unified Bailian`	+3/-0
ko.ts `Add UI labels for unified Bailian`	+3/-0
zh-CN.ts `Add UI labels for unified Bailian`	+3/-0
zh-TW.ts `Add UI labels for unified Bailian`	+3/-0

github-actions · 2026-07-13T04:30:46Z

PR Reviewer Guide 🔍

(Review updated until commit `fec5331`)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ No major issues detected

H-Chris233 · 2026-07-13T14:00:59Z

开始审查

github-actions · 2026-07-13T15:05:28Z

Persistent review updated to latest commit 5f4078b

LauraGPT

整体协议主路径、跨平台接线和测试覆盖做得很完整。我按阿里云当前公开 API 文档逐项核对后，发现 3 个 CI 不会捕获、但会影响真实用户的边界；已在对应行留下可复现说明和最小修复方向。

LauraGPT · 2026-07-13T19:26:01Z

+        Ok(crate::asr::bailian::PROVIDER_ID.to_string())
+    } else if model.starts_with("qwen3-asr-flash-realtime") {
+        Ok(crate::asr::qwen_realtime::PROVIDER_ID.to_string())
+    } else if model.starts_with("fun-asr-flash") {


模型前缀会把实时模型误送到批量 HTTP。 官方模型表中 fun-asr-flash-8k-realtime / fun-asr-flash-8k-realtime-2026-01-28 是实时 WebSocket 模型，但它们同样满足这里的 starts_with("fun-asr-flash")，因此手填后会被路由到 multimodal-generation，协议必然不匹配。UI 明确允许手填 DashScope 模型，所以这条路径可达。建议先显式识别并拒绝（当前录音链是 16 kHz，而该系列要求 8 kHz），同时把批量 Flash 收紧为当前受支持模型的 allowlist，并加一个 fun-asr-flash-8k-realtime 不得进入 DashScopeMultimodal 的回归测试。官方模型矩阵：https://help.aliyun.com/en/model-studio/asr-model/

LauraGPT · 2026-07-13T19:26:01Z

+                DashScope ASR 模型),协议按模型名在后端自动路由,接口地址随协议自动推导,
+                故不暴露 endpoint 字段。模型框下一行提示当前模型是「实时」还是「录音文件」,
+                解决三种模型看不出区别的问题。 */}
+            {!unifiedBailian && (


隐藏 endpoint 后，统一入口实际上只能使用中国区默认域名。 后端在 unified Bailian 下也会忽略已存 endpoint，固定改用 dashscope.aliyuncs.com；但北京与新加坡 API Key 分区，官方要求新加坡使用 dashscope-intl.aliyuncs.com 或 {WorkspaceId}.ap-southeast-1.maas.aliyuncs.com，Qwen Realtime 和 Fun-ASR-Flash 都如此。新用户又无法选择隐藏别名，因此新加坡用户没有任何可配置路径。建议保留 endpoint/区域选择，按所选模型只规范化 scheme/path、不要丢掉 host；至少补中国区和新加坡区 host 的路由测试。官方 Flash endpoint：https://help.aliyun.com/en/model-studio/fun-asr-recorded-speech-recognition-http-api ，Qwen Realtime endpoint：https://help.aliyun.com/en/model-studio/qwen-asr-realtime-interaction-process

LauraGPT · 2026-07-13T19:26:01Z

+        "format": "wav",
+        "sample_rate": "16000",
+    });
+    if let Some(vocabulary_id) = vocabulary_id.filter(|value| !value.trim().is_empty()) {


fun-asr-flash-2026-06-15 的同步 HTTP 参数中没有 vocabulary_id。 官方文档只列出 format 和可选 sample_rate；该模型的准确率增强方式是把领域词作为 input.messages[].content[{type:"input_text"}] 上下文，而不是自定义热词 ID。当前统一入口会继承实时 Fun-ASR 用户已有的 asr.vocabulary_id 并注入这里，可能被服务端拒绝或静默忽略，同时 UI 会误导用户认为它生效。建议 Flash 路径不发送该字段并隐藏对应输入；若要复用词表，可把应用内短语按 400 字限制映射为 input_text 上下文。官方请求参数：https://help.aliyun.com/en/model-studio/fun-asr-recorded-speech-recognition-http-api

H-Chris233 · 2026-07-14T02:12:06Z

整体协议主路径、跨平台接线和测试覆盖做得很完整。我按阿里云当前公开 API 文档逐项核对后，发现 3 个 CI 不会捕获、但会影响真实用户的边界；已在对应行留下可复现说明和最小修复方向。

问题已收到，开始解决

github-actions · 2026-07-14T02:24:20Z

Persistent review updated to latest commit 39eddf9

# Conflicts: # openless-all/app/src-tauri/src/coordinator/dictation.rs

github-actions · 2026-07-14T02:39:14Z

Persistent review updated to latest commit fec5331

feat(asr): 百炼 ASR 支持 fun-asr-flash 并统一三协议为单一入口

ec31219

github-actions Bot added the Review effort 3/5 label Jul 13, 2026

fix(asr): harden DashScope model routing and transcription

5f4078b

github-actions Bot added Review effort 4/5 and removed Review effort 3/5 labels Jul 13, 2026

LauraGPT reviewed Jul 13, 2026

View reviewed changes

fix(asr): address Bailian protocol review feedback

39eddf9

github-actions Bot added Review effort 3/5 and removed Review effort 4/5 labels Jul 14, 2026

Merge remote-tracking branch 'upstream/beta' into codex/pr-793-fix

fec5331

# Conflicts: # openless-all/app/src-tauri/src/coordinator/dictation.rs

H-Chris233 merged commit 6e79a0c into Open-Less:beta Jul 14, 2026
5 checks passed

bigsongeth deleted the feat/bailian-asr-unify branch July 22, 2026 10:55

Navigation Menu

Uh oh!

feat(asr): 百炼 ASR 支持 fun-asr-flash 并统一三协议为单一入口 - #793

feat(asr): 百炼 ASR 支持 fun-asr-flash 并统一三协议为单一入口#793
H-Chris233 merged 4 commits into
Open-Less:betafrom
bigsongeth:feat/bailian-asr-unify

bigsongeth commented Jul 13, 2026 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

H-Chris233 commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

LauraGPT left a comment

Uh oh!

LauraGPT Jul 13, 2026

Uh oh!

LauraGPT Jul 13, 2026

Uh oh!

LauraGPT Jul 13, 2026

Uh oh!

H-Chris233 commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

bigsongeth commented Jul 13, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

User description

摘要

修复 / 新增 / 改进

兼容

测试计划

PR Type

Description

Diagram Walkthrough

File Walkthrough

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Reviewer Guide 🔍

(Review updated until commit fec5331)

Uh oh!

H-Chris233 commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

LauraGPT left a comment

Choose a reason for hiding this comment

Uh oh!

LauraGPT Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

LauraGPT Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

LauraGPT Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

H-Chris233 commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bigsongeth commented Jul 13, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading

(Review updated until commit `fec5331`)