Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ Every item below is one more layer sedimented into a default — a capability yo
- 🎨 **Style Pack Marketplace** — browse, install, and like community **style packs** from the in-app Marketplace, and publish your own (custom system prompt per pack, switchable by hotkey). Backed by a moderated marketplace backend; uploads are reviewed before they go public.
- ⚡ **Streaming insertion** — polished text is written to the cursor character by character to reduce perceived latency, with an automatic one-shot-paste fallback. Toggle in Settings → Recording.
- **Toggle and push-to-talk** recording modes, plus a **MediaPlayPause trigger** so wired-earbud inline controls can start and stop recording. `Esc` cancels at any phase, including polish and insertion.
- **Cloud ASR**: Volcengine streaming ASR (bigasr), iFlytek realtime ASR (RTASR), Alibaba Cloud Bailian (classic realtime / Qwen3 realtime / Fun-ASR-Flash file transcription), StepFun StepAudio (batch + realtime), Zhipu GLM-ASR, Xiaomi MiMo ASR, ElevenLabs Scribe, OpenAI-compatible batch transcription (OpenAI Whisper / Groq / SiliconFlow SenseVoice / OpenRouter), and Apple Speech (macOS).
- **Cloud ASR**: Volcengine streaming ASR (bigasr), iFlytek realtime ASR (RTASR), Alibaba Cloud Bailian (classic realtime / Qwen3 realtime / Fun-ASR-Flash file transcription), StepFun StepAudio (batch + realtime), Zhipu GLM-ASR, Xiaomi MiMo ASR, ElevenLabs Scribe, OpenAI-compatible batch transcription (OpenAI Whisper / Groq / SiliconFlow SenseVoice / OpenRouter / ZenMux), and Apple Speech (macOS).
- **Local ASR**: bundled Qwen3-ASR (0.6B / 1.7B) via vendored `Open-Less/qwen-asr` (macOS); Windows Foundry Local Whisper and sherpa-onnx (experimental) variants.
- **Polish providers**: Ark (Volcengine), DeepSeek, OpenAI, Google Gemini, Codex OAuth, SiliconFlow, Atlas Cloud, Xiaomi MiMo, CometAPI, OpenRouter, Alibaba Cloud Coding Plan, CodingPlanX, MiniMax, and StepFun — plus any OpenAI-compatible endpoint you bring.
- **Four output modes**: raw, light polish, structured (**AI-prompt mode**), and formal. Plus a **translation hotkey** that converts speech directly into the configured target language ([#43](../../issues/43)).
Expand Down Expand Up @@ -338,7 +338,7 @@ Long-term reference rewrites are stored as `raw → polished → rule` triples a
The dictionary handles your proper nouns, product names, names of people, and new words. Today it supports:

- Manually adding the correct spelling, a category, and notes. You do not need to maintain misspellings or context hints.
- Enabled entries are sent to the ASR provider that supports hotwords (Volcengine `context.hotwords`, StepFun `hotwords`, Whisper-compatible `prompt`, Bailian vocabulary ID) so they are recognized correctly during transcription. iFlytek realtime ASR has no request-level hotword parameter — configure personalized hotwords in the iFlytek console instead.
- Enabled entries are sent to the ASR provider that supports hotwords (Volcengine `context.hotwords`, StepFun `hotwords`, Whisper-compatible `prompt` — except ZenMux, whose JSON protocol does not carry `prompt`/`hotwords`, Bailian vocabulary ID) so they are recognized correctly during transcription. iFlytek realtime ASR has no request-level hotword parameter — configure personalized hotwords in the iFlytek console instead.
- Entries are also injected into the polish prompt: the model decides per sentence whether to substitute. If "Cloud" clearly refers to the AI product `Claude` in context, it is corrected; if it genuinely means cloud computing, it is left as is.
- The app auto-learns candidate corrections such as `Claude`, `ChatGPT`, and `OpenLess` from your history and offers them later.

Expand Down
4 changes: 2 additions & 2 deletions README.zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,7 +180,7 @@ OpenLess 只做一件事:**把语音变成可用的书面文字(尤其是 AI 提
- 🎨 **风格包市场**——在应用内的 Marketplace 浏览、安装、点赞社区**风格包**,并发布自己的(每个包一套自定义系统提示词,可用快捷键切换)。由经过审核的市场后端支撑;上传内容公开前会经过审核。
- ⚡ **流式插入**——润色后的文本逐字符写入光标以降低感知延迟,并带有自动的一次性粘贴回退。可在 设置 → 录音 中切换。
- **切换式与按住说话(push-to-talk)** 两种录音模式,外加 **MediaPlayPause 触发**,让有线耳机的线控也能开始 / 停止录音。`Esc` 可在任意阶段取消,包括润色与插入。
- **云端 ASR**:Volcengine 流式 ASR(bigasr)、讯飞实时语音转写(RTASR)、阿里云百炼(经典实时 / Qwen3 实时 / Fun-ASR-Flash 录音文件)、阶跃星辰 StepAudio(批式 + 实时)、智谱 GLM-ASR、小米 MiMo ASR、ElevenLabs Scribe、OpenAI 兼容批量转写(OpenAI Whisper / Groq / 硅基流动 SenseVoice / OpenRouter),以及 Apple Speech(macOS)。
- **云端 ASR**:Volcengine 流式 ASR(bigasr)、讯飞实时语音转写(RTASR)、阿里云百炼(经典实时 / Qwen3 实时 / Fun-ASR-Flash 录音文件)、阶跃星辰 StepAudio(批式 + 实时)、智谱 GLM-ASR、小米 MiMo ASR、ElevenLabs Scribe、OpenAI 兼容批量转写(OpenAI Whisper / Groq / 硅基流动 SenseVoice / OpenRouter / ZenMux),以及 Apple Speech(macOS)。
- **本地 ASR**:通过 vendored 的 `Open-Less/qwen-asr` 内置 Qwen3-ASR(0.6B / 1.7B)(macOS);Windows 上的 Foundry Local Whisper 与 sherpa-onnx(实验性)变体。
- **润色提供方**:Ark(火山方舟)、DeepSeek、OpenAI、Google Gemini、Codex OAuth、硅基流动、Atlas Cloud、小米 MiMo、CometAPI、OpenRouter、阿里云 Coding Plan、CodingPlanX、MiniMax、StepFun,以及你自带的任意 OpenAI 兼容端点。
- **四种输出模式**:原文、轻度润色、结构化(**AI 提示词模式**)、正式。另有一个**翻译快捷键**,将语音直接转换为所配置的目标语言([#43](../../issues/43))。
Expand All @@ -192,7 +192,7 @@ OpenLess 只做一件事:**把语音变成可用的书面文字(尤其是 AI 提
- **Beta 频道(可选加入)**——设置 → 关于 → 加入 Beta 频道,可下载最新预发布版本进行手动安装。Beta 版本绝不会自动推送给 Stable 用户(见[贡献流程](#贡献流程))。
- **分发渠道**——从 [Releases](../../releases) 直接下载 DMG/EXE、Homebrew Cask(`brew install --cask openless`)、Windows 安装包。
- **单实例锁**——防止两个 OpenLess 进程争抢同一个快捷键边沿。
- 词典条目注入到支持热词的 ASR 提供方(Volcengine 的 `context.hotwords`、StepFun 的 `hotwords`、Whisper 兼容的 `prompt`、百炼的 vocabulary_id),并在润色时作为语义提示;命中次数按会话累计。讯飞实时语音转写标准版没有请求级热词参数,需在讯飞控制台配置个性化热词。
- 词典条目注入到支持热词的 ASR 提供方(Volcengine 的 `context.hotwords`、StepFun 的 `hotwords`、Whisper 兼容的 `prompt`(ZenMux 除外——其 JSON 协议不携带 `prompt`/`hotwords`)、百炼的 vocabulary_id),并在润色时作为语义提示;命中次数按会话累计。讯飞实时语音转写标准版没有请求级热词参数,需在讯飞控制台配置个性化热词。
- 平台原生全局快捷键:macOS 上为 CGEventTap,Windows 上为低级键盘钩子(`WH_KEYBOARD_LL`)。

## 下载与安装(终端用户)
Expand Down
168 changes: 168 additions & 0 deletions openless-all/app/src-tauri/src/asr/whisper.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,17 +22,28 @@ pub const PROMPT_CHAR_BUDGET: usize = 240;
/// 区切り文字(ASCII)。Whisper のトークナイザはどの言語でも安定して扱える。
const PROMPT_SEPARATOR: &str = ", ";

/// ZenMux 聚合平台的默认端点与模型(issue #837)。与前端 `ASR_PRESETS` 的
/// `zenmux` 条目保持一致;`read_whisper_credentials` 在 active 为 zenmux 且
/// 用户未填时回退到这里。
pub const ZENMUX_DEFAULT_ENDPOINT: &str = "https://zenmux.ai/api/v1";
pub const ZENMUX_DEFAULT_MODEL: &str = "qwen/qwen3-asr-flash";

/// `/audio/transcriptions` 请求体编码方式。
///
/// OpenAI 官方及多数兼容厂商用 `multipart/form-data`(file + model)。
/// OpenRouter 虽路径相同、也走 Bearer,但请求体是 `application/json`:
/// `{model, input_audio:{data:<base64 wav>, format:"wav"}}`(issue #582)。
/// ZenMux 与 OpenRouter 同形(issue #837),另支持可选的 `language` 与
/// `enable_itn`(数字归一化)字段,故单独一个变体,避免给 OpenRouter
/// 的请求体塞未知字段(部分实现会对未知字段 4xx)。
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum AsrRequestFormat {
/// `multipart/form-data`(既有行为,默认)。
Multipart,
/// OpenRouter `application/json` + base64 音频。
OpenRouterJson,
/// ZenMux `application/json` + base64 音频(`language` 可选、`enable_itn` 恒发)。
ZenMuxJson,
}

pub struct WhisperBatchASR {
Expand All @@ -52,6 +63,12 @@ pub struct WhisperBatchASR {
verbose_json: bool,
/// 请求体编码方式。默认 `Multipart`,OpenRouter 走 `OpenRouterJson`。
request_format: AsrRequestFormat,
/// ZenMux 的 `language` 字段(ISO 639-1,如 `zh`)。None = 不发送,服务端
/// 自动检测。仅 `ZenMuxJson` 编码下生效。
language: Option<String>,
/// ZenMux 的 `enable_itn` 字段(数字/单位归一化)。默认 true,与 ZenMux
/// 文档示例及中文 ASR 预期一致;仅 `ZenMuxJson` 编码下生效。
enable_itn: bool,
/// 一等 `hotwords` 参数(JSON 数组字符串)。StepFun 等厂商不认 `prompt`
/// (静默忽略),但提供专门的热词字段——用它词典才真正生效。空 = 不发。
hotwords: Vec<String>,
Expand All @@ -75,6 +92,8 @@ impl WhisperBatchASR {
max_chunk_duration_ms,
verbose_json,
request_format: AsrRequestFormat::Multipart,
language: None,
enable_itn: true,
hotwords: Vec::new(),
buffer: Mutex::new(Vec::new()),
}
Expand All @@ -94,6 +113,20 @@ impl WhisperBatchASR {
self
}

/// 设置 ZenMux 的 `language` 字段(ISO 639-1 码)。None = 不发送(自动检测)。
/// 仅 `ZenMuxJson` 编码下生效。
pub fn with_language(mut self, language: Option<String>) -> Self {
self.language = language;
self
}

/// 设置 ZenMux 的 `enable_itn`(数字归一化)开关,默认 true。仅 `ZenMuxJson`
/// 编码下生效。
pub fn with_enable_itn(mut self, enable_itn: bool) -> Self {
self.enable_itn = enable_itn;
self
}

/// Stop collecting audio, encode the buffer as WAV, and POST to the
/// Whisper transcriptions endpoint.
///
Expand Down Expand Up @@ -218,6 +251,29 @@ impl WhisperBatchASR {
}
request.json(&body)
}
AsrRequestFormat::ZenMuxJson => {
// ZenMux /audio/transcriptions(issue #837):application/json,
// 音频走标准 base64。语言跟随 OpenLess 工作语言映射;enable_itn
// 由设置页开关决定,恒显式发送。不带 multipart 专属字段。
let mut body = serde_json::json!({
"model": self.model,
"input_audio": {
"data": base64::engine::general_purpose::STANDARD.encode(&wav),
"format": "wav",
},
"enable_itn": self.enable_itn,
});
if let Some(language) = self.language.as_ref() {
if !language.trim().is_empty() {
body["language"] = serde_json::Value::String(language.trim().to_string());
}
}
let mut request = client.post(&url);
if !self.api_key.trim().is_empty() {
request = request.header("Authorization", format!("Bearer {}", self.api_key));
}
request.json(&body)
}
};

let resp = request
Expand Down Expand Up @@ -858,6 +914,118 @@ mod tests {
server.join().unwrap();
}

#[tokio::test]
async fn zenmux_format_posts_json_with_language_and_itn() {
// issue #837:ZenMuxJson 走 application/json + input_audio.data(base64),
// 语言有映射时发送、enable_itn 恒显式发送;响应仍按 {text} 解析。
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
listener.set_nonblocking(true).unwrap();
let addr = listener.local_addr().unwrap();
let server = thread::spawn(move || {
let deadline = Instant::now() + Duration::from_secs(5);
let mut stream = loop {
match listener.accept() {
Ok((stream, _)) => break stream,
Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => {
assert!(
Instant::now() < deadline,
"timed out waiting for ASR test request"
);
thread::sleep(Duration::from_millis(10));
}
Err(err) => panic!("accept ASR test request failed: {err}"),
}
};
stream.set_nonblocking(false).unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let request = read_http_request(&mut stream);
let request_text = String::from_utf8_lossy(&request);
let lower = request_text.to_ascii_lowercase();
assert!(request_text.starts_with("POST /audio/transcriptions HTTP/1.1"));
assert!(lower.contains("content-type: application/json"));
assert!(lower.contains("authorization: bearer key"));
assert!(request_text.contains("input_audio"));
assert!(request_text.contains(r#""format":"wav""#));
assert!(request_text.contains(r#""language":"zh""#));
assert!(request_text.contains(r#""enable_itn":true"#));
assert!(!lower.contains("multipart/form-data"));
write_json_response(&mut stream, r#"{"text":"zenmux ok"}"#);
});
let base_url = format!("http://{}", addr);

let asr = WhisperBatchASR::new(
"key".to_string(),
base_url,
"qwen/qwen3-asr-flash".to_string(),
None,
Some(30_000),
false,
)
.with_request_format(AsrRequestFormat::ZenMuxJson)
.with_language(Some("zh".to_string()))
.with_enable_itn(true);
let pcm = vec![0u8; 32_000 * 2];
asr.consume_pcm_chunk(&pcm);

let transcript = asr.transcribe().await.unwrap();
assert_eq!(transcript.text, "zenmux ok");
server.join().unwrap();
}

#[tokio::test]
async fn zenmux_format_omits_language_when_unset_and_sends_false_itn() {
// language 无映射(None)时不发送该字段;enable_itn=false 显式发送 false。
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
listener.set_nonblocking(true).unwrap();
let addr = listener.local_addr().unwrap();
let server = thread::spawn(move || {
let deadline = Instant::now() + Duration::from_secs(5);
let mut stream = loop {
match listener.accept() {
Ok((stream, _)) => break stream,
Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => {
assert!(
Instant::now() < deadline,
"timed out waiting for ASR test request"
);
thread::sleep(Duration::from_millis(10));
}
Err(err) => panic!("accept ASR test request failed: {err}"),
}
};
stream.set_nonblocking(false).unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let request = read_http_request(&mut stream);
let request_text = String::from_utf8_lossy(&request);
assert!(request_text.contains(r#""enable_itn":false"#));
assert!(!request_text.contains(r#""language""#));
write_json_response(&mut stream, r#"{"text":"zenmux no-lang ok"}"#);
});
let base_url = format!("http://{}", addr);

let asr = WhisperBatchASR::new(
"key".to_string(),
base_url,
"qwen/qwen3-asr-flash".to_string(),
None,
None,
false,
)
.with_request_format(AsrRequestFormat::ZenMuxJson)
.with_language(None)
.with_enable_itn(false);
let pcm = vec![0u8; 32_000 * 2];
asr.consume_pcm_chunk(&pcm);

let transcript = asr.transcribe().await.unwrap();
assert_eq!(transcript.text, "zenmux no-lang ok");
server.join().unwrap();
}

#[tokio::test]
async fn empty_api_key_omits_authorization_header() {
// openai-compatible 允许 API Key 留空(LAN 无鉴权端点):请求不得携带
Expand Down
Loading
Loading