From b17994e75a37f1ae0ea0a5ba4bf739e68f72f0e4 Mon Sep 17 00:00:00 2001 From: Colin L Date: Wed, 29 Apr 2026 19:27:04 -0700 Subject: [PATCH] docs: document the reasoning-model gotcha in README Several Nebius-hosted models (Kimi-K2.5, DeepSeek-V3.2, GLM-5, Qwen3 thinking variants) emit hidden reasoning tokens before producing visible output. Those tokens count against max_tokens. Empirically, a 64-token request to Kimi-K2.5 comes back with empty content because the entire budget is consumed by reasoning, but no error is raised and the response looks structurally fine. Adds a "Reasoning models" subsection under Configure that lists the known reasoning-style models, explains the implication for max_tokens budgets, and points users at GET /v1/models for verification. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/README.md b/README.md index bcac5b9e..e316e7df 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,31 @@ VISION_MODEL="Qwen/Qwen2.5-VL-72B-Instruct" STRIP_IMAGE_CONTEXT="true" ``` +#### Reasoning models + +Several Nebius-hosted models emit *hidden* reasoning tokens before producing +visible output. These tokens count against `max_tokens`, so very small budgets +can return empty content. Known reasoning-style models on Nebius: + +- `moonshotai/Kimi-K2.5` +- `deepseek-ai/DeepSeek-V3.2` +- `zai-org/GLM-5` +- `Qwen/Qwen3-Next-80B-A3B-Thinking` +- `Qwen/Qwen3-235B-A22B-Thinking-2507-fast` + +Implication: keep `MAX_TOKENS_LIMIT` and per-request `max_tokens` generous +(>=4096 is recommended; 16k+ is safer for agentic tool-use loops). If a +reasoning model returns empty text with a non-zero `output_tokens` count, the +budget was exhausted by reasoning before any visible output was produced — +raise the limit and retry. + +Verify model availability and pick alternatives at: + +```bash +curl -s https://api.tokenfactory.nebius.com/v1/models \ + -H "Authorization: Bearer $OPENAI_API_KEY" | jq '.data[].id' +``` + ### Run ```bash