Add vlm_vision_weights helper and adopt it in VLM vision sub-models - #336
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors common VLM vision-tower weight preprocessing into a shared helper to remove duplicated boilerplate across vision sub-models, while preserving existing behavior and adding focused unit coverage.
Changes:
- Added
vlm_vision_weights(state_dict, prefixes)tosrc/mobius/_weight_utils.pyto (a) filter by vision-related prefixes and (b) renamemlp.fc1/fc2→mlp.up_proj/down_projto matchFCMLPinitializer naming. - Updated LLaVA, Gemma3, and Mllama vision sub-models to delegate their
preprocess_weights()implementations to the new helper with the appropriate prefix tuple. - Added unit tests covering filter+rename behavior, single-prefix usage, and the no-match case.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/mobius/models/mllama.py | Replaces local vision weight filtering/rename loop with vlm_vision_weights(..., ("vision_model.",)). |
| src/mobius/models/llava.py | Replaces local vision weight filtering/rename loop with vlm_vision_weights(..., ("vision_tower.", "multi_modal_projector.")). |
| src/mobius/models/gemma3.py | Replaces local vision weight filtering/rename loop with vlm_vision_weights(..., ("vision_tower.", "multi_modal_projector.")). |
| src/mobius/_weight_utils.py | Introduces the shared vlm_vision_weights helper (filter + fc1/fc2 rename). |
| src/mobius/_weight_utils_test.py | Adds unit tests for vlm_vision_weights covering key behaviors and edge cases. |
Performance Comparison
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
The vision sub-models of LLaVA, Gemma3 and Mllama hand-wrote the same
preprocess_weights body: filter to the vision-tower prefixes and rename
the HF vision MLP projections (mlp.fc1 → mlp.up_proj, mlp.fc2 →
mlp.down_proj) to match our FCMLP component naming.
Add `vlm_vision_weights(state_dict, prefixes)` to `_weight_utils.py`
next to the existing `vlm_decoder_weights` / `vlm_embedding_weights`
helpers, and adopt it in those three models. The only per-model
difference is the prefix tuple (gemma3/llava use
("vision_tower.", "multi_modal_projector."); mllama uses
("vision_model.",)). Behaviour-preserving.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchu/vlm-vision-weights-helper
branch
from
June 5, 2026 15:39
5df878f to
e1781d5
Compare
justinchuby
added a commit
that referenced
this pull request
Jun 5, 2026
Reflect the public-API and DRY changes shipped in PRs #333/#334/#336: - build_from_gguf is now a top-level export; update the import examples in docs/api/build_from_gguf.md and docs/getting-started.md to `from mobius import build_from_gguf`. - weight-name-alignment skill: add a 'Shared helpers' section documenting the _weight_utils rename helpers, including the new rename_weight_keys and vlm_vision_weights, so future model work reuses them instead of hand-written rename loops. - multimodal-models skill: point at the shared vlm_* weight helpers. - moe-models skill: note Qwen35MoEBlock subclasses Qwen2MoELayer and fix the class file paths (models/qwen.py -> models/qwen35.py). Depends on #333 (build_from_gguf export), #334 (rename_weight_keys) and #336 (vlm_vision_weights) landing first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Jun 5, 2026
Updates outdated docs and agent skills to match the public-API + DRY refactor shipped across the themed PRs. ## Changes - **`build_from_gguf` import** — now a top-level export, so `docs/api/build_from_gguf.md` and `docs/getting-started.md` use `from mobius import build_from_gguf` instead of the internal `mobius.integrations.gguf` path. - **weight-name-alignment skill** — new *Shared helpers* section documenting the `_weight_utils` rename helpers, including the new `rename_weight_keys` and `vlm_vision_weights`, so future model work reuses them instead of hand-written rename loops. - **multimodal-models skill** — points at the shared `vlm_*` weight helpers. - **moe-models skill** — notes `Qwen35MoEBlock` now subclasses `Qwen2MoELayer`, and fixes stale class file paths (`models/qwen.py` → `models/qwen35.py`). ## Dependencies Depends on the API PRs landing first (the helpers/exports documented here only exist on those branches): - #333 — `build_from_gguf` top-level export - #334 — `rename_weight_keys` - #336 — `vlm_vision_weights` Docs-only; no code or tests affected. --------- Signed-off-by: Justin Chu <11205048+justinchuby@users.noreply.github.com> Co-authored-by: Justin Chu <11205048+justinchuby@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DRY refactor: dedupe the VLM vision-tower
preprocess_weightsboilerplate.The vision sub-models of LLaVA, Gemma3, and Mllama hand-wrote the identical loop: filter to the vision-tower prefixes, then rename the HuggingFace vision MLP projections
mlp.fc1→mlp.up_projandmlp.fc2→mlp.down_projto match ourFCMLPcomponent naming.Changes
vlm_vision_weights(state_dict, prefixes)to_weight_utils.py, alongside the existingvlm_decoder_weights/vlm_embedding_weightshelpers.("vision_tower.", "multi_modal_projector."), mllama uses("vision_model.",).Behaviour-preserving: same filtering and same renames, just centralised.
Verification
ruff check+ruff format --checkclean.