Add CUDA backend support - #15
Open
greenjava wants to merge 8 commits into
Open
Conversation
- enable cuda inference via cmake flag - add cuda backend initialization logic - implement flash attention with manual fallback - replace pool_1d with manual mean pooling - add cuda inference test suite - update build and readme documentation
- add --encode-img-size cli argument to override default grid dimensions - update help text to include the new flag - crop global attention RoPE frequencies to effective grid size - derive feature map sizes from actual encoder token count - support variable spatial grids in position embeddings and FPN - add timing logs for pcs stages (text, geo, fusion, detr, seghead)
- store applied prompt in state to detect edits - auto-reset tracker when text input changes - extract reset logic into reset_all function - update Reset button to use new function - save propagated mask logits for pending masklets - fix masklet id churn by storing mask data early
- configure cmake install rules for headers, libs, and binaries - generate package config and version files for downstream usage - add install targets for all example and test executables - enable cpack to build tgz archives
- add cuda build matrix for linux ci and release - replace manual packaging with cpack - standardize artifact naming and upload paths
- upgrade checkout, upload, download artifact, and release actions
- replace system("mkdir") calls with portable macro
- move ensure_dir logic to dedicated test_fs.h header
greenjava
marked this pull request as ready for review
August 6, 2026 10:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The Linux CPU version is currently functional but extremely slow. I took the liberty of adding a CUDA backend to significantly improve performance. As with the Metal backend, this also required a few changes to ggml.
The implementation seems to work well in my testing, so I thought I'd submit these changes in case they are useful or of interest to the project.
Summary of changes
-DSAM3_CUDA=ON)ci.ymlandrelease.ymlto usecmake --installand the CPack package targetggmlsubmodule to thegreenjava/ggmlfork (commit499c8a76, branchsam3-ops), which includes:WIN_PART/WIN_UNPARTkernelsWIN_PART/WIN_UNPARTkernels (ported fromsam3-metal-ops)conv2d_transposeand head-size-32 attentionencode-img-sizeargumentHow to test
git submodule update --init --recursiveCPU / Metal:
CUDA:
Build the package:
Note about ggml
ggmlsubmodule currently points to mygreenjava/ggmlfork (branchsam3-ops, commit499c8a76), which contains the required changes.ggmlrepository yet, so the fork is used temporarily until the Metal/CUDA changes can be upstreamed.sam3-metal-opsbranch to bringWIN_PARTsupport to the Metal backend. This involved resolving a large number of conflicts. Since I'm not very familiar with the Metal backend and I don't have access to a Mac, I wasn't able to test those changes. My apologies if I inadvertently broke anything on that platform.