fix(galleryop): persist cancellable so restarted in-flight ops stay cancellable - #10454
Merged
Conversation
…ancellable
In distributed mode a model/backend install marks OpStatus.Cancellable=true
while downloading, but the gallery_operations row never recorded it:
UpdateStatus persisted only progress/status and Create left the cancellable
column at its zero value. After a replica restart Hydrate rebuilt the op with
cancellable=false, /api/operations reported false, and the UI hid the cancel
button - the orphaned op then lingered until the 30-minute stale reaper
expired it ("stays there on restart, can't cancel, after a bit it expires").
Persist the flag on every progress tick and at row creation (installs are
cancellable, deletes are not), and clear it on terminal transitions. A
rehydrated in-flight op is now cancellable, so an admin can dismiss the
orphaned op immediately instead of waiting out the reaper. The functional
cancel path already survived restart (CancelOperation persists store.Cancel
even with no live CancelFunc); this restores the UI affordance that drives it.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
mudler
added a commit
that referenced
this pull request
Jun 22, 2026
…10460) PR #10454 added a `cancellable bool` parameter to GalleryStore.UpdateProgress but missed two callers under tests/e2e/distributed, breaking the build on master (golangci-lint and tests-e2e-backend both failed to compile with "not enough arguments in call to ... UpdateProgress"). Pass cancellable=true (both ops are downloading installs, which are cancellable) and assert the flag is persisted, exercising the new behavior. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In distributed mode, when LocalAI restarts while a model/backend install is in flight, the operation reappears in the Operations panel after restart but cannot be cancelled - the cancel button is gone. It only clears once the 30-minute stale reaper expires it.
Root cause
The
cancellableflag was never persisted to thegallery_operationstable:OpStatus.Cancellable = trueon every progress tick, butGalleryService.UpdateStatuspersisted only progress and status (UpdateProgress/UpdateStatus), never thecancellablecolumn.Createleft the column at its zero valuefalse.So after a restart,
Hydrate()rebuilt the op from the DB row withCancellable: false,/api/operationsreported"cancellable": false, and the UI hid the cancel button. The orphaned op then lingered untilCleanStale/ReapStaleOperationsflipped it tofailed(~30 min) - exactly the reported "stays there on restart, can't cancel, after a bit it expires".The functional cancel path already survived restart (
CancelOperationpersistsstore.Canceleven with no liveCancelFuncin distributed mode, see existingcancel_persist_test.go); the only thing missing was the UI affordance that drives it.Fix
distributed/gallery.go:UpdateProgresspersistscancellableon every tick;UpdateStatusclears it on terminal transitions (a done/cancelled op is never cancellable).galleryop/service.go: passop.Cancellablethrough to the store; setCancellableat row-create time (truefor installs,!op.Deletefor models) to cover the brief "pending" window before the first progress tick.cancellable_persist_test.goreproduces the restart-then-rehydrate scenario and asserts the op stays cancellable.A rehydrated in-flight op is now cancellable, so an admin can dismiss the orphaned op immediately instead of waiting out the reaper.
Testing
go test ./core/services/galleryop/passes (new spec red before the fix, green after).