Skip to content

server : support slot save/restore with media inputs - #26640

Merged
ngxson merged 6 commits into
ggml-org:masterfrom
CHIPMUNK-T0T:feat/mtmd-slot-media-save-restore
Aug 12, 2026
Merged

server : support slot save/restore with media inputs#26640
ngxson merged 6 commits into
ggml-org:masterfrom
CHIPMUNK-T0T:feat/mtmd-slot-media-save-restore

Conversation

@CHIPMUNK-T0T

@CHIPMUNK-T0T CHIPMUNK-T0T commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR adds support for saving and restoring slots that contain media input. Upstream stores the KV sequence state and a plain token list, but does not preserve the media positions and mtmd_input_chunk state required to reconstruct a media-containing slot, so saving such a slot is currently rejected with HTTP 501.

This PR uses mtmd_input_chunk_save() / mtmd_input_chunk_load() from #26645 and stores the complete server_tokens state - tokens, media start positions, and serialized media chunks - as a single versioned packed payload. The same format is used for both text-only and media-containing slots.

Because the packed payload can be larger than the slot's n_ctx, the restore path first queries the saved payload size through llama_state_seq_load_file() before allocating the restore buffer. After restore, the reconstructed media state can participate in the existing media matching / prefix reuse path.

Closes #25854

Serialization format

Newly saved slots use the same packed server_tokens format for both text-only and media-containing slots.

[LLAMA_TOKEN_NULL]
[version]
[n_tokens][tokens...]
[n_media][start_idx...]
(
  [chunk_size][serialized media chunk]
)...
[zero padding]

The current format version is 1.

n_tokens and n_media are stored as element counts for their respective vectors. All media start indices are written first, followed by the serialized media chunks in the same order.

  • A text-only slot has n_media = 0
  • The byte stream is zero-padded with 0-3 trailing bytes so that it can be stored as whole llama_token words

Backward compatibility

Slot files written before this PR contain a plain token list in the token payload.

The restore path distinguishes the formats as follows:

  • starts with LLAMA_TOKEN_NULL: versioned packed server_tokens format
  • otherwise: legacy plain token list

Therefore, text-only slot files saved by older servers can still be restored after this PR.

Newly saved text-only slots use the packed format as well, so restoring a new-format slot file on an older server is not supported.

server_tokens serialization

This PR adds server_tokens::serialize() and server_tokens::deserialize().

Internal reader / writer helpers handle scalar and vector fields. Media chunks are serialized through mtmd_input_chunk_save() and restored through mtmd_input_chunk_load().

The packed state stores the token sequence, media start positions, and serialized media chunks. deserialize() validates the format and payload boundaries, reconstructs the media chunks, checks the supported media types (IMAGE and AUDIO), and validates trailing padding. After deserialization, server_tokens::validate() validates text token IDs against the current vocabulary and checks the mapping between media chunks and their token ranges.

An mmproj is required only when the restored state actually contains media.

Slot save

llama_state_seq_save_file() returning 0 is handled as an error. The n_saved API field continues to report the logical number of tokens held by the slot, rather than the size of the packed representation.

Slot restore

Serialized media chunks can make the packed payload larger than the slot's n_ctx.

This PR therefore allows llama_state_seq_load_file() to query the saved payload count when tokens_out == nullptr, without restoring the sequence state. The server uses that count to allocate the payload buffer before performing the actual load.

After deserialization, the logical token count is checked against the slot's n_ctx and the restored tokens are validated against the current model. If restore or validation fails, the slot is cleared before returning the error.

Slot erase

Erase remains unsupported for slots containing media; its media gate is unchanged.

Testing

Local validation:

  • test-mtmd-c-api with assertions enabled: passed
  • test_slot_save.py + test_vision_api.py: 28 passed

Main API-level cases verified:

  • text-only and image-containing slot save / restore
  • restore of a legacy plain-token slot file
  • multiple images
  • same-image prefix reuse, including across a server restart
  • no image-prefix reuse with a different image
  • restoration of a packed payload larger than n_ctx
  • graceful failure without an mmproj, leaving the slot usable

Benchmark

Qwen3.5-2B Q8_0 + BF16 projector / RTX 4070 / Flash Attention (-fa on) / --cache-ram 0

3316-token multimodal prompt with a 3303-token reusable prefix, median of 3 runs:

  • cold prompt processing: 292.38 ms
  • restore + same image prompt processing: 21.82 ms
  • prompt processing speedup: 13.4x
  • cold completion HTTP request: 298.77 ms
  • restore + same image completion HTTP request: 28.16 ms
  • completion HTTP speedup: 10.6x
  • restore API: 9.80 ms
  • saved state size: 60.9 MB

With a different image, the restored image prefix was not reused.

Additional information

  • Media slot files are expected to be restored with the same model / mmproj configuration.
  • Broader portability across server versions or endianness is not guaranteed.
  • For SWA models, prefix reuse after restore requires --swa-full.
  • A future sequence-state API for arbitrary raw user data could remove the current llama_token-width packing and padding workaround.

Requirements

  • I have read and agree with the contributing guidelines

  • AI usage disclosure: YES

    • Motivation & Design: I designed the slot save / restore integration based on the proposal in server : fix 501 on multimodal models blocking text-only slot save/restore (#21133) #25076 and updated it to use the mtmd_input_chunk serialization API introduced in mtmd: add chunk save/load function #26645.
    • Investigation & Code Review (Fable 5 / Opus 5, 4.8): AI agents were used to investigate the relevant multimodal and slot-state paths and assist with code review under my instructions.
    • Implementation (Opus 5, 4.8 / Fable 5): AI assistance was used to draft parts of the implementation and tests based on my technical instructions.
    • Draft Translation (Opus 5 / Fable 5): The PR text was written by me in Japanese first, with AI used to assist with the English translation.
    • Validation & Responsibility: I manually reviewed the submitted changes, ran the local tests and GPU benchmarks, and take full responsibility for the submitted changes.

@CHIPMUNK-T0T
CHIPMUNK-T0T requested review from a team and ggerganov as code owners August 5, 2026 14:31
@github-actions github-actions Bot added testing Everything test related server mtmd Related to multimodal functionality (video/image/audio) labels Aug 5, 2026

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good - will do more detailed review later. We would need @ngxson to comment on the mtmd change.

@ggerganov ggerganov self-assigned this Aug 5, 2026
@CHIPMUNK-T0T CHIPMUNK-T0T changed the title Feat/mtmd slot media save restore server : support slot save/restore with image inputs Aug 5, 2026

@ngxson ngxson left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mtmd change looks quite hacky. I already acknowledge such use case and will push a separate PR for deserialize / serialize mtmd_input_chunk

Comment thread tools/server/server-common.h Outdated
Comment on lines +210 to +211
std::vector<uint8_t> serialize_media_state() const;
static server_tokens deserialize_media_state(const llama_tokens & tokens, bool has_mtmd, const uint8_t * data, size_t size);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API design: should we simply allow serialize the whole server_tokens object instead of just media state?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ngxson
Rather than serializing the whole server_tokens object separately, for now I am planning to serialize only what the llama state does not already store: the media chunks and their positions.

The token sequence is already saved by llama_state_seq_save_file(), and the position and length of each image are already carried there as runs of LLAMA_TOKEN_NULL. Saving the token sequence on the server_tokens side as well would hold the same information twice.

On restore, server_tokens is rebuilt by matching the token sequence recovered from the llama state against the saved chunk positions and the serialized chunks. That keeps the existing checks: that each image range lines up with its LLAMA_TOKEN_NULL run, and that no orphan media token is left without a descriptor.

This is how I plan to proceed.

@ngxson ngxson Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the llama_state_seq_save_file takes the list of llama_token and directly serialize it as raw bytes, completely unrelated to the saved state. In other words, I see the API like this:

// this is what we currently have:
llama_state_seq_save_file(..., const llama_token * tokens, size_t n_token_count);

// equivalent to:
// data_len % sizeof(llama_token) == 0
llama_state_seq_save_file(..., const char * data, size_t data_len);

My idea is that instead of storing the list of plain tokens, we can just have a server-specific serializer and the data will be stored in the place of tokens. Something like this:

std::vector<char> data = slot.serialize();
GGML_ASSERT(data.size() % 4 == 0);
const llama_token * data_ptr = reinterpret_cast<const llama_token *>(data);
llama_state_seq_save_file(..., data_ptr, data.size() / 4);

CC @ggerganov not sure if you are agree on this solution, or we should probably have a specific version of llama_state_seq_save_file that stores raw bytes as addition info ?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's a good point. The llama_state_seq_save/load API should be promoted to save any user data - no reason to restrict this to tokens only. We can do the trick that @ngxson proposes for now, and in a follow-up PR we can update the API.

@CHIPMUNK-T0T CHIPMUNK-T0T Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ngxson @ggerganov
Understood. One question before I update the serialization.

My preference would be to keep this simple and use a single server_tokens packed representation for both text-only and multimodal slots, with the media state simply empty for text-only slots.

Would you prefer that, or should we preserve the existing text-only on-disk format for compatibility and only use the packed representation for media slots?

The current PR says that below additional infomation.

Text-only slot save / restore behavior is unchanged.

So I’d like your opinion on whether we should keep that compatibility guarantee or simplify the serialization by using a single representation.

I assume the generic raw user-data API itself can remain a follow-up, as discussed.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A proper serializer is cleaner and will save us some headaches in the long run, I don't see why we don't impl it right now (plus, we can impl versioning along the way, similar to what mtmd save/load already having)

On compatibility: Simply write a LLAMA_TOKEN_NULL as the magic bytes for this new serializing format. Old save files generated prior to this PR never have LLAMA_TOKEN_NULL inside it anyway. If we read a file that doesn't start with null token, fallback to old version

So header for this format will be: 4 bytes LLAMA_TOKEN_NULL, followed by 4 bytes for version number

@ngxson

ngxson commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

PTAL on the chunk save/load API: #26645

@CHIPMUNK-T0T

Copy link
Copy Markdown
Contributor Author

@ngxson
Thanks. I have looked at #26645, and I will switch to using that API as the mtmd_input_chunk save / load foundation this feature needs.

The cache-only mtmd fields and mtmd_input_chunk_init_image_cache() added in this PR will be removed in favor of mtmd_input_chunk_save() / mtmd_input_chunk_load().

@CHIPMUNK-T0T

Copy link
Copy Markdown
Contributor Author

@ggerganov
Please hold off on the detailed review.

Once #26645 is merged, I will rebase this PR and update the implementation to use its mtmd_input_chunk save/load API. I will let you know when the PR is ready for review again.

@CHIPMUNK-T0T
CHIPMUNK-T0T force-pushed the feat/mtmd-slot-media-save-restore branch from 0478749 to badb515 Compare August 7, 2026 12:24
@CHIPMUNK-T0T
CHIPMUNK-T0T force-pushed the feat/mtmd-slot-media-save-restore branch from badb515 to 5bddd7b Compare August 7, 2026 12:27
@CHIPMUNK-T0T

Copy link
Copy Markdown
Contributor Author

@ngxson @ggerganov

Following the suggested direction, I have updated the implementation to serialize / deserialize the whole server_tokens object as a versioned packed representation.

LLAMA_TOKEN_NULL is used as the format marker, with versioning and a fallback to the legacy plain-token format. The tests and the PR description have also been updated to match the current implementation.

The PR description summarizes the changes and the validation results.

The implementation is ready for another review. Please take a look.

Comment on lines +8 to +17
STATE_FILE_HEADER_SIZE = 12
# the token payload holds a packed server_tokens object (see server_tokens::serialize()):
# LLAMA_TOKEN_NULL(4) version(4) n_tokens(4) tokens, media list, zero padding to whole tokens
PACKED_HEADER_SIZE = 12 # LLAMA_TOKEN_NULL, version, n_tokens
LLAMA_TOKEN_NULL = 0xFFFFFFFF # -1 read back as an unsigned word

# media list layout in the packed payload: n_media(4), then per image: start_idx(4) chunk_size(4) chunk blob
N_MEDIA_FIELD_SIZE = 4
START_IDX_FIELD_SIZE = 4
CHUNK_SIZE_FIELD_SIZE = 4

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to test these. Those are detailed impl, the test system only need to test the API surface

@CHIPMUNK-T0T CHIPMUNK-T0T Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.
I kept coverage of observable API behavior, and the other tests were removed because they exercised packed-format implementation details outside the API boundary.

Comment thread tools/server/server-common.cpp Outdated
Comment on lines +243 to +246
template <typename T>
void server_tokens_state_write(std::vector<char> & data, T value) {
const auto * ptr = reinterpret_cast<const char *>(&value);
data.insert(data.end(), ptr, ptr + sizeof(value));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to make sure T is copyable trivially (same pattern as mtmd_serialization)

@CHIPMUNK-T0T CHIPMUNK-T0T Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added std::is_trivially_copyable<T> checks to the reader / writer templates.

Comment thread tools/server/server-context.cpp Outdated
Comment on lines +57 to +61
// the token payload of a sequence state file holds a packed server_tokens object (see server_tokens::serialize()), so it can be longer than the number of tokens the slot holds.
// read its size from the file header, falling back to n_ctx if the header cannot be trusted - llama_state_seq_load_file() then reports the malformed file.
// TODO: remove this once llama_state_seq_save_file() can store arbitrary user data
static size_t state_file_payload_size(const std::string & filepath, size_t fallback) {
constexpr std::streamoff header_size = 3 * sizeof(uint32_t); // magic, version, payload size

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is quite hacky tbh, we can simply modify llama_state_seq_load_file to return wanted size via n_token_count_out if tokens_out == nullptr (actual state load will be skipped in that case). such change will be less than 10 lines of code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed the server-side header parsing and updated llama_state_seq_load_file() to return the required size via n_token_count_out when tokens_out == nullptr.

Comment thread tools/server/server-common.cpp Outdated
static_assert(sizeof(llama_token) == sizeof(uint32_t), "unexpected llama_token size");

std::vector<char> data;
server_tokens_state_write(data, (llama_token) LLAMA_TOKEN_NULL);

@ngxson ngxson Aug 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we already had the RAII server_tokens_state_reader , why not also having server_tokens_state_writer ?

You can abstract the mtmd_chunk save into server_tokens_state_writer so that server_tokens::serialize() will be minimal and more readable, similar to serialize() functions in mtmd.

Plus, having write<std::vector<T>> will make the code much cleaner:

writer.write(LLAMA_TOKEN_NULL);
writer.write(SERVER_TOKENS_STATE_VERSION);
writer.write(tokens); // overload write<std::vector<T>>

std::vector<size_t> map_keys;
// TODO: copy map_idx_to_media keys to map_keys
writer.write(map_keys);

// custom writer for mtmd_chunks
for (const auto & item : map_idx_to_media) {
    const auto * chunk = item.second.get();
    std::vector<char> chunk_data;
    // TODO: get chunk_size, resize chunk_data, then write to chunk_data
    writer.write(chunk_data); // will copy, but it's cleaner than mtmd_input_chunk_save() writing directly to output buf
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I added server_tokens_state_writer to pair with the reader and moved scalar/vector and image-chunk serialization into it. server_tokens::serialize() now only defines the field order and performs media-specific validation.

@CHIPMUNK-T0T

Copy link
Copy Markdown
Contributor Author

@ngxson @ggerganov

I've updated the implementation based on the review feedback:

  • simplified the tests around the API surface
  • added trivially-copyable checks to the serializer helpers
  • replaced server-side header parsing with the llama_state_seq_load_file() size-query path
  • added server_tokens_state_writer and simplified server_tokens::serialize()

Comment thread tools/server/server-common.cpp Outdated
Comment on lines +538 to +542
if (n_tokens == 0 || n_pos <= 0) {
throw std::runtime_error("Invalid image chunk in server tokens");
}
if (id == nullptr || id[0] == '\0') {
throw std::runtime_error("Slot save with image tokens without an ID is not supported");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these checks are too defensive, they never trigger in practice - should remove them

server_tokens can't handle either of these, something else will break before we even get into this code

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed the redundant serialize-side checks as suggested.

Comment thread tools/server/server-common.cpp Outdated
Comment on lines +587 to +588
if (mtmd_input_chunk_get_type(chunk.get()) != MTMD_INPUT_CHUNK_TYPE_IMAGE) {
throw std::runtime_error("Unsupported media type in server tokens state");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

audio?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ngxson
Initially, the mtmd state was modality-specific, so supporting image/audio would have required separate handling.
With #26645, both can use the same chunk save/load path, so I can remove the image-only check and handle both uniformly.

Does that sound good?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok for removing this

in general, these checks are too defensive as I pointed out. most of the checks you added are either existing somewhere else in the code base, or can be absorbed into something like server_tokens::validate()

Comment thread tools/server/server-common.cpp Outdated
Comment on lines +590 to +604
const char * id = mtmd_input_chunk_get_id(chunk.get());
if (id == nullptr || id[0] == '\0') {
throw std::runtime_error("Image ID is missing in server tokens state");
}
const size_t n_chunk_tokens = mtmd_input_chunk_get_n_tokens(chunk.get());
const llama_pos n_pos = mtmd_input_chunk_get_n_pos(chunk.get());
if (n_chunk_tokens == 0 || n_pos <= 0 || start_idx < last_end ||
start_idx > tokens.size() || n_chunk_tokens > tokens.size() - start_idx) {
throw std::runtime_error("Invalid image range in server tokens state");
}
for (size_t j = start_idx; j < start_idx + n_chunk_tokens; ++j) {
if (tokens[j] != LLAMA_TOKEN_NULL) {
throw std::runtime_error("Image range does not match server tokens");
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all these checks should be absorbed by server_tokens::validate()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Moved the media/token consistency checks into server_tokens::validate().

Comment thread src/llama-context.cpp

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

llama.h documentation need to be updated to reflect this new behavior

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, add comment

Comment thread tools/server/server-common.cpp Outdated
Comment on lines +619 to +629
for (size_t i = 0; i < tokens.size();) {
if (tokens[i] != LLAMA_TOKEN_NULL) {
++i;
continue;
}
const auto it = result.map_idx_to_media.find(i);
if (it == result.map_idx_to_media.end()) {
throw std::runtime_error("Image token has no descriptor");
}
i += mtmd_input_chunk_get_n_tokens(it->second.get());
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should already be part of one of existing method inside server_tokens, consider reuse existing functions, not to duplicate the logic

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed the duplicate token scan and now rely on server_tokens::validate().

@CHIPMUNK-T0T CHIPMUNK-T0T changed the title server : support slot save/restore with image inputs server : support slot save/restore with multimodality inputs Aug 9, 2026
@CHIPMUNK-T0T CHIPMUNK-T0T changed the title server : support slot save/restore with multimodality inputs server : support slot save/restore with media inputs Aug 9, 2026
@CHIPMUNK-T0T
CHIPMUNK-T0T force-pushed the feat/mtmd-slot-media-save-restore branch from 4a67feb to 94ab2a6 Compare August 9, 2026 13:31
@CHIPMUNK-T0T

Copy link
Copy Markdown
Contributor Author

@ngxson @ggerganov

I've updated the implementation based on the latest review feedback:

  • removed the redundant serialize-side checks
  • enabled audio through the same media chunk save/load path
  • moved media/token consistency checks into server_tokens::validate()
  • removed the duplicate deserialize-side token scan
  • documented the tokens_out == nullptr behavior in llama.h

Comment thread tools/server/server-context.cpp Outdated
}
// Gate on slot content, consistent with save/restore.
if (!check_slot_no_media(*slot, task.id)) {
if (!check_slot_no_media_for_erase(*slot, task.id)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we actually need to check for "no media" here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ggerganov
I agree. Since this PR changes how media chunks are handled, I don't think this check is necessary anymore.
On the other hand, you've already approved the PR. Should I remove it in this PR?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed the media check for slot erase.

Comment on lines +2600 to +2606
std::vector<char> packed;
try {
packed = slot->prompt.tokens.serialize();
} catch (const std::exception & err) {
send_error(task, err.what(), ERROR_TYPE_NOT_SUPPORTED);
break;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a follow-up PR, should be relatively simple to append the prompt.checkpoints so that we can restore them too.

Comment thread tools/server/server-common.h Outdated

llama_tokens get_text_tokens() const;

// packed into the token payload of a sequence state file: [LLAMA_TOKEN_NULL][version][n_tokens][tokens][n_media][start_idx]...([chunk_size][media chunk])...[zero padding]

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to specify this in the comment, the code also reflect the details

@CHIPMUNK-T0T CHIPMUNK-T0T Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed this comment.

Comment thread tools/server/server-common.cpp Outdated
if (reader.read<uint8_t>() != 0) {
throw std::runtime_error("Invalid padding in server tokens state");
}
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we really need to be this defensive? what's the problem if padding is not 0 ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Removed padding validation.

Comment thread tools/server/server-common.cpp Outdated
Comment on lines +571 to +572
if (type != MTMD_INPUT_CHUNK_TYPE_IMAGE && type != MTMD_INPUT_CHUNK_TYPE_AUDIO) {
throw std::runtime_error("Unsupported media type in server tokens state");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought you said to remove this check?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CHIPMUNK-T0T CHIPMUNK-T0T Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I apologize. It must have slipped my mind.
Removed this check and moved to validate().

@ngxson
ngxson merged commit 5d9e5ac into ggml-org:master Aug 12, 2026
25 of 32 checks passed
gabe-l-hart added a commit to gabe-l-hart/llama.cpp that referenced this pull request Aug 12, 2026
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>

* origin/master: (383 commits)
  cmake :  introduce semantic versioning  (ggml-org#26839)
  gguf : harden loader against malformed tensor dims and metadata types (ggml-org#25596)
  kleidiai: Add runtime feature detection mechanism for aarch64/kleidiai (ggml-org#26076)
  model : disallow integer dflash sliding_window_pattern (ggml-org#26900)
  sync : ggml
  cmake : add config version support (ggml/1582)
  server : support slot save/restore with media inputs (ggml-org#26640)
  ui: add read_media tool (ggml-org#25877)
  opencl: default FA c8 cluster width to 16 on X1E (ggml-org#26433)
  tests : update speculative params (ggml-org#26925)
  vulkan: add TQ2_0 (ternary) support (ggml-org#25850)
  wavtokenizer-dec : bound posnet/convnext block_count against n_layer_all (ggml-org#26892)
  convert : handle per_layer_config in Gemma4 (transformers 5.15) (ggml-org#26882)
  opencl: use flat mv q5_k when weight exceeds image1d_buffer_t limit (ggml-org#26880)
  chat : fix muse-glimmer detection of tool calls after EOM (ggml-org#26879)
  ci : add missing release check (ggml-org#26923)
  CUDA: only disable CUDA graphs when mul_mat_id actually needs a stream sync (ggml-org#26802)
  cuda : add warp-per-row wkv7 kernel for single-token decode (ggml-org#26111)
  spec : update speculative-simple (ggml-org#26904)
  chat : tighten bare function parsing for Qwen models (ggml-org#26793)
  ...
huaxel pushed a commit to huaxel/CachyLLama that referenced this pull request Aug 12, 2026
* server : save serialized image chunks at the end of the llama state

* server : support multimodal slot state save/restore with packed payload

* server : refine image slot state serialization

* server : support media slot state and centralize media validation

* server : remove unnecessary comment

* server : remove defensive media checks and move the chunk type check to validate()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

mtmd Related to multimodal functionality (video/image/audio) server testing Everything test related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support slot save/restore for slots containing images

3 participants