llama : make tensor-split regex patterns static to avoid per-token recompilation

[eduardopessin]

Overview

I hit this while running Qwen3.6-35B-A3B with -sm tensor, -np 5, and a large context on 3 consumer GPUs over PCIe Gen3, without NVLink.

The visible symptom was simple: the decode thread was stuck at 100% CPU, while the GPUs were only around 30% utilized. They were basically waiting on the CPU.

I profiled only the decode thread, not the whole process, to avoid mixing this with the HTTP worker threads:

perf record -e cpu-clock -F 299 -t <decode-tid> --call-graph dwarf

That showed a large amount of time under llama_meta_device_get_split_state() and the related split-state path. About 30% of the decode thread was in std::regex compilation frames, including _Compiler::_M_*, _BracketMatcher, std::collate::do_transform, __strxfrm_l, and __dynamic_cast. Another ~22% was in the ggml_backend_meta_get_split_state machinery.

The problem is that llama_meta_device_get_split_state() declared 29 std::regex objects as local variables. In tensor split mode, this callback runs once per tensor, per token, because the graph is rebuilt on each decode step. That means the same regexes are compiled again and again in the decode path.

This moves those patterns to file scope (compiled once, reused). The patterns are literals and do not carry mutable state, so the behavior is unchanged. Keeping them outside the function also avoids the function-local static concern raised in review.

After the change, the std::regex compilation frames disappear from the profile. get_split_state drops from roughly 22% to roughly 13%, freeing about 40% of the decode thread in this setup. With concurrent load, the GPUs are then able to saturate.

I am not quoting absolute tok/s numbers here, since those depend heavily on the hardware and workload. The useful result is the profile delta.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES — I used AI assistance while profiling, diagnosing, and preparing this change. I ran the profiling myself on my own hardware and validated the result locally.

[eduardopessin]

Done

[CISC]

This just introduces MT issues, I suppose they need to live outside llama_meta_device_get_split_state.

Also, please use our PR template in the OP.

[eduardopessin]

Done, plz let me know if you need something else

[eduardopessin]

Also tested and profiled, regex completly gone from hot path

[JohannesGaessler]

Regarding multi-threading issues: to my knowledge since C++ 11 the creation of local static variables is thread safe so it should be fine to just add static to the code on master in order to keep the variables contained to the function. Also if at all possible, please post benchmark numbers to show the actual impact of this change.

[JohannesGaessler]

@CISC can you re-review?

[JohannesGaessler]

@ggml-org/maintainers can I get a second approval, please?