Question: speed

[kalle07]

i tried to keep all same in my python code as in lm-studio settings

    if model_type == "gguf":
        print(f"System: {system_content}\n")
        print(f"User: {user_content}\n")
        print(f"Assistant _GGUF_ with seed:{seed}:\n", file=sys.stderr)

        llm = Llama(
            model_path=gguf_model_path,
            n_ctx=12000,
            n_threads=os.cpu_count() or 8,
            n_gpu_layers=-1 if torch.cuda.is_available() else 0,
            use_mmap=False,
            use_mlock=False,
            flash_attn=True,
            seed=seed,
            verbose=False,
            n_mlx_experts=10,
            offload_kqv=True,
        )

        max_tokens = int(summary_word_limit * 2.4)

        # ---------------- Streaming with batching ----------------
        flush_every_tokens = 5

        token_buffer = ""
        buffer_token_count = 0

        output = ""
        generated_tokens = 0

        gen_start = time.perf_counter()

        for chunk in llm(
            chat_prompt,
            max_tokens=max_tokens,
            temperature=0.1,
            top_p=0.95,
            top_k=40,
            repeat_penalty=1.1,
            min_p=0.05,
            stream=True,
        ):

and i also see it directly becasue the difference is large
30token/s the python with your latest llama.cpp (https://github.com/JamePeng/llama-cpp-python/releases/download/v0.3.20-cu126-Basic-win-20260104/llama_cpp_python-0.3.20-cp312-cp312-win_amd64.whl)
38token/s lm-studio (cuda12.cpp)
34token/s lm-studio (cuda.cpp)

what is your impression, you do make such test?

[JamePeng]

LM Studio directly calls functions from the llama.cpp library in C++. Naturally, Python's efficiency is not as high as direct access from C++. You can think of it as having an extra hook layer in the middle. This is why I try to avoid adding too much unnecessary code to Python to implement overly heavy functions.
In addition, LM Studio uses the AVX2 compilation version by default. The Basic version does not include any AVX instruction acceleration, which will also have an impact.

Of course, I've also been trying to break down and optimize the original llama-cpp-python's large and inefficient code classes to minimize unnecessary memory consumption, CPU execution steps, and CPU runtime.

[JamePeng]

I've made some logic optimizations; you can try the new version later.

old logic:

new logic:

[kalle07]

all fine ... yes c++ faster

so i can try a AVX2 version ? my intel-cpu support that!

[JamePeng]

You can try cloning the latest 0.3.21 code locally and then compiling it into an AVX2+CUDA version before giving it another try.

[JamePeng]

@kalle07 Version 0.3.21 has been compiled. You can compare it with the previous versions to see if there are any changes.

[kalle07]

self compile i dont get it with your version (original llama.cpp worked) but in the end i dont need the exe, only the python.

error:

(venv) c:\Users\123\Documents\python\summarizer\venv\src\llama-cpp-python>cmake -B build -DLLAMA_CURL=OFF -DGGML_CUDA=ON -DLLAMA_MULTIMODAL=ON -DGGML_NATIVE=OFF -DLLAMA_AVX2=ON -DLLAMA_FMA=ON -DLLAMA_KV_QUANT=ON
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.26100.0 to target Windows 10.0.19045.
-- The C compiler identification is MSVC 19.44.35219.0
-- The CXX compiler identification is MSVC 19.44.35219.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:101 (add_subdirectory):
  The source directory

    C:/Users/123/Documents/python/summarizer/venv/src/llama-cpp-python/vendor/llama.cpp

  does not contain a CMakeLists.txt file.

This directory doesn't exist in the first place, so ?!?
(i will try your compiled version)

btw you get my info parallel compiling?
set CMAKE_BUILD_PARALLEL_LEVEL=8
or
cmake --build build --config Release -j 8

[kalle07]

I only see the installation, numpy is the highest version, I think that's unnecessary (or?) and leads to incompatibilities with at least opencv.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
opencv-python 4.12.0.88 requires numpy<2.3.0,>=2; python_version >= "3.9", but you have numpy 2.4.1 which is incompatible.

so i need to reinstall numpy again

[kalle07]

ups only 200mb not 600?
ups ... error while calling my script

Traceback (most recent call last):
  File "C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\_ctypes_extensions.py", line 82, in load_shared_library
    return ctypes.CDLL(str(lib_path), **cdll_args)  # type: ignore
  File "C:\ProgramData\anaconda3\Lib\ctypes\__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\lib\ggml.dll' (or one of its dependencies). Try using the full path with constructor syntax.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\123\Documents\python\summarizer\sum02.py", line 10, in <module>
    from llama_cpp import Llama
  File "C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\__init__.py", line 1, in <module>
    from .llama_cpp import *
  File "C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\llama_cpp.py", line 8, in <module>
    from ._ggml import (
  File "C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\_ggml.py", line 21, in <module>
    libggml = ctypes_ext.load_shared_library("ggml", libggml_base_path)
  File "C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\_ctypes_extensions.py", line 84, in load_shared_library
    raise RuntimeError(f"Failed to load shared library '{lib_path}': {e}")
RuntimeError: Failed to load shared library 'C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\lib\ggml.dll': Could not find module 'C:\Users\123\Documents\python\summarizer\venv\Lib\site-packages\llama_cpp\lib\ggml.dll' (or one of its dependencies). Try using the full path with constructor syntax

switch back to 0.3.20 works ...

[JamePeng]

I think you could try downloading the pre-compiled wheel from GitHub Actions first. Your error message indicates that the compiled version is incompatible with your computer environment.

[kalle07]

oh the error is the pre-compiled
llama_cpp_python-0.3.21-cp312-cp312-win_amd64.whl (200MB)

the
llama_cpp_python-0.3.20-cp312-cp312-win_amd64.whl (700MB)
works fine

same install
pip install your-prewheel.whl --force-reinstall

no errors while install both except numpy

[JamePeng]

oh the error is the pre-compiled llama_cpp_python-0.3.21-cp312-cp312-win_amd64.whl (200MB)

the llama_cpp_python-0.3.20-cp312-cp312-win_amd64.whl (700MB) works fine

same install pip install your-prewheel.whl --force-reinstall

no errors while install both except numpy

I didn't see which pre-compiled version you're using is 200MB. You need to choose the correct version based on your local CUDA environment.

I saw you used the cu126 before, try the 0.3.21 cu126:https://github.com/JamePeng/llama-cpp-python/releases/tag/v0.3.21-cu126-Basic-win-20260111

[kalle07]

my fault ...sry !
could be better name it like your head line
llama_cpp_python-0.3.21-cuda12-cp312-cp312-win_amd64.whl

[JamePeng]

Indeed, I will look into how to improve the recognizability of the name "wheel".

[JamePeng]

i tried to keep all same in my python code as in lm-studio settings

if model_type == "gguf":
    print(f"System: {system_content}\n")
    print(f"User: {user_content}\n")
    print(f"Assistant _GGUF_ with seed:{seed}:\n", file=sys.stderr)

    llm = Llama(
        model_path=gguf_model_path,
        n_ctx=12000,
        n_threads=os.cpu_count() or 8,
        n_gpu_layers=-1 if torch.cuda.is_available() else 0,
        use_mmap=False,
        use_mlock=False,
        flash_attn=True,
        seed=seed,
        verbose=False,
        n_mlx_experts=10,
        offload_kqv=True,
    )

    max_tokens = int(summary_word_limit * 2.4)

    # ---------------- Streaming with batching ----------------
    flush_every_tokens = 5

    token_buffer = ""
    buffer_token_count = 0

    output = ""
    generated_tokens = 0

    gen_start = time.perf_counter()

    for chunk in llm(
        chat_prompt,
        max_tokens=max_tokens,
        temperature=0.1,
        top_p=0.95,
        top_k=40,
        repeat_penalty=1.1,
        min_p=0.05,
        stream=True,
    ):

and i also see it directly becasue the difference is large 30token/s the python with your latest llama.cpp (https://github.com/JamePeng/llama-cpp-python/releases/download/v0.3.20-cu126-Basic-win-20260104/llama_cpp_python-0.3.20-cp312-cp312-win_amd64.whl) 38token/s lm-studio (cuda12.cpp) 34token/s lm-studio (cuda.cpp)

what is your impression, you do make such test?

Did you use version 0.3.21 to compare with version 0.3.20? Was there a slight improvement?

[kalle07]

had to check a bit...
sry ... no different maybe 2%
but lm ~ 1.2-1.3 times faster

all is offload so fare i can see
added
offload_kqv=True/False, makes a differ but it was enabled by default

with my is used a bit less Vram 10gb my, 12gb lm-studio (seems depend on context preload, lm always consums it even its not there)
and RAM is it the other way
my +2gb , lm-studio +8gb