flux dev Q4 problem

[rakib91221]

please fix this problem or please share your workflow .. my model rtx 3060 12gb

[Yaruze66]

On my setup (8GB VRAM, 32GB RAM + 32GB swap file), I can run fp16 flux-dev + fp16 clip without any issues, but Q8_0.gguf throws an out-of-memory error. The strangest thing is that much less memory is actually being used at the time...

Console log

D:\ComfyUI_windows_portable_nvidia>.\python_embeded\python.exe -s ComfyUI\main.py --fp16-vae --windows-standalone-build
[START] Security scan
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2024-08-15 13:26:15.279233
** Platform: Windows
** Python version: 3.11.9 (tags/v3.11.9:de54cf5, Apr  2 2024, 10:12:12) [MSC v.1938 64 bit (AMD64)]
** Python executable: D:\ComfyUI_windows_portable_nvidia\python_embeded\python.exe
** ComfyUI Path: D:\ComfyUI_windows_portable_nvidia\ComfyUI
** Log path: D:\ComfyUI_windows_portable_nvidia\comfyui.log

Prestartup times for custom nodes:
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\rgthree-comfy
   1.1 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-Manager

Total VRAM 8192 MB, total RAM 32705 MB
pytorch version: 2.3.1+cu121
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 2070 SUPER : cudaMallocAsync
Using pytorch cross attention
[Prompt Server] web root: D:\ComfyUI_windows_portable_nvidia\ComfyUI\web
Adding extra search path checkpoints D:/stable-diffusion-webui-1.9.3/models/Stable-diffusion
Adding extra search path configs D:/stable-diffusion-webui-1.9.3/models/Stable-diffusion
Adding extra search path vae D:/stable-diffusion-webui-1.9.3/models/VAE
Adding extra search path loras D:/stable-diffusion-webui-1.9.3/models/Lora
Adding extra search path loras D:/stable-diffusion-webui-1.9.3/models/LyCORIS
Adding extra search path upscale_models D:/stable-diffusion-webui-1.9.3/models/ESRGAN
Adding extra search path upscale_models D:/stable-diffusion-webui-1.9.3/models/RealESRGAN
Adding extra search path upscale_models D:/stable-diffusion-webui-1.9.3/models/SwinIR
Adding extra search path embeddings D:/stable-diffusion-webui-1.9.3/embeddings
Adding extra search path hypernetworks D:/stable-diffusion-webui-1.9.3/models/hypernetworks
Adding extra search path controlnet D:/stable-diffusion-webui-1.9.3/models/ControlNet
### Loading: ComfyUI-Impact-Pack (V6.2)
### Loading: ComfyUI-Impact-Pack (Subpack: V0.6)
[Impact Pack] Wildcards loading done.
### Loading: ComfyUI-Inspire-Pack (V0.83)
### Loading: ComfyUI-Manager (V2.48.6)
### ComfyUI Revision: 2542 [0f9c2a78] | Released on '2024-08-14'
(pysssss:WD14Tagger) [DEBUG] Available ORT providers: AzureExecutionProvider, CPUExecutionProvider
(pysssss:WD14Tagger) [DEBUG] Using ORT providers: CUDAExecutionProvider, CPUExecutionProvider
[comfyui_controlnet_aux] | INFO -> Using ckpts path: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\comfyui_controlnet_aux\ckpts
[comfyui_controlnet_aux] | INFO -> Using symlinks: False
[comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider']
D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\comfyui_controlnet_aux\node_wrappers\dwpose.py:26: UserWarning: DWPose: Onnxruntime not found or doesn't come with acceleration providers, switch to OpenCV with CPU device. DWPose might run very slowly
  warnings.warn("DWPose: Onnxruntime not found or doesn't come with acceleration providers, switch to OpenCV with CPU device. DWPose might run very slowly")
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
Efficiency Nodes: Attempting to add Control Net options to the 'HiRes-Fix Script' Node (comfyui_controlnet_aux add-on)...Success!

[rgthree] Loaded 42 fantastic nodes.
[rgthree] Will use rgthree's optimized recursive execution.

[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
WAS Node Suite: BlenderNeko's Advanced CLIP Text Encode found, attempting to enable `CLIPTextEncode` support.
WAS Node Suite: `CLIPTextEncode (BlenderNeko Advanced + NSP)` node enabled under `WAS Suite/Conditioning` menu.
WAS Node Suite: OpenCV Python FFMPEG support is enabled
WAS Node Suite Warning: `ffmpeg_bin_path` is not set in `D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\was-node-suite-comfyui\was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.
WAS Node Suite: Finished. Loaded 219 nodes successfully.

        "Art is a harmony parallel with nature." - Paul Cézanne


Import times for custom nodes:
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\websocket_image_save.py
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI_Noise
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI_TiledKSampler
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-dpmpp_2m_alt-Sampler
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI_ADV_CLIP_emb
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-WD14-Tagger
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-Custom-Scripts
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI_UltimateSDUpscale
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-GGUF
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\rgthree-comfy
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI_IPAdapter_plus
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\efficiency-nodes-comfyui
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\comfyui_controlnet_aux
   0.0 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-Inspire-Pack
   0.2 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI_smZNodes
   0.2 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-Manager
   0.5 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-Impact-Pack
   1.2 seconds: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\was-node-suite-comfyui

Starting server

To see the GUI go to: http://127.0.0.1:8188
FETCH DATA from: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-Manager\extension-node-map.json [DONE]
got prompt
Failed to validate prompt for output 9:
* ModelSamplingFlux 30:
  - Required input is missing: model
Output will be ignored
invalid prompt: {'type': 'prompt_outputs_failed_validation', 'message': 'Prompt outputs failed validation', 'details': '', 'extra_info': {}}
FETCH DATA from: D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-Manager\extension-node-map.json [DONE]
got prompt
[rgthree] Using rgthree's optimized recursive execution.
[rgthree] First run patching recursive_output_delete_if_changed and recursive_will_execute.
[rgthree] Note: If execution seems broken due to forward ComfyUI changes, you can disable the optimization from rgthree settings in ComfyUI.

ggml_sd_loader:
 1                             476
 8                             304


model weight dtype torch.bfloat16, manual cast: torch.float16
model_type FLUX
clip missing: ['text_projection.weight']
Requested to load FluxClipModel_
Loading 1 new model
loaded partially 5882.2 5852.23095703125 0
D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy\ldm\modules\attention.py:407: UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
  out = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=mask, dropout_p=0.0, is_causal=False)
Requested to load Flux
Loading 1 new model
!!! Exception during processing!!! Allocation on device
Traceback (most recent call last):
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\execution.py", line 152, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\execution.py", line 82, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\execution.py", line 75, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy_extras\nodes_custom_sampler.py", line 612, in sample
    samples = guider.sample(noise.generate_noise(latent), latent_image, sampler, sigmas, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise.seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy\samplers.py", line 706, in sample
    self.inner_model, self.conds, self.loaded_models = comfy.sampler_helpers.prepare_sampling(self.model_patcher, noise.shape, self.conds)
                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy\sampler_helpers.py", line 66, in prepare_sampling
    comfy.model_management.load_models_gpu([model] + models, memory_required=memory_required, minimum_memory_required=minimum_memory_required)
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy\model_management.py", line 527, in load_models_gpu
    cur_loaded_model = loaded_model.model_load(lowvram_model_memory, force_patch_weights=force_patch_weights)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy\model_management.py", line 325, in model_load
    raise e
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy\model_management.py", line 321, in model_load
    self.real_model = self.model.patch_model(device_to=patch_model_to, patch_weights=load_weights)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\comfy\model_patcher.py", line 352, in patch_model
    self.model.to(device_to)
  File "D:\ComfyUI_windows_portable_nvidia\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1173, in to
    return self._apply(convert)
           ^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 779, in _apply
    module._apply(fn)
  File "D:\ComfyUI_windows_portable_nvidia\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 779, in _apply
    module._apply(fn)
  File "D:\ComfyUI_windows_portable_nvidia\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 779, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-GGUF\ops.py", line 54, in _apply
    self.weight = fn(self.weight)
                  ^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1159, in convert
    return t.to(
           ^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\ComfyUI\custom_nodes\ComfyUI-GGUF\ops.py", line 25, in to
    new = super().to(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ComfyUI_windows_portable_nvidia\python_embeded\Lib\site-packages\torch\_tensor.py", line 1443, in __torch_function__
    ret = func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: Allocation on device

Got an OOM, unloading all loaded models.
Prompt executed in 282.46 seconds

UPD: Tried enabling CUDA System Memory Fallback, but that didn't help either.

[rakib91221]

please share workflow.. i can't find set clip device

[bananasss00]

please share workflow.. i can't find set clip device

https://github.com/city96/ComfyUI_ExtraModels

[Yaruze66]

@rakib91221 did you have any luck fixing that issue?

[rakib91221]

@rakib91221 did you have any luck fixing that issue?

no. but stable diffusion forge working fine

[rakib91221]

please share workflow.. i can't find set clip device

https://github.com/city96/ComfyUI_ExtraModels

thanQ

[Zuellni]

Try starting ComfyUI with --disable-smart-memory, that fixed it for me.

[fatality14]

It also seems like the Q4 model use more than 8Gb (have 4070 mobile 8Gb).
For me its stuck with long time sampling step. Yet it is slower than fp8 version.
I think there is some optimizations made in comfy basic loader.

[xThIsIsBoToXx]

It also seems like the Q4 model use more than 8Gb (have 4070 mobile 8Gb). For me its stuck with long time sampling step. Yet it is slower than fp8 version. I think there is some optimizations made in comfy basic loader.

I am having the same issue with the Q8 model, it seems to be running as a BF16 model VRAM wise.

[PrometheusDante]

I am testing the Q4 version at the moment on a 4080, which should cause no problems, but the first time queuing always results in oom at first, and queuing again it runs fine.

The purge VRAM node from LayerStyle is run at the end of each (successful) queue, so every 2nd attempt goes oom and every other succeeds, resulting in the model being unloaded, which causes it to oom again next queue on the fresh reload.

Muting the purge VRAM part allows consecutive runs without issue after the first oom. So something during initialization seems to be culprit

[city96]

I think there's some issues with pytorch not de-allocating memory, as well as comfy not being aware what the actual model size is (the tensors are not parameters of the model, so it may be reading the model as taking 0 MB of VRAM, causing the odd behavior with every second run working).

[city96]

Okay, I added two fixes that should hopefully make it work with lowvram mode in addition to taking less CPU memory. Could you guys do a git pull and maybe re-test?

[XylitolJ]

Can someone help me, it runs very slow on my machine, I see a warning line in the console but don't know how to fix it

Below is a picture when running with the original version

[ycyy]

Can someone help me, it runs very slow on my machine, I see a warning line in the console but don't know how to fix it有人可以帮我吗，它在我的机器上运行得很慢，我在控制台中看到一条警告线，但不知道如何修复它

Below is a picture when running with the original version下面是使用原始版本运行时的图片

I had a similar problem

[fatality14]

Okay, I added two fixes that should hopefully make it work with lowvram mode in addition to taking less CPU memory. Could you guys do a git pull and maybe re-test?

Works as intended now for Q4. At least with my 8Gb GPU. Thanks!