请教一下 real 模式下如何进行内存管理，避免输入参数没改变的节点仍然重新推理一遍呢？

[binarytahr]

我缺乏 Python 或者其它代码开发方面的基础知识，自己搜索了一下，看到有用@cached @lru_cache 等 decorator 去做缓存的，不知道这是否是文档中提到的自己管理缓存的正确方法呢？如果不是，能否麻烦给一下相关方法的关键词我去查一查，谢谢。

另外，感谢一下做这个repo，我之前用 huggingface 的 diffusers ，感觉对于我这种菜鸟来说，里面的 class function 设计的抽象化程度要么有点整要么有点过细，没有 comyui 的颗粒度适合我。之前就在想要是 comfyui 有 library 的话就好了，但是我自己无从下手，没想到真的有人做出来了。

[Chaoses-Ib]

cache 和 lru_cache 应该是用 == 对新旧参数进行比较的，部分类型可能不支持。目前 real 模式会对 node 输出附加工作流信息到 _self_virtual_output 属性来实现自动保存工作流，理论上可以自定义下 cache，用附加的工作流进行比较。这几天我实现一下吧，现在可以先试试用 cache 或者手写比较。

[binarytahr]

cache 和 lru_cache 应该是用 == 对新旧参数进行比较的，部分类型可能不支持。目前 real 模式会对 node 输出附加工作流信息到 _self_virtual_output 属性来实现自动保存工作流，理论上可以自定义下 cache，用附加的工作流进行比较。这几天我实现一下吧，现在可以先试试用 cache 或者手写比较。

谢谢解答，听起来是通过比较 _self_virtual_output 而非输入参数来实现，我先去学习一下 cache 的具体玩法。

另外，我发现同样的 fp16 模型在 kaggle 的 Tesla T4 GPU 上推理时，RAM VRAM 的占用以及推理所用时长都没 diffusers 表现的好，不知道这是正常的吗，或者是有什么地方我没设置好

[Chaoses-Ib]

我之前看到的讨论都是说 ComfyUI 比 diffusers 要快的，配置有问题的可能性比较大。用 torch 2.0 以上，再安装下 xformers 可能会有帮助。另外也可以试试下面这些命令行参数，比如 VRAM 足够的话可以用 --highvram：

ComfyScript/src/comfy_script/runtime/__init__.py

Lines 72 to 147 in b5ddc48

	usage: [-h] [--listen [IP]] [--port PORT] [--enable-cors-header [ORIGIN]] [--max-upload-size MAX_UPLOAD_SIZE] [--extra-model-paths-config PATH [PATH ...]] [--output-directory OUTPUT_DIRECTORY]
	[--temp-directory TEMP_DIRECTORY] [--input-directory INPUT_DIRECTORY] [--auto-launch] [--disable-auto-launch] [--cuda-device DEVICE_ID] [--cuda-malloc | --disable-cuda-malloc]
	[--dont-upcast-attention] [--force-fp32 | --force-fp16] [--bf16-unet | --fp16-unet | --fp8_e4m3fn-unet | --fp8_e5m2-unet] [--fp16-vae | --fp32-vae | --bf16-vae] [--cpu-vae]
	[--fp8_e4m3fn-text-enc | --fp8_e5m2-text-enc | --fp16-text-enc | --fp32-text-enc] [--directml [DIRECTML_DEVICE]] [--disable-ipex-optimize] [--preview-method [none,auto,latent2rgb,taesd]]
	[--use-split-cross-attention | --use-quad-cross-attention | --use-pytorch-cross-attention] [--disable-xformers] [--gpu-only | --highvram | --normalvram | --lowvram | --novram | --cpu]
	[--disable-smart-memory] [--deterministic] [--dont-print-server] [--quick-test-for-ci] [--windows-standalone-build] [--disable-metadata] [--multi-user]
	options:
	-h, --help show this help message and exit
	--listen [IP] Specify the IP address to listen on (default: 127.0.0.1). If --listen is provided without an argument, it defaults to 0.0.0.0. (listens on all)
	--port PORT Set the listen port.
	--enable-cors-header [ORIGIN]
	Enable CORS (Cross-Origin Resource Sharing) with optional origin or allow all with default '*'.
	--max-upload-size MAX_UPLOAD_SIZE
	Set the maximum upload size in MB.
	--extra-model-paths-config PATH [PATH ...]
	Load one or more extra_model_paths.yaml files.
	--output-directory OUTPUT_DIRECTORY
	Set the ComfyUI output directory.
	--temp-directory TEMP_DIRECTORY
	Set the ComfyUI temp directory (default is in the ComfyUI directory).
	--input-directory INPUT_DIRECTORY
	Set the ComfyUI input directory.
	--auto-launch Automatically launch ComfyUI in the default browser.
	--disable-auto-launch
	Disable auto launching the browser.
	--cuda-device DEVICE_ID
	Set the id of the cuda device this instance will use.
	--cuda-malloc Enable cudaMallocAsync (enabled by default for torch 2.0 and up).
	--disable-cuda-malloc
	Disable cudaMallocAsync.
	--dont-upcast-attention
	Disable upcasting of attention. Can boost speed but increase the chances of black images.
	--force-fp32 Force fp32 (If this makes your GPU work better please report it).
	--force-fp16 Force fp16.
	--bf16-unet Run the UNET in bf16. This should only be used for testing stuff.
	--fp16-unet Store unet weights in fp16.
	--fp8_e4m3fn-unet Store unet weights in fp8_e4m3fn.
	--fp8_e5m2-unet Store unet weights in fp8_e5m2.
	--fp16-vae Run the VAE in fp16, might cause black images.
	--fp32-vae Run the VAE in full precision fp32.
	--bf16-vae Run the VAE in bf16.
	--cpu-vae Run the VAE on the CPU.
	--fp8_e4m3fn-text-enc
	Store text encoder weights in fp8 (e4m3fn variant).
	--fp8_e5m2-text-enc Store text encoder weights in fp8 (e5m2 variant).
	--fp16-text-enc Store text encoder weights in fp16.
	--fp32-text-enc Store text encoder weights in fp32.
	--directml [DIRECTML_DEVICE]
	Use torch-directml.
	--disable-ipex-optimize
	Disables ipex.optimize when loading models with Intel GPUs.
	--preview-method [none,auto,latent2rgb,taesd]
	Default preview method for sampler nodes.
	--use-split-cross-attention
	Use the split cross attention optimization. Ignored when xformers is used.
	--use-quad-cross-attention
	Use the sub-quadratic cross attention optimization . Ignored when xformers is used.
	--use-pytorch-cross-attention
	Use the new pytorch 2.0 cross attention function.
	--disable-xformers Disable xformers.
	--gpu-only Store and run everything (text encoders/CLIP models, etc... on the GPU).
	--highvram By default models will be unloaded to CPU memory after being used. This option keeps them in GPU memory.
	--normalvram Used to force normal vram use if lowvram gets automatically enabled.
	--lowvram Split the unet in parts to use less vram.
	--novram When lowvram isn't enough.
	--cpu To use the CPU for everything (slow).
	--disable-smart-memory
	Force ComfyUI to agressively offload to regular ram instead of keeping models in vram when it can.
	--deterministic Make pytorch use slower deterministic algorithms when it can. Note that this might not make images deterministic in all cases.
	--dont-print-server Don't print server output.
	--quick-test-for-ci Quick test for CI.
	--windows-standalone-build
	Windows standalone build: Enable convenient things that most people using the standalone windows build will probably enjoy (like auto opening the page on startup).
	--disable-metadata Disable saving prompt metadata in files.
	--multi-user Enables per-user storage.

用法是 load(args=ComfyUIArgs('--force-fp16', '--highvram')) 这样。

[binarytahr]

我之前看到的讨论都是说 ComfyUI 比 diffusers 要快的，配置有问题的可能性比较大。用 torch 2.0 以上，再安装下 xformers 可能会有帮助。另外也可以试试下面这些命令行参数，比如 VRAM 足够的话可以用 --highvram：

ComfyScript/src/comfy_script/runtime/__init__.py

Lines 72 to 147 in b5ddc48

	usage: [-h] [--listen [IP]] [--port PORT] [--enable-cors-header [ORIGIN]] [--max-upload-size MAX_UPLOAD_SIZE] [--extra-model-paths-config PATH [PATH ...]] [--output-directory OUTPUT_DIRECTORY]
	[--temp-directory TEMP_DIRECTORY] [--input-directory INPUT_DIRECTORY] [--auto-launch] [--disable-auto-launch] [--cuda-device DEVICE_ID] [--cuda-malloc | --disable-cuda-malloc]
	[--dont-upcast-attention] [--force-fp32 | --force-fp16] [--bf16-unet | --fp16-unet | --fp8_e4m3fn-unet | --fp8_e5m2-unet] [--fp16-vae | --fp32-vae | --bf16-vae] [--cpu-vae]
	[--fp8_e4m3fn-text-enc | --fp8_e5m2-text-enc | --fp16-text-enc | --fp32-text-enc] [--directml [DIRECTML_DEVICE]] [--disable-ipex-optimize] [--preview-method [none,auto,latent2rgb,taesd]]
	[--use-split-cross-attention | --use-quad-cross-attention | --use-pytorch-cross-attention] [--disable-xformers] [--gpu-only | --highvram | --normalvram | --lowvram | --novram | --cpu]
	[--disable-smart-memory] [--deterministic] [--dont-print-server] [--quick-test-for-ci] [--windows-standalone-build] [--disable-metadata] [--multi-user]
	options:
	-h, --help show this help message and exit
	--listen [IP] Specify the IP address to listen on (default: 127.0.0.1). If --listen is provided without an argument, it defaults to 0.0.0.0. (listens on all)
	--port PORT Set the listen port.
	--enable-cors-header [ORIGIN]
	Enable CORS (Cross-Origin Resource Sharing) with optional origin or allow all with default '*'.
	--max-upload-size MAX_UPLOAD_SIZE
	Set the maximum upload size in MB.
	--extra-model-paths-config PATH [PATH ...]
	Load one or more extra_model_paths.yaml files.
	--output-directory OUTPUT_DIRECTORY
	Set the ComfyUI output directory.
	--temp-directory TEMP_DIRECTORY
	Set the ComfyUI temp directory (default is in the ComfyUI directory).
	--input-directory INPUT_DIRECTORY
	Set the ComfyUI input directory.
	--auto-launch Automatically launch ComfyUI in the default browser.
	--disable-auto-launch
	Disable auto launching the browser.
	--cuda-device DEVICE_ID
	Set the id of the cuda device this instance will use.
	--cuda-malloc Enable cudaMallocAsync (enabled by default for torch 2.0 and up).
	--disable-cuda-malloc
	Disable cudaMallocAsync.
	--dont-upcast-attention
	Disable upcasting of attention. Can boost speed but increase the chances of black images.
	--force-fp32 Force fp32 (If this makes your GPU work better please report it).
	--force-fp16 Force fp16.
	--bf16-unet Run the UNET in bf16. This should only be used for testing stuff.
	--fp16-unet Store unet weights in fp16.
	--fp8_e4m3fn-unet Store unet weights in fp8_e4m3fn.
	--fp8_e5m2-unet Store unet weights in fp8_e5m2.
	--fp16-vae Run the VAE in fp16, might cause black images.
	--fp32-vae Run the VAE in full precision fp32.
	--bf16-vae Run the VAE in bf16.
	--cpu-vae Run the VAE on the CPU.
	--fp8_e4m3fn-text-enc
	Store text encoder weights in fp8 (e4m3fn variant).
	--fp8_e5m2-text-enc Store text encoder weights in fp8 (e5m2 variant).
	--fp16-text-enc Store text encoder weights in fp16.
	--fp32-text-enc Store text encoder weights in fp32.
	--directml [DIRECTML_DEVICE]
	Use torch-directml.
	--disable-ipex-optimize
	Disables ipex.optimize when loading models with Intel GPUs.
	--preview-method [none,auto,latent2rgb,taesd]
	Default preview method for sampler nodes.
	--use-split-cross-attention
	Use the split cross attention optimization. Ignored when xformers is used.
	--use-quad-cross-attention
	Use the sub-quadratic cross attention optimization . Ignored when xformers is used.
	--use-pytorch-cross-attention
	Use the new pytorch 2.0 cross attention function.
	--disable-xformers Disable xformers.
	--gpu-only Store and run everything (text encoders/CLIP models, etc... on the GPU).
	--highvram By default models will be unloaded to CPU memory after being used. This option keeps them in GPU memory.
	--normalvram Used to force normal vram use if lowvram gets automatically enabled.
	--lowvram Split the unet in parts to use less vram.
	--novram When lowvram isn't enough.
	--cpu To use the CPU for everything (slow).
	--disable-smart-memory
	Force ComfyUI to agressively offload to regular ram instead of keeping models in vram when it can.
	--deterministic Make pytorch use slower deterministic algorithms when it can. Note that this might not make images deterministic in all cases.
	--dont-print-server Don't print server output.
	--quick-test-for-ci Quick test for CI.
	--windows-standalone-build
	Windows standalone build: Enable convenient things that most people using the standalone windows build will probably enjoy (like auto opening the page on startup).
	--disable-metadata Disable saving prompt metadata in files.
	--multi-user Enables per-user storage.

用法是 load(args=ComfyUIArgs('--force-fp16', '--highvram')) 这样。

谢谢～是的，我之前用图形界面的 comfyui 也感觉比 diffusers 要快。

我环境里的 torch 是 2.0 以上的，不过我忘记安装 xformers 了，因为 diffusers 在新 torch 里面会自动使用 scaled dot product attention ，就不需要开 xformers 的 flash attention 了。 新换到 comfyui 我把 xformers 给忘记了。明天我试试加一下启动参数的效果。

[binarytahr]

cache 和 lru_cache 应该是用 == 对新旧参数进行比较的，部分类型可能不支持。目前 real 模式会对 node 输出附加工作流信息到 _self_virtual_output 属性来实现自动保存工作流，理论上可以自定义下 cache，用附加的工作流进行比较。这几天我实现一下吧，现在可以先试试用 cache 或者手写比较。

我查了一下 cache 和 lru_cache ，发现它们都不能用于缓存 list 等可变容器，而 comfyui 很多 node 的输出都是 list，虽然可以自定义 lru_cache 但是仍然会有浅拷贝的问题。

另一个问题是 cache 和 lru_cache 是针对 function 的，在 real 模式下每行都相当于一个 function，想实现 comfyui 那种以 node 为最小单位的自动缓存，好像需要对每行单独包成 function。

我现在只能手动注释掉某行代码，或者在loop中让某些 node 只运行一次。

[Chaoses-Ib]

先简单实现了下，还没仔细测试：

所有 node 使用同一 cache：

# Use `dict` (`{}`) for simple unbounded cache. For advanced cache, [cachetools](https://github.com/tkem/cachetools) or other libraries can be used.
cache = {}

with Workflow(cache=cache):
    model, clip, vae = CheckpointLoaderSimple(Checkpoints.sd_v1_4)
    conditioning = CLIPTextEncode('beautiful scenery nature glass bottle landscape, , purple galaxy bottle,', clip)
    conditioning2 = CLIPTextEncode('text, watermark', clip)
    latent = EmptyLatentImage(512, 512, 1)
    latent = KSampler(model, 2, 20, 8, 'euler', 'normal', conditioning, conditioning2, latent, 1)
    image = VAEDecode(latent, vae)
    image = SaveImage(image, 'ComfyUI')
    print(image)

with Workflow(cache=cache):
    model, clip, vae = CheckpointLoaderSimple(Checkpoints.sd_v1_4)
    conditioning = CLIPTextEncode('beautiful scenery nature glass bottle landscape, , purple galaxy bottle,', clip)
    conditioning2 = CLIPTextEncode('text, watermark', clip)
    latent = EmptyLatentImage(512, 512, 1)
    latent = KSampler(model, 2, 20, 8, 'euler', 'normal', conditioning, conditioning2, latent, 1)
    image = VAEDecode(latent, vae)
    image = SaveImage(image, 'ComfyUI')
    print(image)

del cache

每个 node 使用独立 cache：

node_cache = {}
cache = lambda node: node_cache.setdefault(node, {})

with Workflow(cache=cache):
    model, clip, vae = CheckpointLoaderSimple(Checkpoints.sd_v1_4)
    conditioning = CLIPTextEncode('beautiful scenery nature glass bottle landscape, , purple galaxy bottle,', clip)
    conditioning2 = CLIPTextEncode('text, watermark', clip)
    latent = EmptyLatentImage(512, 512, 1)
    latent = KSampler(model, 2, 20, 8, 'euler', 'normal', conditioning, conditioning2, latent, 1)
    image = VAEDecode(latent, vae)
    image = SaveImage(image, 'ComfyUI')
    print(image)

print(*node_cache.keys())
# CheckpointLoaderSimple CLIPTextEncode EmptyLatentImage KSampler VAEDecode SaveImage

with Workflow(cache=cache):
    model, clip, vae = CheckpointLoaderSimple(Checkpoints.sd_v1_4)
    conditioning = CLIPTextEncode('beautiful scenery nature glass bottle landscape, , purple galaxy bottle,', clip)
    conditioning2 = CLIPTextEncode('text, watermark', clip)
    latent = EmptyLatentImage(512, 512, 1)
    latent = KSampler(model, 2, 20, 8, 'euler', 'normal', conditioning, conditioning2, latent, 1)
    image = VAEDecode(latent, vae)
    image = SaveImage(image, 'ComfyUI')
    print(image)

del cache, node_cache

Note that for node output, any changes made by user code instead of nodes will be ignored.

[binarytahr]

❤️ 非常感谢，你反馈的速度好快啊，我先学习一下你的实现方法