Is it necessary to train with a very high graphics card to be able to train

[zhenzi0322]

When training with four p40 graphics cards, the following error is always reported.

RuntimeError: CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 23.88 GiB total capacity; 22.90 GiB already allocated; 76.25 MiB free; 23.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[13700k]

Why PyTorch reserved 23.12 GiB in the first place?
And what is the actual CUDA memory requirement for training?

I saw lots of CUDA out of memory thing that sort of like blackbox but no explanation provided.

[Night-Chan]

I think this is a bug in pytorch. Using any graphics card i own, this error always appears whenever i attempt to do anything with a resolution higher than 512x512. Messing with the max split size, what i gathered is that pytorch reserves at the very least few megabytes too much memory, and that leaves just too little space to train or generate anything. Maybe im doing something wrong though, i dont have much experience with pytorch. The only solution for me was to decrease the resolution to 256x256 or 256x512, the latter working most but not all of the time.
Hope this helps