Why is GPU inference slow?

[budgetdevv]

Find out why. I'm thinking because I am not using IOBindings.

[budgetdevv]

Apparently it is just slow on DirectML - Full GPU utilization when inferencing on CUDA.

I guess it can still be faster, though, via using IOBindings.

[budgetdevv]

Inference speed for full, non-quantized model is around 6 seconds on my 8700K RTX 3070 setup ( Fits nicely into VRAM )