Add Nvidia Frame Buffer Capture (NVFBC and NVIFR) source please for hookless high performance capture in games!

[v00d00m4n]

Nvidia uses hardware FBC for its own streaming and recording software Expirience. Steam also uses it for streaming to steam link.

They work independant from any API uses in game or sofware. FBC effective for full screen framebuffer capture and IFR for windowed capture, her some details:

NVFBC

Captures the framebuffer (front buffer) without any involvement from OpenGL or Direct3D.

Effectively a direct copy of the framebuffer irrespective of which application(s) drew it.

It generally only works sensibly in fullscreen mode. If you render in windowed mode and use NVFBC, it is going to capture the entire screen including your desktop and other unrelated windows.

NVIFR

Slightly more complicated and less performant than NVFBC, this can capture a single application.

In my experience this used to be how Steam would stream windowed-mode applications. I have not seen this capture path in a very long time and I am glad because performance was awful whenever it was used.

Here some documentation https://developer.nvidia.com/sites/default/files/akamai/designworks/docs/NVIDIA%20Capture%20SDK%20Programming%20Guide.pdf

http://on-demand.gputechconf.com/gtc/2016/presentation/s6307-shounak-deshpande-get-to-know-the-nvidia-grid-sdk.pdf

Here the capture SDK itself: https://developer.nvidia.com/capture-sdk

ATI also has something similar, but im not very familiar with it, maybe google it.

Anyway, implementation of this will reduce CPU load and performance impact completely, please do this as soon as you can.

[v00d00m4n]

Anyone?

[zomfg]

Try this:
get the last build of Prismatik, set it to whatever framerate you realistically want to use with your LEDs (or something like 30fps to make it simple)
run TimeSpy for example

quit Prismatik, setup Shadowplay with same framerate and a low quality preset (I don't know if all this is possible), start capture
run same benchmark

and a run of benchmark alone

report results

[v00d00m4n]

Well its not much about GPU but about CPU unloading, which not always affects games, since not all of them CPU performance sensitive, and its also about compatibility and hardware accelerated capture. NVFBC allows to capture any fullscreen app no matter what API it uses, since capture is taken directly from videocard frame buffer so no hooks and CPU processing required at all. In similar fashion NVIFR allows to capture desktop and anything windowed, even some windows that does not work wind desktop dublication, you can also capture game which uses unsupport api like Vulkan, OpenGL or some old DirectDraw games or even some UWP apps with video stream (try IVI uwp from windows store, it video stream does not get capture by prizmatic, only UI affects it). And most important - its very low latency. Basically its low level direct hardware access to frame buffer, you may capture even some mad skills asm written demos that does not use dx or ogl or vk and instead pure low level with direct GPU access.

Here is example of open source software that uses NVFBC https://github.com/gnif/LookingGlass/tree/master/host/Capture just examine this code and repeat it, and download SDK for headers, and thats it.
(as a bonus this one also has nice DXGI capture code, implement it to prismatic as well would a nice alternative to find sweet spot with good performance and compatibility).

Another even better example of usage of NVFBC in open source software is here
https://github.com/bloodelves88/CloudyNvCapture

Please, take a look.

[zomfg]

I'm not arguing, I just wanted to have a rough idea of performance difference which I can not test myself (and neither other benefits since I'm on the red team right now).
Also, unless we are talking crazy capture framerates (and even then..), Prismatik's CPU load comes mainly from averaging the colors of widgets, especially with full sized frames (or more accurately, depending on the resolution and widget size). To help with this, the last release includes downscaling for ddupl (width/8 and height/8) which is made by the GPU. So unless color averaging is ported to CUDA or equivalent, I doubt we'll see a significant difference in performance (if any, I don't have any experience with GPGPU, I'm just speculating). But even then, it'll mainly benefit (IF it benefits at all) the old capture methods that only get full sized frames.
I'm saying this just to lower your expectations in the performance department, but I'm still curious to see some numbers, so if you get a chance to bench...

[psieg]

I also don't expect too much of an improvement. Desktop Duplication is also largely API independent (no injection) and hardware optimized. It no doubt has better compatibility than anything Nvidia specific.
Anyone interested is welcome to give it a shot, but since my time is limited these days, don't get your hopes up for me doing it.

[sblantipodi]

@psieg now that Desktop Duplication is broken (it disables variable refresh rate) it could worth a try.
is it a lot of work to add NVFBC and NVIFR support?

[Benik3]

Is it nVidia exclusive or it will work also on AMD?

[sblantipodi]

it's nvidia but amd has its own

[Benik3]

AMD has ReLive and I didn't find any API. So it mean add to Prismatik two new captures - one for nVidia and second for AMD...
But how e.g. twitch connects to these streaming? Isn't possible use this way to get the picture, so it would be universal for AMD or nVidia? I don't know much about this, but maybe it will help someone to find a way :)

[zomfg]

@psieg now that Desktop Duplication is broken (it disables variable refresh rate) it could worth a try.
is it a lot of work to add NVFBC and NVIFR support?

Find something that uses one of those and compare to ddupl

[v00d00m4n]

I also don't expect too much of an improvement. Desktop Duplication is also largely API independent (no injection) and hardware optimized. It no doubt has better compatibility than anything Nvidia specific.
Anyone interested is welcome to give it a shot, but since my time is limited these days, don't get your hopes up for me doing it.

Improvement is huge - DD is stil a software capture that requires translation of many api calls into other api calls and driver calls and it adds noticable latency, and uses CPU resources, ITS HIGH LEVEL. Yet again NVFBC and NVIRC are PURE HARDWARE LOW LEVEL capture that skips all the middle men apis and talks directly to hardware, skipping a lot of unnecessary stuff.

DD cant be as fast as NVFBC because in battle SOFTWARE vs HARDWARE accelerated, anything HARDWARE ACCELERATED always wins. Same goes for HIGH vs LOW level, low level wins.

Difference in resource usage and performance is the same as difference between OBS and NVidia epxirience streaming - OBS is software capture tool and waste CPU and slows down games and you loosing like 2-10 fps depending on game. Nvidia Expirience on the other hand absolutely does not load CPU and does not waste any FPS during streams. OBS usually adds 10-15% CPU load and NVE adds 0% cpu load. And just to compare - right now with DD Prizmatik eats about same 15% of CPU and i guess implementation of NV or ATI direct hardware framebuffer capture will reduce load to less than 3-5%, which is quite a lot for heavy games.

Another reason why it needs to be done is compatibility - like i said before - DD does not always works, For example it does not capture video streams from UWP (just search for IVI in windows store and test with any video from it, it has plenty of free movies), maybes its done for DRM sake, to prevent video capture via DD to not allow anyone to record video streams from paid services, im not sure why i happens, but with NVFBC nad NVIFR i can easily record same UWP aplication and anything, even DRM protected web streams, because NVFBC, yet again, does not work with other APIS, it directly takes whatever is now in frame buffer of video card, and since Windows UI is compossed via video card, now matter how hard they try to DRM, they have to put video stream layer in overall desktop composition, so its exist there as is.

Also it works with old games without any dependency on API used to render things, anything that renders via hardware video card buffer gets capture no matter tha API it uses. Which is again not so with DD. You can find many reports on internet that DD does not capture everything. And since you dont need DX hooking - its a better both for compar and performance and even also is safer for using in anti cheat protected games and games that has some mods that already does DX hooking.

As far as i know - Steam has all 3 API implemented for streaming - DD for general purpose, NVFBC for Geforces and ATI thing for Radeons, plus it uses NVENC for hardware lagless encoding of video stream which also improves performance alot comparing to DD and software encoding steam does on stream if no NV or API option is selected.

So DD is good in general purpose as a fail safe method, while NV and ATI apis are better for gaming and even DRM protected streamed videos.

As for implementation - please read all the PDFs and source examples i linked in my posts above - its very easy, all you have to do - copy past general code "shell" you use for DD and just replace DD calls with NVFBC and ATI api calls , and do some minor tweaks. I thinks its even possible to use direct frame buffer transition to video card memory to reduce its size and sample zone colors, but im not sure about this part.

P.S - Oh btw, recent SDK may be limited to use with pro cards only, so if it would not work, just look for older version of SKD that did not had such restriction.

[v00d00m4n]

For AMD capture api i think you need to look at this https://gpuopen.com/gaming-product/advanced-media-framework/ but im not AMD user and dont know much about them, so im not sure.