Better anti-aliasing solution compatible with VR

[Frooxius]

Currently the only AA solution that really works in VR is FXAA, which tends to make things blurry.

If anyone's interested in tasks to work on, this might be a good one to take on!

There's a few approaches I could see:

• Getting SMAA to work in VR
• Getting MSAA to work with Deferred (this might be tricky, but from what I read is theoretically possible)
• Finding other AA algorithms

If anyone's interested in taking this on, we'd be happy to add new AA types to the FrooxEngine side enum (once we open-source Renderite.Shared.dll, you'll be able to PR this yourself too).

[0xFLOATINGPOINT]

Currently the only AA solution that really works in VR is FXAA, which tends to make things blurry.

If anyone's interested in tasks to work on, this might be a good one to take on!

There's a few approaches I could see:

* Getting SMAA to work in VR

* Getting MSAA to work with Deferred (this might be tricky, but from what I read is theoretically possible)

* Finding other AA algorithms

If anyone's interested in taking this on, we'd be happy to add new AA types to the FrooxEngine side enum (once we open-source Renderite.Shared.dll, you'll be able to PR this yourself too).

SMAA can work inside of VR!! Tho.. I am not sure it can be done on Unity 2019. My Unity 6000 Renderer has fully working SMAA in VR and its really nice!. It just kinda auto worked due to how I handled Post-processing in my renderer. I can mess around and see if my changes can be up-streamed into this repo. As per MSAA with Deferred this will never be a clean thing. Yes there have been hacks at it before but they are SUPER(like Unity 2017 max) old and never worked good in the first place and forget about VR support with it. This is just a hard limit of Unity's Built-In Render Pipeline and we will only get MSAA when moving away from Unity outright

[Comptr-user]

a hard limit of Unity's Built-In Render Pipeline

What about the newer Universal Render Pipeline? Would it be more feasable there, or would the work required to support URP be too difficult/time-consuming to be worth attempting?

[0xFLOATINGPOINT]

a hard limit of Unity's Built-In Render Pipeline

What about the newer Universal Render Pipeline? Would it be more feasable there, or would the work required to support URP be too difficult/time-consuming to be worth attempting?

Out of scope. Switching to URP(Universal Render Pipeline) Would mean rewriting all 88 shaders for URP. This would yes fix the problem but would be a unholy amount of work that is not worth it in the end. I did spend a bit of time looking into this and its just too much work to switch

[Frooxius]

Switching to URP would be a huge endeavor. It would require all of the shaders to be re-written for it, there's potentially compatibility problems to work through and so on.

The main reason we haven't done that is because if we're to put that amount of work into the renderer, we'd rather put it into switching away from Unity completely instead.

[0xFLOATINGPOINT]

Switching to URP would be a huge endeavor. It would require all of the shaders to be re-written for it, there's potentially compatibility problems to work through and so on.

The main reason we haven't done that is because if we're to put that amount of work into the renderer, we'd rather put it into switching away from Unity completely instead.

Yeah exactly its a crazy amount of work for not a huge net bump given we are switching away from Unity. If we going to stay with Unity then it would be a different situation

[Raidriar796]

I've been experimenting with this for a while and it's frustrating, but here's my findings:

• Getting SMAA to work in VR

SMAA does work in VR, at least with Reshade, but this is a limitation with Unity's implementation of SMAA for reasons unknown to me. If we want to be able to force SMAA to work, we might be able to add in a separate SMAA implementation? At that point it may not be worth it.

• Getting MSAA to work with Deferred (this might be tricky, but from what I read is theoretically possible)

MSAA can be used right now but with some major caveats. A while ago I wrote a mod that forces Resonite to use MSAA, but to achieve this I also had to force Resonite to exclusively use Forward rendering. While this does work, and looks quite fantastic, not everything is affected by it (some post processing effects like bloom were still aliasing, for example). In addition to this, Resonite's existing history of content is authored for Deferred, so the performance cost sky rockets, and if you turn down per pixel lights to compensate, content looks very weird and lighting shifts around constantly. Overall, it'd likely require some heavy modifications to Unity to either make MSAA work with Deferred, or find a different implementation, but in general it might not be worth it long term.

• Finding other AA algorithms

I have recently discovered the existence of CMAA (Conservative Morphological Anti Aliasing), which seems to be an improved combination of FXAA and SMAA in terms of cost and quality. While this one looks promising, the only existing implementation I've seen is for Unity URP. This could potentially be adapted to BRP, as CMAA is a post processing AA, but I am unsure on specifics right now. I am also unsure if this works in VR, but given it's similarities to FXAA and SMAA, I don't see why it couldn't work with VR.

Overall, I'm sure this does not inspire confidence, but there are still some unexplored avenues at least.

[Frooxius]

The source of the Unity's SMAA is part of this repo - the shaders & C# code for the PostProcessing package. There might be a way to get it to work in VR perhaps?

I haven't looked into it myself. But the code is there, so if anyone wants to experiment with it, this should make it easier to do so.

[Raidriar796]

I wasn't sure if that was included in the repo, I may take a look then. Setting up the old Unity version Resonite uses is gonna be messy since I'm on Linux. If I get around to it I'll see what I can do.

[DoubleStyx]

It's possible to implement MSAA for deferred rendering in theory, but it would require source-level changes to the Unity 2019 built-in render pipeline's deferred path. So it's probably not feasible here. I'm not confident it's even possible to implement in URP either even with scriptable pipelines since you don't have a direct way to change the sample count for the G-buffer.

One relevant source that you might have come across is this blog post which explains the high-level idea behind deferred MSAA.

[kitlith]

https://stackoverflow.com/a/73371803 describes an approach to deferred MSAA that is a bit closer to how MSAA works in the forward pipeline. In short, your G-buffer becomes multi-sampled, and you compute lighting on a per-sample basis. To optimize this so that it's not just a form of supersampling, you only do per-sample lighting on the edges by marking the edges in the stencil buffer.

As @DoubleStyx mentioned, this still isn't necessarily possible to implement in Unity. It might be interesting for a custom renderer to look at, though.

However, it's a little questionable Resonite in general, since messing with the stencil buffer would likely break existing content that touches the stencil buffer, which means we would need a find a different way to avoid sample-shading the whole scene. Though... if all materials that interact with stenciling are made to be forward rendered, you could resolve your deferred color first, clear the stencil buffer, then proceed with forward rendering everything else. I guess that would work without interfering with existing content.