Request: Support for dynamically scalable effect resolution

[gkjohnson]

Hello! Great work on the effects.

It would be great to support dynamically scalable effect resolution for some of the more intensive effects or at least a fixed effect scale on instantiation. SSAO, for example, can be a bit expensive depending on how many samples you want to use but this can be improved by rendering it at half or quarter resolution. A depth / normal aware upscale might be needed afterward to remove some artifacts depending on the effect.

To explain a bit more in my application I iteratively increase / decrease the resolution of or enable / disable the effects based on the measured framerate of the web app. So if the application is running slowly we might lower the resolution scale of the SSAO or boom effects or even disable them until the framerate reaches an acceptable threshold. And inversely if we have some extra frame time we'll enable or improve effect quality until we see frame dips.

Here's something like I'm imagining:

bloomEffect = new BloomEffect();
composer.addPass(new RenderPass(scene, camera));
composer.addPass(new EffectPass(camera, bloomEffect));

// ...

bloomEffect.resolutionScale = 0.5;

Thanks again for the awesome work!

[vanruesc]

Hi!

It would be great to support dynamically scalable effect resolution for some of the more intensive effects or at least a fixed effect scale on instantiation.

Some effects do have a resolutionScale setting but this feature has been deprecated in postprocessing@6.6.0 after it was pointed out that this approach caused render inconsistencies. You can see the issue in this example (try changing the height of the output pane). Using a fixed size instead of a relative resolution scale produces reliable results across different devices and screen resolutions.

Blur effects like bloom usually need to be configured manually to look right for a given scene. The final parameter configuration is always tied to a specific resolution at which the bloom overlay will be rendered. Changing the resolution without adjusting the blur kernel size or blur scale would produce different results. So instead of scaling render resolutions based on the screen resolution, consider defining parameter configurations for specific fixed resolution settings and switch between those sets depending on the available performance budget.

Note that only the render height or width needs to be specified, either via the constructor or the resolution property. The counterpart will be calculated based on the aspect ratio.

SSAO, for example, can be a bit expensive depending on how many samples you want to use but this can be improved by rendering it at half or quarter resolution.

The effects included in this library do render things at lower resolutions whenever possible and are configurable in that regard as explained above. Although it makes sense to render SSAO at a lower resolution, I intentionally ditched down-scaling and blurring to avoid issues like haloing and flickering. However, I agree that it would be nice to have an option to trade quality for performance after all.

A depth / normal aware upscale might be needed afterward to remove some artifacts depending on the effect.

That's a nice idea! I'll try that out when I get a chance. I'm still worried about ugly gaps between objects and AO, though..

[gkjohnson]

Thanks for the example!

Changing the resolution without adjusting the blur kernel size or blur scale would produce different results. So instead of scaling render resolutions based on the screen resolution, consider defining parameter configurations for specific fixed resolution settings and switch between those sets depending on the available performance budget.

Right -- I think this would typically mean that effects need to be scale-aware and ensure that they are resolution independent in some way or that the parameters are specified in normalized screen space. Some effects may depend on exact pixel samples, though, which I'm not necessarily sure how to handle so generally.

Note that only the render height or width needs to be specified, either via the constructor or the resolution property. The counterpart will be calculated based on the aspect ratio.

Great I see now this is dynamically settable, now. I'll take a deeper look.

That's a nice idea! I'll try that out when I get a chance. I'm still worried about ugly gaps between objects and AO, though..

It might not get rid of artifacts entirely but it could help improve performance quite a bit and be "good enough" in most cases. My impression is that this is a common technique in modern games but I'm just diving into this stuff. And of course setting the scale to 1.0 would get you back to the current behavior.

Thanks again!

[vanruesc]

I think this would typically mean that effects need to be scale-aware and ensure that they are resolution independent in some way or that the parameters are specified in normalized screen space.

I contemplated going down this path before arriving at the fixed resolution solution. AFAIK, there's no trivial solution for scale-aware parameters. Say a developer specifies samples, density and decay for a GodRaysEffect and defines a resolution scale of 0.5. How would we calculate the "actual" samples, density and decay for the actual resolution? I wouldn't want to add complicated and potentially unpredictable heuristics to produce somewhat acceptable results. I'm open to suggestions, but I'm currently convinced that the fixed resolution approach is the best solution.

On a side note, games usually offer steplike settings and often hide the fine-grained parameters. Some MMOs like WoW and GW perform realtime optimizations in crowded areas, but I think they also adjust the settings more roughly while focusing on the critical stuff such as the number of animated characters.

And of course setting the scale to 1.0 would get you back to the current behavior.

Not quite, actually. Right now, SSAO is performed entirely in the main effect shader and uses no additional render operations. If we want to render AO into a down-scaled render target, the shader needs to be converted into an independent material first. The SSAO effect shader would then just blend the AO texture with the scene colors while taking the depth texture into account to filter artifacts. Assuming we'd add a resolutionScale parameter, a setting of 1.0 would result in more work because we'd still have to render AO into a full-res render target and then blend it with the scene colors. Performance would be worse than the current implementation in this case.

I just realized that SSAO also produces inconsistent results across different screen resolutions (the sampling radius is fixed, but the resolution isn't). The result still looks surprisingly good at very low resolutions, though, because the effect relies on the scene depth and normals.

[vanruesc]

I tried to add a resolution setting to the SSAOEffect but ran into a problem. The effect relies on depth and normals: depth is rendered as a full-res depth texture attachment and normals are rendered at the same resolution in an additional render pass using an override material. The issue is that undesired patterns start to appear when the height of the ambient occlusion render target is not a multiple of the depth and normal texture height.

For example, everything is fine if the screen resolution is 1920x480 and the SSAO resolution is 960x240. But if the screen resolution is 1920x490 and SSAO renders at 940x240, blocky patterns appear:

main480/ssao240 (good)	main490/ssao240 (bad)

None of the SSAO parameters affect the pattern. I tried basing the sampling step size on the main resolution but that didn't help. I'm trying to find a good solution, but all I can think of is requiring the SSAO resolution to be a multiple of the main render resolution. Any ideas?

Sandbox for direct testing: https://codesandbox.io/s/ssao-resolution-9xpel (check line 118 in App.js)

[drcmda]

but all I can think of is requiring the SSAO resolution to be a multiple of the main render resolution.

that sounds like something that wouldn't cause too much trouble in user land, no? i would most certainly use it if it came out like that.

[vanruesc]

that sounds like something that wouldn't cause too much trouble in user land, no?

I consider this solution a last resort because the dependence on the main resolution defeats the purpose of the fixed resolution setting. It would be too fragile.

I'm currently looking into depth/normal downsampling to fix the patterns. And while I'm at it I'll also implement depth-aware SSAO upscaling based on this article: https://eleni.mutantstargoat.com/hikiko/depth-aware-upsampling-6

Hopefully the downscaling will solve the pattern issue.

[gkjohnson]

I'm currently looking into depth/normal downsampling to fix the patterns. And while I'm at it I'll also implement depth-aware SSAO upscaling based on this article: https://eleni.mutantstargoat.com/hikiko/depth-aware-upsampling-6

I was taking a look at that article a little bit ago, as well. I haven't implemented depth aware upscaling but it seems like fixing the AO resolution to a 1/2 or 1/4 size like you suggested might be easiest. I'm not sure what I would do in the case where the main resolution is not a factor of 2, though...

I consider this solution a last resort because the dependence on the main resolution defeats the purpose of the fixed resolution setting. It would be too fragile.

I'm not sure how the AO effect works in this library but it seems like an effect where you could divide out the resolution to make the effect scale independent right? Though I know you mentioned it might be more complicated than that.

[vanruesc]

Hopefully the downscaling will solve the pattern issue.

Sadly, this didn't remove the pattern, but added more artifacts.

In summary, using a smaller render target for SSAO results in regular blocky patterns when the depth texture size is not a multiple of the SSAO render target size (had that backwards last time). Using a custom downsampling algorithm that picks the most representative depth in a 2x2 neighborhood from the high-res depth texture results in additional regular artifacts that look like broken lines. This issue could be a bug in the downsampling shader, but the resulting depth texture looks fine.

My guess is that there's a bug somewhere in the SSAO sampling algorithm. I'll try replacing it with another one to check if that affects anything.

it seems like an effect where you could divide out the resolution to make the effect scale independent right?

Yes, and if the SSAOEffect had a general fixed resolution setting, it would be easy to set it specifically to 1/2 or 1/4 the size of the main frame buffer.

[Ben-Mack]

SSAO effect has large performance impact, thus having independent res is much needed.
In my case, adding SSAO drops the framerate from near 60 to ~10fps on mobile, decreasing samples doesn't help much. I think compromise quality for performance is totally reasonable.

[vanruesc]

I revised the SSAO shader based on https://research.nvidia.com/publication/scalable-ambient-obscurance and managed to improve the overall quality of the effect. One important change is that the sampling radius will now be properly scaled based on how far the fragments are away from the camera. The new shader also handles the artifacts from the downsampled depth texture much better. Raising the bias factor slightly above 0.0 completely removes all these artifacts, including the blocky patterns that are caused by render target size mismatches.

While I was debugging the effect, I also found another issue that isn't too obvious: the sampling algorithm produces incorrect shadows at the screen edge when the edge touches a flat surface in a certain angle. Apparently this is a well-known limitation of SSAO, but I couldn't find an official fix for it. My guess is that most people don't bother with this because it's a bit of an edge case (no pun intended).

Anyway, I fixed this by skipping samples that lie outside the screen. The range check adds a few GPU cycles, but the performance impact should be minimal. Here are some screenshots:

Without range check	With range check

The effect is almost complete; I still need to add the depth-aware upscaling logic to the final blend shader and I have to figure out how to make all these changes backward-compatible. One particularly important change is that depth and normals are packed into a single floating-point texture which adds a new hardware requirement to the effect.

[gkjohnson]

@vanruesc those sound like some great improvements! World-space scale for AO size is a nice addition.

For fun I've been working on a GTAO implementation to learn a bit which is a bit more expensive but produces some really nice results. To help with low resolution and low sample count I've tried out the depth and normal-aware upscaling / blur and it looks pretty nice:

	Without Blur	With Blur
1 / 4 scale
1 / 2 scale
1 / 1 scale

The code's a bit messy with me just hacking around but you can check out the demo here -- the images were taken with 2 rotation and 6 steps which works out to 24 texture samples. Still trying to find a good jitter method to smooth out the initial sampling noise better. I'll be interested in seeing your AO looks up close with world space scaling.

Feel free to check out the code for the upscale blur if you want -- that's located here.

I haven't decided how far I'm going to take this yet but there's a bit of optimization that could be done, yet, but I'm pretty happy with the way it looks. The approach is probably a bit heavier than you'd want for lower end platforms, though, so I'm not sure how much real use it could get.

As an added bonus after seeing someones screen space illumination experiment bounce around on Twitter I added a color gather too so you can get a little bit of bounce light illumination:

Without Light Bounce	With Light Bounce	5x Exaggerated Light Bounce

It only works okay without the base color pass. The bounce light in the sponza scene makes it look more washed out than anything. Maybe MRT support will come some day 😬

[vanruesc]

Cool stuff, thanks for sharing! Haven't heard of GTAO before.

[vanruesc]

Added a resolution setting to SSAOEffect in postprocessing@6.14.0.