ambientMode: Proposal to prepare an application method other than addition of ambientColor

[inaridarkfox4231]

Increasing Access

Currently, the way ambientColor is applied in phongFrag is simply adding up the calculated color values.
This is a layer application method called addition in illustration.

In illustration, however, there are many ways to apply color other than additive.
The four most commonly used are addition, multiplication, screen, and overlay. Otherwise, soft light and hard light may be used.

So I wanted to be able to apply it. This issue suggests it.

In the current situation where the only way to apply ambientColor is simply adding up, for example if the directionalLight is pure white, it will always be pure white if the ambientLight is pure white.
However, for example, if the application method is an overlay, the light and dark will be sharp when it is pure white.
Overlay is useful because the effect of approximating the applied color is easier to understand than additive.

The following image shows ambientColor applied to a gray torus using ADD and OVERLAY methods, respectively, using a feature to be implemented.
condition:

directionalLight(255, 255, 255, 0, 0, -1);
ambientLight(255);
ambientMaterial("white");
specularMaterial(64);
shininess(40);
fill(128);

The more options you have, the more expressions you can express, so I thought it would be meaningful to implement it.

Most appropriate sub-area of p5.js?

• Accessibility
• Color
• Core/Environment/Rendering
• Data
• DOM
• Events
• Image
• IO
• Math
• Typography
• Utilities
• WebGL
• Build Process
• Unit Testing
• Internalization
• Friendly Errors
• Other (specify if possible)

Feature request details

I will explain how to implement it.

First, let Renderer.GL have an integer property called ambientMode. This is used as a flag in phong.frag.

  this.ambientMode = 0; // default is 0 : ADD

The correspondence with blendMode is 0: ADD, 1: SCREEN, 2: MULTIPLY, 3: OVERLAY, 4: SOFT_LIGHT, 5: HARD_LIGHT.
Then have a function called ambientMode() (kind of like textureMode()). Change the value of ambientMode by substituting the above blending constant into the argument of this. For example, if you want the application method to be OVERLAY, use ambientMode(OVERLAY).
In this function, we convert the blending constant to an integer case by case. Here is an overview of the functions:

p5.prototype.ambientMode = function (mode) {
  this._assert3d('ambientMode');
  p5._validateParameters('ambientMode', arguments);
  if (mode === ADD){ this._renderer.ambientMode = 0; }
  else if (mode === SCREEN){ this._renderer.ambientMode = 1; }
  else if (mode === MULTIPLY){ this._renderer.ambientMode = 2; }
  else if (mode === OVERLAY){ this._renderer.ambientMode = 3; }
  else if (mode === SOFT_LIGHT){ this._renderer.ambientMode = 4; }
  else if (mode === HARD_LIGHT){ this._renderer.ambientMode = 5; }
  else {
    console.warn('There are only 6 blendModes that can be used to apply ambientColor: ADD, SCREEN, MULTIPLY, OVERLAY, SOFT_LIGHT, HARD_LIGHT.');
  }
  return this;
};

Then pass this value to uAmbientColor in _setFillUniforms().

fillShader.setUniform('uAmbientMode', this.ambientMode);

Receive this in phong.frag and use it in the ambientColor application part. Rewrite it like this:

before:

  gl_FragColor = vec4(diffuse * baseColor.rgb + 
                    vAmbientColor * uAmbientMatColor.rgb + 
                    specular * uSpecularMatColor.rgb + 
                    uEmissiveMatColor.rgb, baseColor.a);

after:

  diffuse *= baseColor.rgb;
  diffuse += specular * uSpecularMatColor.rgb;
	
  vec3 ambient = vAmbientColor * uAmbientMatColor.rgb;
  if (uAmbientMode == 0) { diffuse += ambient; } // ADD.
  if (uAmbientMode == 1) { diffuse = diffuse + ambient - diffuse * ambient; } // SCREEN.
  if (uAmbientMode == 2) { diffuse *= ambient; } // MULTIPLY.
  if (uAmbientMode == 3) { diffuse = overlay(ambient, diffuse); } // OVERLAY.
  if (uAmbientMode == 4) { diffuse = softLight(ambient, diffuse); } // SOFT_LIGHT.
  if (uAmbientMode == 5) { diffuse = overlay(diffuse, ambient); } // HARD_LIGHT.

  diffuse += uEmissiveMatColor.rgb;

  gl_FragColor = vec4(diffuse, baseColor.a);

overlay and softLight are calculated by creating separate functions.

// overlay.
vec3 overlay(in vec3 src, in vec3 dst){
  vec3 result;
  if(dst.r < 0.5){ result.r = 2.0*src.r*dst.r; }else{ result.r = 2.0*(src.r+dst.r-src.r*dst.r)-1.0; }
  if(dst.g < 0.5){ result.g = 2.0*src.g*dst.g; }else{ result.g = 2.0*(src.g+dst.g-src.g*dst.g)-1.0; }
  if(dst.b < 0.5){ result.b = 2.0*src.b*dst.b; }else{ result.b = 2.0*(src.b+dst.b-src.b*dst.b)-1.0; }
return result;
}

// softLight.
vec3 softLight(in vec3 src, in vec3 dst){
  vec3 result;
  if(src.r < 0.5){ result.r = 2.0*src.r*dst.r + dst.r*dst.r*(1.0-2.0*src.r); }
  else{ result.r = 2.0*dst.r*(1.0-src.r) + sqrt(dst.r)*(2.0*src.r-1.0); }
  if(src.g < 0.5){ result.g = 2.0*src.g*dst.g + dst.g*dst.g*(1.0-2.0*src.g); }
  else{ result.g = 2.0*dst.g*(1.0-src.g) + sqrt(dst.g)*(2.0*src.g-1.0); }
  if(src.b < 0.5){ result.b = 2.0*src.b*dst.b + dst.b*dst.b*(1.0-2.0*src.b); }
  else{ result.b = 2.0*dst.b*(1.0-src.b) + sqrt(dst.b)*(2.0*src.b-1.0); }
  return result;
}

That's all there is to the implementation.

Let me explain another motivation. Watch the video below.
noise&rotate

2023-05-01.04-06-02.mp4

As you can see, the result of applying lights() is much lighter in 1.6.0 than in 1.4.2. This is due to a change in how lighting is applied in 1.5.0.

This is not 128, 128 like lights(), but 255, 255 are set to MAX, and the result of applying with OVERLAY is shown here.

2023-05-01.04-12-56.mp4

In this way, the light and dark became clear and different appearance.

Appearance is a matter of individual subjectivity, so there is no correct answer. So this specification change does not change the default at all. AmbientMode defaults to ADD, so if you don't do anything, it will remain the same and backward compatibility will be maintained. What I want to do is increase the options in drawing.

I would like to add a simple benchmark test etc. later.

[inaridarkfox4231]

Below are the differences between the six modes. The torus is colored gray with fill(128).
ambientMode

2023-05-01.10-43-42.mp4

As for performance, first I tried it on a browser with a simple code that rotates 800 cubes and emits them from the center, and there was not much difference.

noAmbientMode(compare)
ambientModeTest

2023-05-01.10-55-58.mp4

It seems that the branch of if and the slightly complicated calculation of overlay and softLight are not very related.

[inaridarkfox4231]

Next, regarding mobile performance, the number of cubes is 300 because it is heavy.

First of all, the performance of those who applied the patch that changed the specifications was around 30fps.

az_recorder_20230501_110142.mp4

Next is the normal version, but again there was no big difference around 30fps.

az_recorder_20230501_110216.mp4

So I don't think this is a problem either.

[davepagurek]

Thanks for writing this specification! I think it's a cool idea. I'm not sure just yet if it should be part of the default lighting system though. I'll explain a little about why the current system is the way it is, and what future considerations we're weighing when looking at changes.

As you can see, the result of applying lights() is much lighter in 1.6.0 than in 1.4.2. This is due to a change in how lighting is applied in 1.5.0.

For some background, you can achieve the same behaviour again by setting the ambient material to be the same color as the specular material. This was done because Processing also lets you define these separately, although in Processing, the ambient material defaults to the fill material if you don't explicitly set it. That detail is probably a feature we should add to p5 too, to make colors look like they did in 1.4 by default.

Currently, the way ambientColor is applied in phongFrag is simply adding up the calculated color values.

Some more background about additive light is that there's some physical basis for it: real-world light is additive in energy, and ambient light is an approximation for the indirect light an object receives, approximating it as being equivalent to receiving a uniform amount of light from all sides. So one way to look at the lighting system is through a physical metaphor: in addition to setting material properties, one can ask how much light the object is receiving from specific point lights, and also from all sides from the environment. (Some extra nuance is that the sRGB color space used by our screens is not linear in energy; threejs recently made a change to convert to energy-linear values first, then perform additive light calculations, then convert back to sRGB in order to address this.)

Of course, a physical metaphor isn't the only way to light a scene, and non-photorealistic rendering (NPR) is an interesting active research area of its own, which the system you describe seems to fit into more. It's nice that it would be consistent with blendMode, so if we opt for an NPR lighting system, it's a nice bonus that programmers would be able to apply that knowledge here.

I think the big questions here are whether p5's lighting system should use a physical metaphor or be more like an NPR system, and how much flexibility should p5's lighting system have. p5 Intentionally doesn't try to be threejs, so being perfectly physically accurate and having lots of material options doesn't have to be a goal, and we also don't want to add too much complexity so that there aren't too many parts to learn. Last time @aferriss and I talked, we felt like we should try to keep p5's lighting system close to Processing's as a way to keep things simple and approachable, and Processing's uses this physical metaphor. This would mean probably not having blend mode options right in the material system unfortunately.

I'm not fully committed to not changing the system though -- one of the Google Summer of Code project ideas we received a few applications for involves adding image-based lighting as a way of replacing constant ambient light with the light from a 360-degree image instead. That proposal still fits into this physical metaphor, but also makes it maybe easier to set up a cohesive scene by dropping in an image from any of the HDRI sites on the internet to get lights from a real-world environment. This too might not be a core lighting change though, and might end up being a library.

However, I think there's a lot of value in having NPR options like the blend modes you describe. I'm wondering if maybe we can do some work on the lighting system to make it easier for people to make libraries for p5 to add a material system like this. Right now, I think a hypothetical lighting library for p5 would have to come in the form of a shader you can set, although it would be hard to write one, since we don't do a great job of documenting anywhere what attributes and uniforms will be provided, and parts you don't change (e.g. the vertex shader) need to be copy-and-pasted from the p5 source code. Maybe we could try to design a system to let people replace specific parts of the lighting system?

Sorry for the long comment, there's just a lot of ideas surrounding this system! To summarize:

• The current policy is to default to what Processing does
  • The lighting difference introduced in 1.5 should probably be corrected (the default values currently deviate from Processing)
• This would make a great library though, maybe we can try to make it easier to make lighting system libraries?
• We can also discuss this some more, the current policy can still be changed

[inaridarkfox4231]

you can achieve the same behaviour again by setting the ambient material to be the same color as the specular material.

I'm not sure, but the specular should determine the color of the reflected light, and I feel that it plays a different role than the ambient color added later.

Some more background about additive light is that there's some physical basis for it

I can understand the claim that the ambient color is added by addition as a result of physical correctness, but the purpose is to add a nice color tone, not to pursue physical correctness. , I think it's a little different.
Also, with the spec change I'm proposing here, simply adding up ambient colors will remain the default method. There is still a way to pursue correct lighting with the current specifications, but is there any problem?

As for reona396, who drew the above sketch, she has been working as a creative coder for more than three years, using p5js.
However, looking at some recent sketches, I can see that some of the lighting is done with lights(). The sketch I introduced in the comments is also from the end of January this year, but it only uses lights(). It seems that the version was reverted for that reason.
The reason for revert version is probably simple. “it looks better that way” There is a gap in consciousness regarding lighting processing. Most people probably don't give it that much thought. I can't tell without looking at other people's sketches...

Personally, I don't feel very comfortable with coders reverting and using versions. There are still remained bugs that we have fixed already, and even if we implement new features, they won't be used. Even though the alpha calculation was correct in 1.6.0, it can't be used. The lighting process may need to be more intuitive.

The method I'm proposing here certainly falls under the NPR you're talking about, and I don't think it's physically correct, but as an option for coders who aren't good at lighting processing, it's a little I personally think it might work (or rather, I'm not very good at it, too).

Maybe we could try to design a system to let people replace specific parts of the lighting system?

First of all, regarding the people I see on Twitter and elsewhere, most of the people who call themselves creative coders are not using their own shaders or using custom shaders.
In the first place, glsl itself is thought to be difficult. Even if glsl can be written, it is not easy to learn how to modify and use a part of the vertex shader or fragment shader. This is because the direction of technology is different from doing so-called board polygraphy with a tool like shadertoy.
To do that, you need to learn webgl from the basics on the following sites (wgld.org). A good background in mathematics helps, but math is also considered difficult.
On the other hand, if you have enough knowledge and skills to modify and use it, copying and pasting and using it will not be a big burden (although I think there are individual differences).

Or are there any plans to improve the current simple ambient color adding up process to make the lighting process more comfortable for the coder, I don't know.
In any case, even if this proposal is realized, the current specifications will remain, so it is possible to present another option for now and prepare a mechanism that can realize proper lighting processing with the current specifications. I think there is.

[davepagurek]

I'm not sure, but the specular should determine the color of the reflected light, and I feel that it plays a different role than the ambient color added later.

How did you feel about the lighting system in 1.4.2? To elaborate on what I was saying about achieving the same behaviour as before, here's a comparison using the sketch you linked:

// v1.4.2 fill(this.c)	// v1.6.0 fill(this.c)	// v1.6.0 const c = color(this.c) ambientMaterial( red(c), green(c), blue(c) ) fill(this.c)

It's obviously not great that you now have to duplicate the fill color in the ambient color, which I'd consider a bug (introduced in 1.5; a partial fix was added in 1.6 but did not fully fix the problem.) So that should definitely be addressed regardless. I've just filed a bug for it here #6125 so that we can address this particular bit separately. But whether we should change the material system more is its own discussion.

First of all, regarding the people I see on Twitter and elsewhere, most of the people who call themselves creative coders are not using their own shaders or using custom shaders.

Definitely agreed with this. If we were to try to make the lighting system changeable, I think the idea would be that we would implement something like the blend mode system as a library for p5 that one would import (similar to p5.sound or p5.EasyCam.) But right now it's still not very feasible for library developers to make lighting libraries, as it involves copy-and-pasting (so library developers need to know glsl AND how to navigate the p5 source code) and still requires the end user to know a bit about P5 shaders to use it (rather than having it integrate more directly with p5.) Maybe better material infrastructure within p5 would let a blend mode library exist that feels "native" for end users?

The reason for revert version is probably simple. “it looks better that way”

I think the main reason me pushing back on making a bigger change to the default material system is that, beyond the default materials, what looks good is very subjective to the artist and what they hope to achieve. I agree that your changes are useful ones, but we also get requests for other features like fog and more physically-based materials, and it's hard to decide which of these are more important than others, as all have value for some projects, and doing all of them makes for a large library. So if we can make our library extendable (maybe even having an official but optional materials library, sort of like p5.sound) could be a way of letting us build all of those?

[inaridarkfox4231]

Thank you. I'll leave it up to you to fix that bug.

Unlike fog effects and physically-based rendering, is it difficult just to devise ways to apply colors? Even if it doesn't come close to Three.js, I think p5.js should take advantage of the goodness of p5.js.
Also, I'm not trying to introduce blend modes in general, but just adding a few major methods. I'm not trying to implement everything.

[davepagurek]

Carrying over some of the discussion from #6125: I think a good reason to make some changes in core p5 instead of relying on libraries to enhance material functionality would be if the existing core functionality doesn't work with all aspects of the material system, and per-vertex fills might be one such feature.

For that, could it make more sense to instead let users define how to blend their fill color (either through per-vertex colors or a solid fill) with the color provided in ambientMaterial/specularMaterial, and then the resulting colors are added together using the same addition as usual? This way we would keep the same physical metaphor as Processing and the same linear lighting setup common across most graphics systems, but still have a way to tweak each material component even if you have vertex colors. Do you think that would provide enough flexibility?

We would still want defaults that match the 1.4.2 behaviour, but would give you the flexibility to say something like, "the ambient material should be the same as my per-vertex colors but tinted a bit more red."

[inaridarkfox4231]

I think the main reason me pushing back on making a bigger change to the default material system is that, beyond the default materials, what looks good is very subjective to the artist and what they hope to achieve. I agree that your changes are useful ones, but we also get requests for other features like fog and more physically-based materials, and it's hard to decide which of these are more important than others, as all have value for some projects, and doing all of them makes for a large library. So if we can make our library extendable (maybe even having an official but optional materials library, sort of like p5.sound) could be a way of letting us build all of those?

I would like to close this issue in anticipation of further development of p5.js. I hope that the lighting process will be easier to use. Thank you for your discussion.