Bugs in fill_outline shader

[jfirebaugh]

The fragment shader does smoothstep(1.0, 0.0, dist). But smoothstep's behavior is undefined when its first argument is >= its second argument. On macOS, it appears that the result is always 0.

Changing it to smoothstep(0.0, 1.0, dist) also doesn't work -- except for a band across the middle of the screen, the result value is always 1.0.

Using the following shader modification, you can see it's calculating the distance from the a horizontal line through the center of the viewport, rather than the desired value:

    float alpha = dist / 100.0;
    gl_FragColor = vec4(alpha, alpha, alpha, 1.0);

cc @kkaefer @ansis

[kkaefer]

Looks like this was broken from the very beginning: mapbox/mapbox-gl-js@63e9f71#diff-97267ac172ce8fd831ea9c2993add6a6R9

[kkaefer]

This is almost certainly the culprit for #1555

[kkaefer]

On macOS, it appears that the result is always 0.

How did you reach that conclusion? I'm seeing different results (no antialiasing) when manually setting float alpha = 0.0.

[jfirebaugh]

Fixed in #9699.