frontend/nine: fix wfog
When wfog support is advertised, unless an orthogonal
projection matrix is detected, w is supposed to be used
instead of z for the fog equation when done in the pixel
shader.
Due to the spec being ambiguous, and tests being incomplete,
it seems we had got things wrong.
New tests confirm the behaviour.
For the explanation we will denote z_vs and w_vs the position
output's z and w channels in the vertex shader, and
z_ps, w_ps the position input z and w channels in the pixel shader.
w_ps = 1/w_vs
z_ps = z_vs/w_vs
In the programmable pixel shader, we used z_ps/w_ps, thus z_vs.
As basically z_vs and w_vs are usually in the same range, we didn't
notice an obvious difference with the correct behaviour.
In the ff pixel shader, we used z_ps for zfog and w_vs else.
z_ps was always used if a programmable vertex shader was detected.
This latter behaviour led to issue
https://gitlab.freedesktop.org/mesa/mesa/-/issues/8341
While using z_ps/w_ps like for programmable ps fixes the issue visually
for the same reason as it did for programmable ps, it breaks
wine tests using XYZRHW. These tests show that when passing
pre-transformed vertices and an orthogonal projection matrix,
z_vs is used, and due to the XYZRHW property, this is not
recovered by the z_ps/w_ps computation (instead z_ps=z_vs).
For the game affected by the issue, the projection matrix set
is not orthogonal.
The direct3D spec indicates that the projection matrix must be
set correctly for fog to work properly, even if we do not use the
transformation pipeline (could be related to xyzrhw, or programmable vs
or both). Previous tests had shown that the projection matrix
has the last two values of the last column tested against 0 and 1,
in order to activate zfog or wfog.
The R500 spec indicates that either z or 1/1/w can be used as source
for the fog computation, but it is not clear whether this is z_vs or
z_ps.
Tests confirmed the intuition that the correct behaviour
is to use z_ps (zfog) when an orthogonal projection matrix is set
(the spec spirit being that in that case z_ps=z_vs),
and 1/w_ps (wfog) else (even if programmable shaders are used).
This patch introduces this behaviour.
Fixes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8341
Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22583>