gallivm: fixes for LLVM-15 opaque pointers in lp_bld_nir_soa.c

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18334>

gallivm: fixes for LLVM-15 opaque pointers in lp_bld_nir_aos.c

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18334>

gallium: drop unused macros in draw_llvm.h

The reason is that these macros use lp_build_struct_get(), which is
being replaced by lp_build_structure_get2() which takes the explicit
pointer type. Rather than attempt to update these unused macros, simply drop
them.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18334>

gallivm: drop unused "emit_load" in lp_llvm_buffer_member()

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18334>

gallivm: LLVM-15 opaque pointers: disable LLVMGetElementType(ptr_type)

with opaque pointers, we can't query the element type of a pointer type

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18334>

llvmpipe: fix front/back face system value in the fs

draw needs to inject the face information for tris in line/point fill
mode since llvmpipe only sees lines and points. But for the new style
face information (which is a system value and not an ordinary input
register) we forgot to tell draw.

Reviewed-by: Brian Paul <brianp@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18285>

vulkan: fixup 1.3.226 update

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Acked-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18397>

anv: remove non present field in upstream spec

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Acked-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18397>

Enable .VALVE_mutable_descriptor_type in physical_device

Signed-off-by: Dawn Han <dawnhan@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17505>

Track bitset when create descriptor pool

Signed-off-by: Dawn Han <dawnhan@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17505>

Update bitset when init descriptor layout.

Signed-off-by: Dawn Han <dawnhan@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17505>

Increase enum type in vn_descriptor_set.

Signed-off-by: Dawn Han <dawnhan@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17505>

ci: uprev piglit 2022-08-30

Acked-by: Juan A. Suarez <jasuarez@igalia.com> # for broadcom
Acked-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com> # for zink
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18318>

util/perf: use ALWAYS_INLINE for tracepoints

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18309>

util/perf: simplify u_trace_instrument

When ut_perfetto_enabled changes, update _u_trace_instrument as well.
This allows u_trace_instrument to check just a single variable.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18309>

util/perf: add u_trace_instrument

It is called from tracepoints.  When it returns false, instrumentation
is disabled.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18309>

util/perf: remove u_trace::enabled

We can set ut_trace_instrument when either GPU_TRACE or
GPU_TRACE_INSTRUMENT is set.  This results in one less variable to check
in tracepoints.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18309>

util/perf: ignore ut->enabled for iterators

With perfetto, instrumentation can be enabled in the middle of cmd
buffer recording even when ut->enabled is false.  It might be better to
be consistent and ignore ut->enabled for iterators.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18309>

mesa: add GLSL_SOURCE

many times I just want to dump glsl and not a trillion lines of IR

Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18280>

anv: enable EXT_mesh_shader

Acked-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18371>

anv: implement draw calls for EXT_mesh_shader

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18371>

anv: implement EXT_mesh_shader

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18371>

anv: replace VK_SHADER_STAGE_[TASK|MESH]_BIT_NV with VK_SHADER_STAGE_[TASK|MESH]_BIT_EXT

They have the same numerical values, so nothing changes.

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18371>

anv: check EXT_mesh_shader whenever NV_mesh_shader is checked

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18371>

intel/compiler: implement EXT_mesh_shader

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18371>

turnip: Treating non-d/s-write pipelines as not having d/s feedback loops.

A subpass in gfxbench has the depth buffer present, but not written to,
for a render pass using the depth buffer as an input attachment.  We can
skip single-prim-mode and the associated "oh no don't use sysmem" in that
case.

Improves gfxbench vk-5-normal perf by 1.56193% +/- 0.0743035% (n=14).
Part of #6327.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18241>

spirv, compiler: add "bool nv" to shader_info.mesh

Not knowing whether we deal with the NV or EXT extension
makes implementation difficult for Intel HW.
NV support will be dropped at some point, so
this ugliness will go away eventually.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

spirv, nir: Handle EmitMeshTasksEXT opcode.

A task shader must use this instruction to specify the dimensions
of the launched mesh shader workgroups.
It is a terminating instruction.

When the task shader doesn't have the optional payload, use the
pre-existing launch_mesh_workgroups intrinsics.

When the task shader has a payload, use a new
launch_mesh_workgroups_with_payload_deref intrinsics which has
a deref that refers to the payload variable.

We also add this new intrinsic to nir_lower_io which lowers this
to the pre-existing explicit intrinsic.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

spirv: Support TaskPayloadWorkgroupEXT storage class.

Just use the task_payload NIR storage class for this.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

spirv: Support the CullPrimitiveEXT mesh shader built-in.

This is a per-primitive builtin output which indicates that a
primitive should be culled (deleted) from the output.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

spirv: Support EXT_mesh_shader SetMeshOutputsEXT.

Use the set_vertex_and_primitive_count intrinsic to
express the number of vertices and primitives that the
mesh shader workgroup outputs.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

spirv: Support EXT_mesh_shader mesh/task stages.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

spirv: Add mesh_shading capability for EXT_mesh_shader.

Indicates support for the EXT_mesh_shader SPIR-V capabilities.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

spirv: Support EXT_mesh_shader indices and mark them per-primitive.

They are not defined as per-primitive in the EXT, but they behave
like per-primitive outputs so it's easier to treat them like that.
They may still require special treatment in the backend in order to
control where and how they are stored.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

vulkan, spirv: Update to Vulkan 1.3.226 and latest SPIR-V headers.

Done using the "khronos-update.py" script, leaving out parts that
are not relevant to Vulkan.

Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18366>

pan/bi: Add and use bi_num_successors helper

Makes a few patterns easier to read.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add and use bi_replace_src helper

This is a common pattern. Make it more ergonomic. With the help of Coccinelle:

   @@
   expression E;
   expression R;
   expression instruction;
   @@

   -instruction->src[E] = bi_replace_index(instruction->src[E], R);
   +bi_replace_src(instruction, E, R);

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Remove assert(bi_is_ssa(dest))

Prior to register allocation, destinations must be in SSA form, except for the
"fake SSA" briefly used in the current RA (for which bi_is_ssa returns true
anyway). These asserts duplicate the asserts in the validator. If there's any
coverage lost here, that's just a sign the validator needs to run more often.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Strengenth assert in the validator

We should only see SSA now.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add and use bi_foreach_ssa_src macro

Frequently, we want to iterate over the SSA variables read by an instruction,
while skipping over constants and uniforms. Add and use a macro for this
pattern. A few redundant asserts are removed.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Simplify bi_get_index prototype

2/3 of its arguments are now unused.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Remove bi_max_temp

This is no longer needed given the flat indexing.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Inline node_to_index

One user, now trivial.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Get rid of bi_get_node

Now that variables have flat naming (no more SSA/reg distinction), we can just
use .value directly. This cleans up the RA a bit.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Remove NIR registers from the IR

They are now unused, so normal == SSA at this point.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Move non-SSA liveness into RA

Now that non-SSA nodes are an internal implementation detail of RA, the
entrypoints should be made private to make sure no other passes use the
RA-specific liveness analysis.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume SSA for helper invocation analysis

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume SSA when translating NIR

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume SSA when scheduling for pressure

This is much simpler now: entire hazards become impossible when the program is
kept in SSA form.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume SSA in minor passes

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume non-null sources in constant folding

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume SSA in CSE

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume SSA in mod prop

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume SSA in copyprop

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't use bi_temp_reg

We've already broken SSA irreparably at this point. We can just change what
"normal" means and avoid the disjointed indexing.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Clear reg in squeeze_index

SSA is broken at this point. Let's not pretend otherwise. The numbering produced
by squeeze_index already disambiguates what-used-to-be SSA variables and
registers. A few shader-db changes because this shuffles the register allocation
around because LCRA is stupid.

total instructions in shared programs: 2658849 -> 2650196 (-0.33%)
instructions in affected programs: 463733 -> 455080 (-1.87%)
helped: 1089
HURT: 135
helped stats (abs) min: 1.0 max: 43.0 x̄: 8.20 x̃: 5
helped stats (rel) min: 0.11% max: 17.05% x̄: 2.28% x̃: 1.52%
HURT stats (abs)   min: 1.0 max: 7.0 x̄: 2.07 x̃: 2
HURT stats (rel)   min: 0.10% max: 2.97% x̄: 1.03% x̃: 0.83%
95% mean confidence interval for instructions value: -7.52 -6.62
95% mean confidence interval for instructions %-change: -2.04% -1.78%
Instructions are helped.

total cycles in shared programs: 140617.52 -> 140613.14 (<.01%)
cycles in affected programs: 227.45 -> 223.08 (-1.92%)
helped: 45
HURT: 1
helped stats (abs) min: 0.015625 max: 0.4375 x̄: 0.10 x̃: 0
helped stats (rel) min: 0.43% max: 17.72% x̄: 2.79% x̃: 1.29%
HURT stats (abs)   min: 0.015625 max: 0.015625 x̄: 0.02 x̃: 0
HURT stats (rel)   min: 1.47% max: 1.47% x̄: 1.47% x̃: 1.47%
95% mean confidence interval for cycles value: -0.12 -0.07
95% mean confidence interval for cycles %-change: -3.76% -1.64%
Cycles are helped.

total cvt in shared programs: 13877.61 -> 13742.41 (-0.97%)
cvt in affected programs: 3432 -> 3296.80 (-3.94%)
helped: 1089
HURT: 135
helped stats (abs) min: 0.015625 max: 0.671875 x̄: 0.13 x̃: 0
helped stats (rel) min: 0.18% max: 28.36% x̄: 5.05% x̃: 3.38%
HURT stats (abs)   min: 0.015625 max: 0.109375 x̄: 0.03 x̃: 0
HURT stats (rel)   min: 0.15% max: 8.33% x̄: 2.87% x̃: 2.27%
95% mean confidence interval for cvt value: -0.12 -0.10
95% mean confidence interval for cvt %-change: -4.46% -3.88%
Cvt are helped.

total quadwords in shared programs: 1442328 -> 1437776 (-0.32%)
quadwords in affected programs: 106240 -> 101688 (-4.28%)
helped: 478
HURT: 17
helped stats (abs) min: 8.0 max: 24.0 x̄: 9.81 x̃: 8
helped stats (rel) min: 1.54% max: 20.00% x̄: 5.27% x̃: 3.70%
HURT stats (abs)   min: 8.0 max: 8.0 x̄: 8.00 x̃: 8
HURT stats (rel)   min: 1.61% max: 14.29% x̄: 6.42% x̃: 6.25%
95% mean confidence interval for quadwords value: -9.61 -8.78
95% mean confidence interval for quadwords %-change: -5.20% -4.54%
Quadwords are helped.

total threads in shared programs: 53557 -> 53550 (-0.01%)
threads in affected programs: 14 -> 7 (-50.00%)
helped: 0
HURT: 7
HURT stats (abs)   min: 1.0 max: 1.0 x̄: 1.00 x̃: 1
HURT stats (rel)   min: 50.00% max: 50.00% x̄: 50.00% x̃: 50.00%
95% mean confidence interval for threads value: -1.00 -1.00
95% mean confidence interval for threads %-change: -50.00% -50.00%
Threads are HURT.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Clean up after converting to SSA

nir_invalidate_divergence was introduced in aa765c54bdd ("pan/bi: Add divergent
intrinsic lowering pass"), which needed divergence information for a late NIR
pass but not otherwise in the backend. Unfortunately, nir_convert_from_ssa is
less aggressive about coalescing when divergence information makes it look like
the coalescing would make the code worse -- even though that's not actually an
issue on Mali.

Now that we don't use nir_convert_from_ssa, we don't need the hack.

Likewise for the vec.reg hack, which was introduced because the split/collect
cache relies on SSA form.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Preserve SSA form from NIR

Don't call nir_convert_from_ssa, preserve the SSA from NIR. This gets us real
SSA for all frontend passes. The RA becomes a black box that takes SSA in and
outputs register allocated code out. The "broken" SSA form never escapes RA,
which means we can clean up special cases in the rest of the compiler. It also
gets us ready for exploiting SSA form in the RA itself.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Reduce some moves when going out-of-SSA

This is still pretty brainless.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add brainless out-of-SSA pass

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add SSA-based liveness pass

Adapted from NIR's liveness analysis. This is different from our non-SSA
liveness pass for a few reasons:

1. It must handle phi nodes. This implies significant changes to the worklist
   algorithm.
2. It only handles SSA. It doesn't need funny labelling schemes for
   handling nir_registers in parallel with SSA defs.
3. It is scalar-only. The vector liveness information isn't interesting when
   vectors are handled via COLLECT and SPLIT. This means it uses a bitset (uses
   8x less memory to store livenss information, should be easier on the caches
   too).

Eventually, this will become our only pre-RA liveness pass. For now, both passes
are maintained in parallel: the SSA pass used before out-of-SSA, the non-SSA
pass used after out-of-SSA and before RA, and the post-RA pass used after RA.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Translate phis from NIR

Code lifted from AGX, which in turn is based on ir3 and aco.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Inline DCE into bi_lower_vector

This needs on after going out-of-SSA, which is a separate DCE algorithm. Split
out.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Validate before&after pre-RA sched

Pre-RA scheduling can move instructions around in nontrivial ways, yet it must
maintain the IR's ordering invariants (for preloads, phis, etc.) Running
validation before and after makes scheduler bugs more obvious.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't reorder phis in pre-RA scheduling

Treat them like preloads which are also required to be at the start of the
block.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't optimize if without else

We need to emit a jump over the else body when constructing the IR, even if the
else body appears empty in NIR, because instructions could be added when
going out-of-SSA. This optimization should instead happen post-RA, but doing so
is nontrivial and the cycle count regression is minimal, so punting on this for
now.

total instructions in shared programs: 2728364 -> 2731495 (0.11%)
instructions in affected programs: 580462 -> 583593 (0.54%)
helped: 0
HURT: 1272
HURT stats (abs)   min: 1.0 max: 17.0 x̄: 2.46 x̃: 2
HURT stats (rel)   min: 0.09% max: 22.22% x̄: 0.88% x̃: 0.59%
95% mean confidence interval for instructions value: 2.37 2.56
95% mean confidence interval for instructions %-change: 0.80% 0.97%
Instructions are HURT.

total cycles in shared programs: 140628.52 -> 140631.50 (<.01%)
cycles in affected programs: 474.45 -> 477.44 (0.63%)
helped: 0
HURT: 116
HURT stats (abs)   min: 0.015625 max: 0.0625 x̄: 0.03 x̃: 0
HURT stats (rel)   min: 0.21% max: 9.09% x̄: 1.16% x̃: 0.39%
95% mean confidence interval for cycles value: 0.02 0.03
95% mean confidence interval for cycles %-change: 0.88% 1.43%
Cycles are HURT.

total cvt in shared programs: 14816.20 -> 14865.12 (0.33%)
cvt in affected programs: 4954.31 -> 5003.23 (0.99%)
helped: 0
HURT: 1272
HURT stats (abs)   min: 0.015625 max: 0.265625 x̄: 0.04 x̃: 0
HURT stats (rel)   min: 0.16% max: 28.57% x̄: 1.88% x̃: 1.17%
95% mean confidence interval for cvt value: 0.04 0.04
95% mean confidence interval for cvt %-change: 1.74% 2.02%
Cvt are HURT.

total quadwords in shared programs: 1476536 -> 1478512 (0.13%)
quadwords in affected programs: 54496 -> 56472 (3.63%)
helped: 0
HURT: 247
HURT stats (abs)   min: 8.0 max: 8.0 x̄: 8.00 x̃: 8
HURT stats (rel)   min: 0.93% max: 20.00% x̄: 5.78% x̃: 4.17%
95% mean confidence interval for quadwords value: 8.00 8.00
95% mean confidence interval for quadwords %-change: 5.21% 6.34%
Quadwords are HURT.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Remove bogus assert lowering branches

We can get BRANCHZ.i16 since we lower JUMP early. This seems to have worked
before mainly by chance. With the change to how we optimize, we can get code
sequences like:

   block2 {
      BRANCHZ.i16.eq u256, u256 -> block5
      BRANCHZ.i16.eq u256, u256 -> block4
   } -> block5  from block1

which would choke the assert.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't lower FAU for phis

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Validate phi ordering

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add phi nodes

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Lift bi_predecessor_index helper

From AGX for handling phi node.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add bi_after_block_logical helper

Useful for lowering phis.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't write registers in optimizer tests

The new SSA-based dead code elimination won't know how to deal with code of that
funny form. In actuality it doesn't have to, so we just make sure the optimizer
tests produce valid IR so this works as expected.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Dynamically allocate source/dests

Instead store a pointer to the storage. Ideally even these pointer would be
removed but that's quite cumbersome in C...

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Reduce memory of scheduler access array

Now that we have accurate source/dest counts, we don't need to worst case. Which
is good, because the worst case will become infinite once we allow phis.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Use builder for atomic lowering in scheduler

The last places where sources are appended. Fix them and we can move on.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Use builder for IADD -> IADDC opt

As before.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Use builder for MUX -> CSEL opt

This is yet another case where we add a source, which will require reallocation.
It's easy enough to rebuild the instruction here.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/va: Use builder in va_lower_isel

..where necessary to avoid appending sources. This removes a bunch more cases
where we would need to reallocate sources. The lifetime management is simpler if
we just reallocate the whole instruction. In theory this is a slight overhead
but it's probably worth it to avoid the complexity otherwise.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Rebuild DISCARD when optimizing

This avoids incrementing I->nr_srcs which will soon be invalid.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Rebuild some instructions when lowering

When we add a source, we need to build a new instruction (or at the very least
reallocate sources). This is less of a hack anyway.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Use safe helpers to remove srcs/dests

Changing I->nr_srcs or I->nr_dests directly is generally unsafe, but the special
case of removing sources/destinations at the end is safe. Add and use helpers to
wrap this operation simplifying the remaining code audit before we can
dynamically allocate sources/destinations.

At this point in the series, nothing modifies I->nr_dests after allocation
except these helpers, so destinations should be safe to make dynamic. There's a
bit more work needed for sources.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Use variable src/dest for collect/split

This avoids the nr_srcs/nr_dests setting dance, and will allow the builder to
handle the required memory allocation when we switch to dynamic src/dest
allocation.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add variable dest/src support to builder

This will allow pseudo instructions to be allocated in a cleaner way when we
dynamically allocate their sources/destinationa.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't read nonexistant sources

Codebase audit. In preparation to dynamically allocate sources.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't read nonexistant destinations

Codebase audit. In preparation to dynamically allocate destinations.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assume destinations are non-NULL

We guarantee this now, no need to check it in every pass. There's an exception
on Bifrost after clause scheduling, but few passes run after clause scheduling
so this doesn't affect much.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Assert destinations are non-NULL

All uses of null destinations have been replaced with more appropriate
constructs. We can now assume this throughout.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Introduce TEXC_DUAL psuedoinstruction

There are two "shapes" of TEXC in the IR:

* Regular texturing. This TEXC writes a single set of staging registers.
* Dual texturing. This TEXC writes two sets of staging registers.

Currently we model both with a 2-destination TEXC, with a null second
destination for the usual case where dual texturing isn't used. This is awkward.
To make the "shapes" of instructions more predictable, make TEXC only write a
single set of staging registers (like the hardware instruction) and split off a
TEXC_DUAL pseudoinstruction for the second case, lowered late.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Use bi_emit_split_i32 for trimming vecs

This ensures the shape of the split is correct (no nulls in the destination).
This should not affect the register allocation, though it does "cost" extra
variable names.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Don't remove dests in DCE

Removing dests without removing instructions only makes sense for certain
pseudo-instructions, but it makes the IR needlessly complicated for all
instructions. There's no real reason to do so, we can signal this in a different
way instead.

No shader-db changes.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Register allocate BLEND dest on Valhall

On Bifrost, BLEND writes to the link register, acting like a function call. On
Valhall, BLEND does not write anything. But the BLEND instruction in our IR is a
pseudo-instruction with Bifrost semantics, expanding to a multi-instruction
sequence on Valhall. So it's not worth special casing Valhall in instruction
selection.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Improve register printing

Use "r0" style to denote machine registers, consistent with the assembly. Use
instead "nr0" for NIR registers so they don't collide. This reduces noise for
the average case (either register allocated or almost SSA).

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Clean up destination printing

Now we can trust in I->nr_dests, avoiding a bunch of special cases and incorrect
printing for instructions without a destination. To compensate for those
instructions being hard to scan now, indent.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Remove trivial dest[0] = bi_null()

It's already NULL and will interfere with the more comprehensive src/dest
overwork we're doing.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Add validation for nr_srcs/nr_dests

Now that we set nr_srcs/nr_dests accurately, assert as much in the validator.
This pass will be deleted later in the series, but having it here is expected to
be useful for bisection, in case there are cases missed. Certainly running the
CTS at this point in the series is helpful to prove completeness of the
beginning of this series.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Use nr_{srcs,dests} in bi_foreach_{src,dest}

This avoids crawling sources that must be null.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Consider nr_srcs/dests in CSE

Now that these are meaningful for COLLECT and SPLIT.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Model 3rd source for ATEST

ATEST on Valhall takes atest-param as an explicit encoded source. The source
also exists on Bifrost, but it's implied in the encoding. Rather than treating
it as a "ghost" source, model it in the IR explicitly so the count is correct.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>

pan/bi: Set I->nr_dests, I->nr_srcs

The builder is the primary producer of instructions, and generally the shape of
an instruction is fixed at build-time. It must set nr_dests/nr_srcs
appropriately. Likewise, when we modify sources later, we need to update
nr_srcs/nr_dests to keep everything consistent (and keep the tests passing).

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17794>