panfrost: Increase blit shader BO size on Bifrost

Otherwise we hit 'offset + program->compiled.size < total_size' assert.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

pan/bi: LOD is a 8.8 fixed point

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

pan/bi: Always emit a LOD/CUBE word for FETCH instructions

There's no flag/mode to reflect when a LOD is zero on FETCH instructions,
we have to emit the LOD/CUBE word unconditionally.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

pan/bi: Only update LOD mode on TEX operations

If we don't add this check we clobber fetch mode when a 0 LOD is
specified on a txf instruction.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

panfrost: Set sample_count when packing bifrost texture descriptors

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

panfrost: Set depth for 3D textures on Bifrost

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

panfrost: Fix decoding of texture payloads

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

panfrost: Get rid of the Sample Count enum

Sample count just needs a log2 modifier. While at it, rename the
"Multisample count" field "Sample count" to be consistent with
other descriptors.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

panfrost: Stop forcing depth to nr_samples

Those are two different things, and the fact that Midgard use the same
offset in the texture descriptor for both things is just an
implementation detail.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

panfrost: Fix RAW8/16/32 component replication

Fixes dEQP-GLES3.functional.fbo.msaa.4_samples.r32f

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

panfrost: Account for sample count in tib offsets

I don't know if we have tests for MRT + MSAA but that would hit this.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

nir: Fix LOD source type for txf_ms instructions

txf_ms takes an integer LOD, not a float.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7653>

aco/ngg: fix division-by-zero in assertion

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7576>

aco: fix GS with no outputs

With NGG, ngg_gs_known_vtxcnt[0] would be false and ngg_gs_finale() would
assert.

With legacy GS, I don't know why the assertion was there.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7576>

radv/llvm,aco/ngg: fix large shift exponent in ngg_gs_vertex_lds_addr

When vertices_out=0, we will try to shift 1u by UINT32_MAX.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7576>

pan/bi: Implement shader-db stats

v2: Drop register tracking since it was wrong, and meaningful accounting
is tricky for Bifrost (which wants round robin RA for at least some
registers)... we'll cross that bridge when we get there, possibly
preferring a "max liveness" estimate to the raw RA output.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

pan/bi: Ensure TEXC src0 is not marked SSA

Prevents attempts to spill tied TEXC src/dest, resulting in a crash,
seen in shaders/tesseract/229.shader_test

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

pan/bi: Emit a combine even if we only pass one staging reg to TEXC

We need that to account for potential swizzling on the source reg.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

pan/bi: Fix off-by-one in RA

Could result in trying to allocate R64 which is clearly wrong.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

pan/bi: Fix varying writemask handling

Allows shaders/supertuxkart/1.shader_test and
shaders/unity/24-Tree.shader_test to compile.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

pan/bi: Implement sampler1D

Enough to get shaders/humus-celshading/1.shader_test compiling, no idea
if it actually works but it looks reasonable.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

pan/mdg: Fix shader-db counter

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

panfrost: Enable indirect uniform indexing

It works fine, and without it, the lowering bloats GLES2 shaders (by a
factor of 20x for one shader on t-rex!)

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

panfrost: Add PAN_GPU_ID debug option

I'd like to run shader-db locally without turning on the Bifrost
machine. I'm lazy, it's just for debug builds, ok? 😇

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7615>

nir/unsigned_upper_bound: decrement num_sources_left before recursing

Otherwise, search_phi_bcsel() will be called with a buf_size that is
slightly lower than it has to be.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7748>

nir/unsigned_upper_bound: fix buffer overflow in search_phi_bcsel

It should only recurse if there's enough space to add the phi sources.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Fixes: 72ac3f60261 ("nir: add nir_unsigned_upper_bound and nir_addition_might_overflow")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7748>

aco: fix v_mul_hi_u32_u24 format

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Fixes: 57c152af9ce ("aco: select v_mul_{hi}_u32_u24 for 24-bit multiplications")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3874
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7759>

Revert "util: Add helpers for various one-time-init patters"

This reverts commit bda4d6e0d01116db59a0a03b0c703a7af6e11949.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "nir: Use get_once() helper for one-time init's"

This reverts commit c9062df1d57df19a56288c1749d4b6d22d7c1418.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "freedreno/ir3: Use get_once() for one-time init"

This reverts commit b4ad27a986e1c6899cbf23355c3e9c6de345a323.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "gallium/hud: Use do_once for one-time init"

This reverts commit 2e81ec5e009e3fbeef3fe1a76f2dfee428b7c160.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "mesa/st: Use do_once for one-time init"

This reverts commit bcb2981e145f0c2a54a637e20b6c55eaf316c04f.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "util: Fix helgrind complaint about one-time init"

This reverts commit f8c7a43f33d4647c16c4892d56706a14e5d6bf17.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "mesa: Fix helgrind complaint about one-time init"

This reverts commit f7102ac376a23a394786085a2b0dffa94c13150c.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "gallium/trace: Fix helgrind complaint about one-time init"

This reverts commit 1aa055539f32fcb3c78d908d1635bb3a3d517bc2.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

Revert "tgsi: Fix helgrind complaint about one-time init"

This reverts commit d91fe7d1c65179e64a6ca294135ac1bad2b16fb9.

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7760>

radv: always use 32-bit predication on compute queues

It seems that only gfx queue doesn't support it, except on GFX10.3
which supports all queues.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7732>

radv: use 32-bit predication for conditional rendering on GFX10.3+

It's now supported.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7732>

ac: add gpu_info::has_32bit_predication

32-bit predication is now supported with GFX10.3.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7732>

v3dv: Wayland WSI support

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7303>

intel/rt: Implement push constants as global memory reads

They're not really "push" anymore but that's because there is no such
thing as push constants in bindless shaders on Intel.  They should be
fast enough, though.  There is some room for debate here as to whether
we want to do the pull in NIR or push it into the back-end.  The
advantage of doing it in the back-end is that it'd be easier to use
MOV_INDIRECT for indirect push constant access rather than falling back
to a dataport message.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add support for hit attributes

For triangle geometry, the hit attributes are always two floats which
contain the barycentric coordinates of the hit.  For procedural
geometry, they're an arbitrary blob of data passed from the intersection
shader to the hit shaders.  In our implementation, we stash that data
right after the HW RayQuery in the ray stack.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a helper to create the raygen trampoline shader

Unlike graphics and compute pipelines, Vulkan ray-tracing pipelines do
not have a single entrypoint.  Instead, the raygen shader is specified
as a one-element shader binding table in the vkCmdTraceRay call.  This
means that raygen shaders have to be bindless shaders just like any
other ray tracing shader.  To launch them, we have a tiny compute shader
that acts as a trampoline and sets up the hotzone and uses btd_spawn to
fire off the raygen shader.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add lowering for combined intersection/any-hit shaders

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add lowering for ray-walk intrinsics in any-hit shaders

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add support for shader buffer record memory

Most of the work for this is done for us by spirv_to_nir which gives us
a load_global from a memory address based on the shader_record_ptr
system values.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Implement the new ray-tracing system values

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Implement traceRay()

This is a little bit more work than executeCallable() because we also
have to set up the MemRay data structure which the ray traversal
hardware uses to keep its state.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/fs: Add and implement intel-specific ray-tracing intrinsics

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Implement support for shader call payloads

Both traceRay() and executeCallable() take a payload parameter which
gets passed from the caller to the callee and which the callee can write
to pass data back to the caller.  We implement these by passing a
pointer to the data structure in the callee to the caller as the second
QWord on its stack.  Coming out of spirv_to_nir, the incoming call
payloads get the nir_var_shader_call_data variable mode allowing us to
easily identify them.  Outgoing call payloads get assigned the
nir_var_shader_temp mode and will have been turned into function_temp by
nir_lower_global_vars_to_local.  All we have to do is crawl the shader
looking for references to the nir_var_shader_call_data variable and
rewrite those to use the passed in pointer.  nir_lower_explicit_io will
do the rest for us.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a helper to create a trivial return shader

These are required for ray-tracing.  There are many cases where the
ray-tracing hardware may decide to execute some but not all of our
shaders.  In these cases, it needs a shader to execute at the end which
will pop the stack back to the shader which called traceRay().

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a pass to lower shader call instructions

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add return instructions at the end of ray-tracing shaders

Each callable ray-tracing shader shader stage has to perform a return
operation at the end.  In the case of raygen shaders, it retires the
bindless thread because the raygen shader is always the root of the call
tree.  In the case of any-hit shaders, the default action is accep the
hit.  For callable, miss, and closest-hit shaders, it does a return
operation.  The assumption is that the calling shader has placed a
BINDLESS_SHADER_RECORD address for the return in the first QWord of the
callee's scratch space.  The return operation simply loads this value
and calls a btd_spawn intrinsic to jump to it.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add support for scratch in ray-tracing shaders

In ray-tracing shader stages, we have a real call stack and so we can't
use the normal scratch mechanism.  Instead, the invocation's stack lives
in a memory region of the RT scratch buffer that sits after the HW ray
stacks.  We handle this by asking nir_lower_io to lower local variables
to 64-bit global memory access.  Unlike nir_lower_io for 32-bit offset
scratch, when 64-bit global access is requested, nir_lower_io generates
an address calculation which starts from a load_scratch_base_ptr.  We
then lower this intrinsic to the appropriate address calculation in
brw_nir_lower_rt_intrinsics.

When a COMPUTE_WALKER command is sent to the hardware with the BTD Mode
bit set to true, the hardware generates a set of stack IDs, one for each
invocation.  These then get passed along from one shader invocation to
the next as we trace the ray.  We can use those stack IDs to figure out
which stack our invocation needs to access.  Because we may not be the
first shader in the stack, there's a per-stack offset that gets stored
in the "hotzone".

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add lowering functions for each ray-tracing stage

These will eventually contain per-stage lowering for various ray-tracing
things.  This is separate from brw_nir_lower_rt_intrinsics because, for
reasons that will become apparent later, brw_nir_lower_rt_intrinsics has
to be run very late in the compile process, right before brw_compile_bs.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a pass to lower the new ray-tracing intrinsics

The new intrinsics we added for doing address calculations are all
things we fetch from the RT_DISPATCH_GLOBALS struct.  We could emit an
RT_DISPATCH_GLOBALS load at every point we want it and trust NIR to CSE
it for us but it's easier to use intermediate intrinsics.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add builder helpers for accessing RT data structures

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/fs: Add and implement a load_global_const_block intrinsic

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a brw_rt.h header with #defines for basic RT data structures

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/compiler: Add support for bindless shaders

The Intel bindless thread dispatch model is very simple.  When a compute
shader is to be used for bindless dispatch, it can request a set of
stack IDs.  These are allocated per-dual-subslice by the hardware and
recycled automatically when the stack ID is returned.  Passed to the
bindless dispatch are a global argument address, a stack ID, and an
address of the BINDLESS_SHADER_RECORD to invoke.  When the bindless
shader is dispatched, it is passed its stack ID as well as the global
and local argument pointers.  The local argument pointer is the address
of the BINDLESS_SHADER_RECORD plus some offset which is specified as
part of the BINDLESS_SHADER_RECORD.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/debug: Add a debug flag for ray-tracing shaders

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir/lower_io: Support shader_call_data in vars_to_explicit_types

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir/lower_io: Allow ray_hit_attrib in lower_vars_to_explicit_types

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir: Add a helper to get the live set at a cursor

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Add BVH data structures

These describe the Intel BVH format used for storing acceleration
structures.

Acked-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Add RT_DISPATCH_GLOBALS and RT_*_SBT_HANDLE structs

The RT_DISPATCH_GLOBALS struct is half HW-defined by the ray-tracing
spec and half SW-defined.  However, due to the addresses in it, it's
convenient for it to all be in GenXML.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Support truncated addresses

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml/pack: Stash the cloned address field

The cloned version is the one that has updated start and end bits
fields.  We're about to start passing those through to a new
__gen_address function and we need the correct start/end in order to do
that reliably.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Add the BINDLESS_SHADER_RECORD data structure

This is the first of the HW data structures added for ray-tracing.
These are added to their own file because it's not really associated
with any hardware we've enabled in Mesa just yet.  Eventually, these
will likely get folded into the appropriate genX.xml file as they are
hardware data structures and needed to be tracked as such.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/dev: Add a gen_device_info::has_ray_tracing bit

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

spirv: Emit nir_jump_halt after TerminateRay or IgnoreIntersection

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir: Add a halt instruction type

Halt is like a return for the entire shader or exit() if you prefer to
think of it that way.  Once an invocation hits a halt, it's 100% dead.
Any writes to output variables which happened before the halt do,
however, still apply.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

meson: add idep_mesautil to components using simple_mtx.h

If valgrind is installed, these components need to find valgrind.h.

Fixes: 53f7d539cd9 ("util: Add helgrind support for simple_mtx")
Closes: #3876
Acked-by: Rob Clark <robclark@freedesktop.org>

Revert "zink: initial implementation of shader keys"

This reverts commit 2be2a500a396fe1dc0e121816e4056874cdd43fc.

Fixes: 2be2a500a39 ("zink: initial implementation of shader keys")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: refcount the shader cache"

This reverts commit b9fdc21bba0724271520462f3f04ba72ae106a26.

Fixes: b9fdc21bba0 ("zink: refcount the shader cache")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: move shader key structs into their own header"

This reverts commit ed7a5a55689ddcf10aae3f671e390252aad2d62f.

Fixes: ed7a5a55689 ("zink: move shader key structs into their own header")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: fill in params for fs shader keys and flag shader for rebuild"

This reverts commit 9aa08221fa6653002dec2a07641d58d3c7fa227a.

Fixes: 9aa08221fa6 ("zink: fill in params for fs shader keys and flag shader for rebuild")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: put those shader keys to work fixing up fragment shaders"

This reverts commit eeff625ab31f0a1bf94739e09ca6b08de00c94f5.

Fixes: eeff625ab31 ("zink: put those shader keys to work fixing up fragment shaders")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: update shader modules in gfx program when flagged dirty"

This reverts commit e96afeeb7bb2306aa0fba76e27163d4c7a86860b.

Fixes: e96afeeb7bb ("zink: update shader modules in gfx program when flagged dirty")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

radv: Fix RB+ blending for VK_FORMAT_E5B9G9R9_UFLOAT_PACK32.

Fixes: e893102bcf5 ("radv: Add VK_FORMAT_E5B9G9R9_UFLOAT_PACK32 rendering support.")
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7716>

mesa: Synchronize get_gl_override()

Fixes helgrind complaint found with piglit glx-multithread-clearbuffer.
This is a legit race because override[api].version is cleared before
parsing the override string.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

tgsi: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

gallium/trace: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

mesa: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

util: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

mesa/st: Use do_once for one-time init

Also, since there is a second call-path into st_init_extensions() from
get_version(), add an extra st_debug_init() call.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

gallium/hud: Use do_once for one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

freedreno/ir3: Use get_once() for one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

nir: Use get_once() helper for one-time init's

Makes the code more concise, and makes helgrind/drd happy at the same
time!

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

util: Add helpers for various one-time-init patters

A fairly common pattern for debug envvars is something like:

   static int should_print = -1;
   if (should_print < 0)
      should_print = env_var_as_unsigned("NIR_PRINT", 0);

Unfortunately helgrind doesn't realize that we expect to always get the
same return value, so we don't actually care about the race condition
here.

Add a helper get_once() and do_once macros, with extra locking to make
helgrind/drd happy.  Note that other than the nir usages (which are
limited to debug builds), other usages are not in hot-paths.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

util: Add helgrind support for simple_mtx

Annoyingly mtypes.h pulls in simple_mtx, which means we end up needing
to sprinkle a lot of idep_mesautil around.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3773
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

spirv: Implement SpvOpConvertUToAccelerationStructureKHR

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Implement OpTerminateRayKHR and OpIgnoreIntersectionKHR

In the final version of SPV_KHR_ray_tracing, these are now block
terminators like OpKill or OpReturn.  This means that they need special
handling in vtn_cfg.c.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Call repair SSA for OpTerminateInvocation

Fixes: 886d2d1a9abcb "spirv: Handle SpvOpTerminateInvocation"

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Implement OpTraceRayKHR and OpExecuteCallableKHR

The old NV version (and the provisional KHR version) specified the data
payload via an integer location.  This was quite annoying for the parser
and potentially error-prone.  The final KHR version of the SPIR-V
ray-tracing spec replaces these integers with actual pointers.  We don't
really need to implement the NV versions but we have the code and
someone might want to parse some NV ray-tracing shaders.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Update JSON and headers from Khronos main

This corresponds to 104ecc356c1bea4476320faca64440cd1df655a3 ("Merge
pull request #180 from dneto0/issue-179") in
https://github.com/KhronosGroup/SPIRV-Headers

Acked-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Rename some ray-tracing intrinsics to NV

For these intrinsics, the NV version and the provisional KHR version
have the same enum value and semantics but the final KHR version is
different on both counts.  Re-name them to NV before we update the
header so the header update isn't a functional change.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

vulkan: Update XML and headers to 1.2.162

This brings in the following new extensions:

 - VK_KHR_deferred_host_operations
 - VK_KHR_pipeline_library
 - VK_KHR_acceleration_structure
 - VK_KHR_ray_tracing_pipeline
 - VK_KHR_ray_query

Acked-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Acked-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

zink: fall back to util_blitter for scaled resolves

Vulkan can't scale while resolving using vkCmdResolveImage. For this we
need to use util_blitter.

The reason this wasn't a problem in the past, was that glBlitFramebuffer
always set pipe_blit_info::render_condition_enable, and we always used
that to bail out to util_blitter. When the latter changed, this broke.

Fixes: 19906022e22 ("zink: more accurately track supported blits")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7745>

nir/opt_peephole_select: respect selection_control when collapsing ifs

Totals from 34 (0.02% of 138013) affected shaders (RAVEN):
CodeSize: 625888 -> 626336 (+0.07%); split: -0.00%, +0.08%
Instrs: 124121 -> 124229 (+0.09%); split: -0.00%, +0.09%
Cycles: 1403072 -> 1403588 (+0.04%); split: -0.01%, +0.04%
VMEM: 5308 -> 5364 (+1.06%); split: +1.07%, -0.02%
Copies: 12773 -> 12838 (+0.51%); split: -0.08%, +0.59%
Branches: 5758 -> 5801 (+0.75%); split: -0.21%, +0.96%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7478>