intel/rt: Add lowering for ray-walk intrinsics in any-hit shaders

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add support for shader buffer record memory

Most of the work for this is done for us by spirv_to_nir which gives us
a load_global from a memory address based on the shader_record_ptr
system values.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Implement the new ray-tracing system values

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Implement traceRay()

This is a little bit more work than executeCallable() because we also
have to set up the MemRay data structure which the ray traversal
hardware uses to keep its state.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/fs: Add and implement intel-specific ray-tracing intrinsics

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Implement support for shader call payloads

Both traceRay() and executeCallable() take a payload parameter which
gets passed from the caller to the callee and which the callee can write
to pass data back to the caller.  We implement these by passing a
pointer to the data structure in the callee to the caller as the second
QWord on its stack.  Coming out of spirv_to_nir, the incoming call
payloads get the nir_var_shader_call_data variable mode allowing us to
easily identify them.  Outgoing call payloads get assigned the
nir_var_shader_temp mode and will have been turned into function_temp by
nir_lower_global_vars_to_local.  All we have to do is crawl the shader
looking for references to the nir_var_shader_call_data variable and
rewrite those to use the passed in pointer.  nir_lower_explicit_io will
do the rest for us.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a helper to create a trivial return shader

These are required for ray-tracing.  There are many cases where the
ray-tracing hardware may decide to execute some but not all of our
shaders.  In these cases, it needs a shader to execute at the end which
will pop the stack back to the shader which called traceRay().

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a pass to lower shader call instructions

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add return instructions at the end of ray-tracing shaders

Each callable ray-tracing shader shader stage has to perform a return
operation at the end.  In the case of raygen shaders, it retires the
bindless thread because the raygen shader is always the root of the call
tree.  In the case of any-hit shaders, the default action is accep the
hit.  For callable, miss, and closest-hit shaders, it does a return
operation.  The assumption is that the calling shader has placed a
BINDLESS_SHADER_RECORD address for the return in the first QWord of the
callee's scratch space.  The return operation simply loads this value
and calls a btd_spawn intrinsic to jump to it.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add support for scratch in ray-tracing shaders

In ray-tracing shader stages, we have a real call stack and so we can't
use the normal scratch mechanism.  Instead, the invocation's stack lives
in a memory region of the RT scratch buffer that sits after the HW ray
stacks.  We handle this by asking nir_lower_io to lower local variables
to 64-bit global memory access.  Unlike nir_lower_io for 32-bit offset
scratch, when 64-bit global access is requested, nir_lower_io generates
an address calculation which starts from a load_scratch_base_ptr.  We
then lower this intrinsic to the appropriate address calculation in
brw_nir_lower_rt_intrinsics.

When a COMPUTE_WALKER command is sent to the hardware with the BTD Mode
bit set to true, the hardware generates a set of stack IDs, one for each
invocation.  These then get passed along from one shader invocation to
the next as we trace the ray.  We can use those stack IDs to figure out
which stack our invocation needs to access.  Because we may not be the
first shader in the stack, there's a per-stack offset that gets stored
in the "hotzone".

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add lowering functions for each ray-tracing stage

These will eventually contain per-stage lowering for various ray-tracing
things.  This is separate from brw_nir_lower_rt_intrinsics because, for
reasons that will become apparent later, brw_nir_lower_rt_intrinsics has
to be run very late in the compile process, right before brw_compile_bs.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a pass to lower the new ray-tracing intrinsics

The new intrinsics we added for doing address calculations are all
things we fetch from the RT_DISPATCH_GLOBALS struct.  We could emit an
RT_DISPATCH_GLOBALS load at every point we want it and trust NIR to CSE
it for us but it's easier to use intermediate intrinsics.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add builder helpers for accessing RT data structures

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/fs: Add and implement a load_global_const_block intrinsic

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/rt: Add a brw_rt.h header with #defines for basic RT data structures

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/compiler: Add support for bindless shaders

The Intel bindless thread dispatch model is very simple.  When a compute
shader is to be used for bindless dispatch, it can request a set of
stack IDs.  These are allocated per-dual-subslice by the hardware and
recycled automatically when the stack ID is returned.  Passed to the
bindless dispatch are a global argument address, a stack ID, and an
address of the BINDLESS_SHADER_RECORD to invoke.  When the bindless
shader is dispatched, it is passed its stack ID as well as the global
and local argument pointers.  The local argument pointer is the address
of the BINDLESS_SHADER_RECORD plus some offset which is specified as
part of the BINDLESS_SHADER_RECORD.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/debug: Add a debug flag for ray-tracing shaders

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir/lower_io: Support shader_call_data in vars_to_explicit_types

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir/lower_io: Allow ray_hit_attrib in lower_vars_to_explicit_types

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir: Add a helper to get the live set at a cursor

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Add BVH data structures

These describe the Intel BVH format used for storing acceleration
structures.

Acked-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Add RT_DISPATCH_GLOBALS and RT_*_SBT_HANDLE structs

The RT_DISPATCH_GLOBALS struct is half HW-defined by the ray-tracing
spec and half SW-defined.  However, due to the addresses in it, it's
convenient for it to all be in GenXML.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Support truncated addresses

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml/pack: Stash the cloned address field

The cloned version is the one that has updated start and end bits
fields.  We're about to start passing those through to a new
__gen_address function and we need the correct start/end in order to do
that reliably.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/genxml: Add the BINDLESS_SHADER_RECORD data structure

This is the first of the HW data structures added for ray-tracing.
These are added to their own file because it's not really associated
with any hardware we've enabled in Mesa just yet.  Eventually, these
will likely get folded into the appropriate genX.xml file as they are
hardware data structures and needed to be tracked as such.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

intel/dev: Add a gen_device_info::has_ray_tracing bit

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

spirv: Emit nir_jump_halt after TerminateRay or IgnoreIntersection

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

nir: Add a halt instruction type

Halt is like a return for the entire shader or exit() if you prefer to
think of it that way.  Once an invocation hits a halt, it's 100% dead.
Any writes to output variables which happened before the halt do,
however, still apply.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7356>

meson: add idep_mesautil to components using simple_mtx.h

If valgrind is installed, these components need to find valgrind.h.

Fixes: 53f7d539cd9 ("util: Add helgrind support for simple_mtx")
Closes: #3876
Acked-by: Rob Clark <robclark@freedesktop.org>

Revert "zink: initial implementation of shader keys"

This reverts commit 2be2a500a396fe1dc0e121816e4056874cdd43fc.

Fixes: 2be2a500a39 ("zink: initial implementation of shader keys")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: refcount the shader cache"

This reverts commit b9fdc21bba0724271520462f3f04ba72ae106a26.

Fixes: b9fdc21bba0 ("zink: refcount the shader cache")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: move shader key structs into their own header"

This reverts commit ed7a5a55689ddcf10aae3f671e390252aad2d62f.

Fixes: ed7a5a55689 ("zink: move shader key structs into their own header")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: fill in params for fs shader keys and flag shader for rebuild"

This reverts commit 9aa08221fa6653002dec2a07641d58d3c7fa227a.

Fixes: 9aa08221fa6 ("zink: fill in params for fs shader keys and flag shader for rebuild")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: put those shader keys to work fixing up fragment shaders"

This reverts commit eeff625ab31f0a1bf94739e09ca6b08de00c94f5.

Fixes: eeff625ab31 ("zink: put those shader keys to work fixing up fragment shaders")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

Revert "zink: update shader modules in gfx program when flagged dirty"

This reverts commit e96afeeb7bb2306aa0fba76e27163d4c7a86860b.

Fixes: e96afeeb7bb ("zink: update shader modules in gfx program when flagged dirty")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7754>

radv: Fix RB+ blending for VK_FORMAT_E5B9G9R9_UFLOAT_PACK32.

Fixes: e893102bcf5 ("radv: Add VK_FORMAT_E5B9G9R9_UFLOAT_PACK32 rendering support.")
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7716>

mesa: Synchronize get_gl_override()

Fixes helgrind complaint found with piglit glx-multithread-clearbuffer.
This is a legit race because override[api].version is cleared before
parsing the override string.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

tgsi: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

gallium/trace: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

mesa: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

util: Fix helgrind complaint about one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

mesa/st: Use do_once for one-time init

Also, since there is a second call-path into st_init_extensions() from
get_version(), add an extra st_debug_init() call.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

gallium/hud: Use do_once for one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

freedreno/ir3: Use get_once() for one-time init

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

nir: Use get_once() helper for one-time init's

Makes the code more concise, and makes helgrind/drd happy at the same
time!

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

util: Add helpers for various one-time-init patters

A fairly common pattern for debug envvars is something like:

   static int should_print = -1;
   if (should_print < 0)
      should_print = env_var_as_unsigned("NIR_PRINT", 0);

Unfortunately helgrind doesn't realize that we expect to always get the
same return value, so we don't actually care about the race condition
here.

Add a helper get_once() and do_once macros, with extra locking to make
helgrind/drd happy.  Note that other than the nir usages (which are
limited to debug builds), other usages are not in hot-paths.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

util: Add helgrind support for simple_mtx

Annoyingly mtypes.h pulls in simple_mtx, which means we end up needing
to sprinkle a lot of idep_mesautil around.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3773
Signed-off-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Kristian H. Kristensen <hoegsberg@google.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7644>

spirv: Implement SpvOpConvertUToAccelerationStructureKHR

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Implement OpTerminateRayKHR and OpIgnoreIntersectionKHR

In the final version of SPV_KHR_ray_tracing, these are now block
terminators like OpKill or OpReturn.  This means that they need special
handling in vtn_cfg.c.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Call repair SSA for OpTerminateInvocation

Fixes: 886d2d1a9abcb "spirv: Handle SpvOpTerminateInvocation"

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Implement OpTraceRayKHR and OpExecuteCallableKHR

The old NV version (and the provisional KHR version) specified the data
payload via an integer location.  This was quite annoying for the parser
and potentially error-prone.  The final KHR version of the SPIR-V
ray-tracing spec replaces these integers with actual pointers.  We don't
really need to implement the NV versions but we have the code and
someone might want to parse some NV ray-tracing shaders.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Update JSON and headers from Khronos main

This corresponds to 104ecc356c1bea4476320faca64440cd1df655a3 ("Merge
pull request #180 from dneto0/issue-179") in
https://github.com/KhronosGroup/SPIRV-Headers

Acked-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

spirv: Rename some ray-tracing intrinsics to NV

For these intrinsics, the NV version and the provisional KHR version
have the same enum value and semantics but the final KHR version is
different on both counts.  Re-name them to NV before we update the
header so the header update isn't a functional change.

Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

vulkan: Update XML and headers to 1.2.162

This brings in the following new extensions:

 - VK_KHR_deferred_host_operations
 - VK_KHR_pipeline_library
 - VK_KHR_acceleration_structure
 - VK_KHR_ray_tracing_pipeline
 - VK_KHR_ray_query

Acked-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Acked-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7734>

zink: fall back to util_blitter for scaled resolves

Vulkan can't scale while resolving using vkCmdResolveImage. For this we
need to use util_blitter.

The reason this wasn't a problem in the past, was that glBlitFramebuffer
always set pipe_blit_info::render_condition_enable, and we always used
that to bail out to util_blitter. When the latter changed, this broke.

Fixes: 19906022e22 ("zink: more accurately track supported blits")
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7745>

nir/opt_peephole_select: respect selection_control when collapsing ifs

Totals from 34 (0.02% of 138013) affected shaders (RAVEN):
CodeSize: 625888 -> 626336 (+0.07%); split: -0.00%, +0.08%
Instrs: 124121 -> 124229 (+0.09%); split: -0.00%, +0.09%
Cycles: 1403072 -> 1403588 (+0.04%); split: -0.01%, +0.04%
VMEM: 5308 -> 5364 (+1.06%); split: +1.07%, -0.02%
Copies: 12773 -> 12838 (+0.51%); split: -0.08%, +0.59%
Branches: 5758 -> 5801 (+0.75%); split: -0.21%, +0.96%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7478>

nir/opt_peephole_select: collapse nested IFs if applicable

Single-sided nested IFs can sometimes be collapsed
even if they cannot be flattened.
This optimization re-uses block_check_for_allowed_instrs()
to determine if it is beneficial to collapse the IFs.
Additionally, it is required that the phis of the outer IF
become trivial after this optimization, so that no additional
bcsel instructions are added.
This optimization turns

   if (cond1) {
      <allowed instruction>
      if (cond2) {
         <any code>
      } else {
      }
   } else {
   }

into

   <allowed instruction>
   if (cond1 && cond2) {
      <any code>
   } else {
   }

Totals from 17044 (12.35% of 138013) affected shaders (RAVEN):
SGPRs: 1246416 -> 1246256 (-0.01%); split: -0.01%, +0.00%
VGPRs: 802752 -> 802736 (-0.00%); split: -0.01%, +0.01%
SpillSGPRs: 45857 -> 45850 (-0.02%); split: -0.07%, +0.05%
CodeSize: 85318240 -> 85208592 (-0.13%); split: -0.15%, +0.02%
Instrs: 16769049 -> 16738195 (-0.18%); split: -0.20%, +0.02%
Cycles: 947328732 -> 947145796 (-0.02%); split: -0.03%, +0.01%
VMEM: 7271539 -> 7274090 (+0.04%); split: +0.05%, -0.01%
SMEM: 925983 -> 927374 (+0.15%); split: +0.19%, -0.04%
VClause: 294334 -> 294340 (+0.00%); split: -0.00%, +0.00%
SClause: 633600 -> 634048 (+0.07%); split: -0.01%, +0.08%
Copies: 1589650 -> 1580573 (-0.57%); split: -0.66%, +0.09%
Branches: 540830 -> 525767 (-2.79%); split: -2.79%, +0.00%
PreSGPRs: 902500 -> 902415 (-0.01%); split: -0.02%, +0.01%
PreVGPRs: 759992 -> 760019 (+0.00%); split: -0.00%, +0.01%

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7478>

v3dv: fix allocation size for BO handles

We were allocating twice the size we need for this array. This was
probably caused by a copy and paste error from the GL driver which
grows this dynamically as BOs are added to the job.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7733>

v3dv: remove obsolete comment

We already check that the feature is present in our kernel when we
initialize the physical device.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7731>

clover: Fix typo in comment.

Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7702>

gallium: fix missing bit field in p_state.h

Marek pointed this out, not sure how we missed it.

Fixes: 3dc6da1ac1f5 ("gallium: add a non-multisample sample mask out behaviour flag.")

Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7741>

lavapipe: enable pipeline stats queries

These pass CTS, but I think are missing some stuff CTS doesn't test.

This is one of the base Vulkan 1.0 features and I'd like to support
it for conformance.

Cc: "20.3" <mesa-stable@lists.freedesktop.org>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7705>

lavapipe: fixup mipmap precsion bits

8 seems more correct, however it fixes a bunch of explict lod
tests but breaks some lod query tests.

Cc: "20.3" <mesa-stable@lists.freedesktop.org>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7705>

llvmpipe: fix multisample lines.

This also needs another lines fix, but at least align the code
with tri and points

Cc: "20.3" <mesa-stable@lists.freedesktop.org>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7705>

llvmpipe: fix multisample point rendering.

Fixes one case in
dEQP-VK.rasterization.primitives_multisample_4_bit.no_stipple.points

Cc: "20.3" <mesa-stable@lists.freedesktop.org>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7705>

llvmpipe/setup: move point stats collection earlier.

You have to count the stats pre-culling here.

Just like dc261cdd42380 did for lines.

VK-GL-CTS dEQP-VK.query_pool.statistics_query.clipping_primitives*point_list

Cc: <mesa-stable@lists.freedesktop.org>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7705>

lavapipe: fix wsi acquire fences

Fixes:
dEQP-VK.wsi.xcb.swapchain.acquire.too_many

Cc: "20.3" <mesa-stable@lists.freedesktop.org>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7705>

lavapipe: fixup device allocate + enable private data

I'd only half ported private memory support, finish the job.

Cc: "20.3" <mesa-stable@lists.freedesktop.org>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7705>

zink: fix layered resolves

Until recently, we ended up using u_blitter here, because
info->render_condition_enable was always true here. But when we recently
fixed that overly broad check, this broke.

So let's fix layered-resolves, by actually checking if the resource has
layers respect them in that case, similar to what we do in blit_native.

Fixes: 19906022e22 ("zink: more accurately track supported blits")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3843
Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7737>

release-calender: Update 20.3

I've been forgetting to remove completed rc's

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7739>

docs: update calendar and link releases notes for 20.2.3

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7739>

docs: Add relnotes for 20.2.3

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7739>

docs: add release notes for 20.2.3

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7739>

aco: optimize v_add+v_lshlrev to v_mad_u32_u24 on GFX6-8

This optimizes v_add(c, v_lshlrev(a, b)) to v_mad_u32_u24(b, 1<<a, c)
if 'a' is a constant (less than or equal to 6 to avoid creating
literals) and 'b' known to be a 16-bit or a 24-bit value.

On GFX9+, this is already optimized to v_lshl_add_u32.

No fossils-db changes.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7673>

aco: optimize v_add+s_lshl to v_mad_u32_u24 on GFX6-8

This optimizes v_add(c, s_lshl(a, b)) to v_mad_u32_u24(a, 1<<b, c)
if 'b' is a constant (less than or equal to 6 to avoid creating
literals) and 'a' known to be a 16-bit or a 24-bit value.

On GFX9+, this is already optimized to v_lshl_add_u32.

fossils-db (Polaris10):
Totals from 1916 (1.36% of 140385) affected shaders:
SGPRs: 88322 -> 87780 (-0.61%); split: -0.66%, +0.05%
CodeSize: 7852668 -> 7851800 (-0.01%); split: -0.01%, +0.00%
Instrs: 1533965 -> 1530459 (-0.23%); split: -0.23%, +0.00%
Cycles: 57001852 -> 56983244 (-0.03%); split: -0.03%, +0.00%
VMEM: 372561 -> 371733 (-0.22%); split: +0.03%, -0.25%
SMEM: 108859 -> 103711 (-4.73%); split: +0.23%, -4.96%
VClause: 37231 -> 37204 (-0.07%)
SClause: 58116 -> 58086 (-0.05%); split: -0.06%, +0.01%
Copies: 199953 -> 199931 (-0.01%); split: -0.03%, +0.02%
Branches: 63478 -> 63477 (-0.00%)
PreSGPRs: 61818 -> 61816 (-0.00%)

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7673>

aco: allow to use the range analysis UB in emit_{sop2,vop2}_instruction()

It will allow to combine v_add+s_lshl or v_add+v_lshlrev to
v_mad_u32_u24 on GFX6-8 if operands are known to be 16-bit or 24-bit.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7673>

aco: add a new Operand flag to indicate that is 24-bit

To indicate that the upper 8-bits are always 0 to optimize more MADs.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7673>

aco/tests: extend the optimize.add_lshl tests to GFX8

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7673>

ac,radv: use better export formats for 8-bit when RB+ isn't allowed

When RB+ is enabled, R8_UINT/R8_SINT/R8_UNORM should use FP16_ABGR
for 2x exporting performance. Otherwise, use 32_R to remove useless
instructions needed for 16-bit compressed exports.

fossils-db (Vega10):
Totals from 8858 (6.35% of 139517) affected shaders:
SGPRs: 801248 -> 801210 (-0.00%); split: -0.01%, +0.00%
VGPRs: 596224 -> 596120 (-0.02%); split: -0.02%, +0.01%
CodeSize: 71462452 -> 71356684 (-0.15%); split: -0.15%, +0.00%
MaxWaves: 37097 -> 37105 (+0.02%); split: +0.04%, -0.02%
Instrs: 13963177 -> 13950809 (-0.09%); split: -0.09%, +0.00%
Cycles: 1476539360 -> 1476489996 (-0.00%); split: -0.00%, +0.00%
VMEM: 2363008 -> 2361349 (-0.07%); split: +0.04%, -0.11%
SMEM: 550362 -> 549977 (-0.07%); split: +0.01%, -0.08%
VClause: 245704 -> 245727 (+0.01%); split: -0.01%, +0.02%
SClause: 485161 -> 485104 (-0.01%); split: -0.01%, +0.00%
Copies: 1420034 -> 1422310 (+0.16%); split: -0.01%, +0.17%
Branches: 518710 -> 518705 (-0.00%)
PreSGPRs: 706633 -> 706584 (-0.01%)
PreVGPRs: 547163 -> 547007 (-0.03%); split: -0.03%, +0.01%

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7512>

radv: add new vk_format_is_*() helpers

I think we should make RADV uses util_format everywhere.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7512>

meson: use a feature option for microsoft-clc

It's less code and makes the configuration easier to fine tune.

Fixes: ff05da7f8dc4aa531704d48f718514e3b1fff45d
       ("microsoft: Add CLC frontend and kernel/compute support to DXIL converter")

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7699>

meson: Don't add extra values to shader-cache

We're trying to move to using a feature here, adding more values breaks
that.

Fixes: 5de56937a3d009659076dc67de6a57379fc7a31b
       ("disk_cache: build option for disabled-by-default")

Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7699>

freedreno/ir3: Fix valgrind complaint about streamout state

The warning is a bit misleading about where it shows up.. it complains
about the shader key, due to shader key being calculated from (among
other things) stream_output state that had some uninitialized garbage
in the padding.

==84572== Uninitialised byte(s) found during client check request
==84572==    at 0x60548E8: blob_write_bytes (blob.c:163)
==84572==    by 0x6534EF7: compute_variant_key (ir3_disk_cache.c:111)
==84572==    by 0x6535143: ir3_disk_cache_retrieve (ir3_disk_cache.c:171)
==84572==    by 0x654D82F: create_variant (ir3_shader.c:251)
==84572==    by 0x654DA2B: ir3_shader_get_variant (ir3_shader.c:301)
==84572==    by 0x645B2CB: ir3_shader_variant (ir3_gallium.c:113)
==84572==    by 0x645B7EB: ir3_shader_create (ir3_gallium.c:219)
==84572==    by 0x645BAA7: ir3_shader_state_create (ir3_gallium.c:285)
==84572==    by 0x6506003: fd6_shader_state_create (fd6_program.c:1136)
==84572==    by 0x64676C7: assemble_tgsi (freedreno_program.c:105)
==84572==    by 0x64679DF: fd_prog_init (freedreno_program.c:188)
==84572==    by 0x6506157: fd6_prog_init (fd6_program.c:1172)
==84572==  Address 0xeff1588 is 424 bytes inside a block of size 480 alloc'd
==84572==    at 0x4866FA4: malloc (vg_replace_malloc.c:307)
==84572==    by 0x605D46F: ralloc_size (ralloc.c:133)
==84572==    by 0x605D52F: rzalloc_size (ralloc.c:166)
==84572==    by 0x654DFF7: ir3_shader_from_nir (ir3_shader.c:473)
==84572==    by 0x645B6C7: ir3_shader_create (ir3_gallium.c:182)
==84572==    by 0x645BAA7: ir3_shader_state_create (ir3_gallium.c:285)
==84572==    by 0x6506003: fd6_shader_state_create (fd6_program.c:1136)
==84572==    by 0x64676C7: assemble_tgsi (freedreno_program.c:105)
==84572==    by 0x64679DF: fd_prog_init (freedreno_program.c:188)
==84572==    by 0x6506157: fd6_prog_init (fd6_program.c:1172)
==84572==    by 0x64CB36F: fd6_context_create (fd6_context.c:154)
==84572==    by 0x59D93BB: st_api_create_context (st_manager.c:917)

Somehow this was showing up with dEQP-GLES31.info.vendor but not other
things.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7717>

freedreno/drm: Quiet timedout error msg

This isn't terribly interesting, but got more chatty when we converted
to mesa_loge() vs debug_printf()

Fixes: 156d7e45f74 ("freedreno: Convert to mesa_log*()")
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7717>

freedreno/a6xx: Clear control mem at context create

We could be getting a recycled bo containing random garbage, which can
confuse check_vsc_overflow().

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7717>

freedreno: Convert one last mtx_t -> simple_mtx_t

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7717>

freedreno: emit_marker() cleanup

1) Propagate the change to only emit markers in debug builds (and add
   the WFI that ensures they are synchronized with GPU.  We could
   consider dropping them entirely, since the GPU devcoredump support
   in newer kernels is more useful.  But it is still an occasionally
   useful fallback.

2) Use p_atomic_inc_return() to placate helgrind

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7717>

mesa: add an environment variable to default enable INTEL_blackhole

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7640>

st: trigger noop if the default value is not true

v2: Verify that PIPE_CAP_FRONTEND_NOOP is available before calling vfunc (Icecream95)

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com> (v1)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7640>

ir3/ra: Fix array reg liveness in scalar pass

Assigning an array reg removes IR3_REG_ARRAY, which means that
definitions and uses can't be tracked back to the array register's name
and liveness for the components of the array aren't correctly
calculated. To fix this we delay assigning array registers until the
scalar pass.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7711>

nir: fix gathering cross invocation info

Fixes: 5b77b14448c ("nir: Use src_is_invocation_id in get_deref_info.")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7730>

swr: Pass draw start information to state update mechanism

This fixes crash in many workloads/tests

Reviewed-by: Krzysztof Raszkowski <krzysztof.raszkowski@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7728>

ci: fix name of the Sienna Cichlid expected failures file

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7729>

v3dv/cmd_buffer: missing (uint8_t *) casting when calling memcmp

Caused to return early wrongly on CmdPushConstants with some tests
using several calls to that method. As we are here we are also
replacing the (void *) casting at the memcpy below.

Fixes: e1c8041cde64 ("v3dv: try harder to skip emission of redundant state")

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7718>

radv: dump BO ranges into bo_ranges.log instead of stderr

Like other dumps during GPU hang detection.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7706>

radv: add RADV_DEBUG=noumr to disable UMR logs during GPU hang detection

Sometimes UMR logs can't be dumped and you would get permission
denied, even if the UMR binary has the setuid bit enabled.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7706>

radv: dump application info in the GPU hang report

Like the name, version, as well as the engine and the API version.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7706>

radv: append a time string to the hang report dump directory

Using the PID only isn't really informative.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7706>

radv: print more debug messages when generating a hang report

If for some reasons the driver can't generate the hang report properly.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7706>

radeonsi: don't allocate LDS for TCS inputs if it's not used

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7623>