ir3/legalize: Construct branch properly

Don't just yeet stuff into regs without updating regs_count, etc. This
will break horribly during the transition otherwise.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Add separate src/dst count in ir3_instr

srcs and dsts will be in separate arrays, so we need everything creating
it to give a separate source and dest max count.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Split ir3_reg_create() into ir3_{src,dst}_create()

Right now they are basically the same, but in the future they will
append to different arrays.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Make ir3_instruction::address a normal register

This fixes an annoying mismatch in the indices between foreach_ssa_src_n
and ir3_delayslots(), and lets us remove a bunch of other special cases.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Add is_reg_special()

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Validate that ir3_register::instr is correct

Catch the mistake fixed in the previous commit.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Update ir3_register::instr when cloning instructions

We happened to not clone any SSA instructions, but we will once address
instructions start counting as SSA. Fix this oversight.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Split read-modify-write array dests in two

Instructions that operate on an array read the previous state of the
array, modify it, and write a new array, at least conceptually before
RA. Previously the same register specified the previous state and acted
as the new state, but this meant that it was both a source and
destination which meant that it was getting in the way of splitting up
sources and destinations. Break out the source into a separate register,
and use the new tied-src infrastructure to share code with a6xx atomics.
With this, there are basically no more special cases for arrays in RA.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

ir3: Make tied sources/destinations part of the IR

Previously this was hard-coded for a6xx atomic instructions. However
we'll need a way for array destinations to point to the source with the
previous value of the array when we split them up. This is conceptually
the same as tied source/destinations for a6xx atomics, except that array
writes sometimes won't have a previous value to point to. So move this
into the IR so that it can be more dynamic. As a bonus we can move the
knowledge of a6xx atomics out of RA, where it's out-of-place, and into
the a6xx-specific code that creates them.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11469>

amd/ci: add hawaii-specific skip and fail lists

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10424>

pan/mdg: Add 16 bytes of padding to the end of shaders

Fixes INSTR_INVALID_PC faults when a shader ends on a 16MB boundary.

Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11551>

anv: fix emitting dynamic primitive topology

Initial implementation missed various fields that derive from the
primitive topology. This patch fixes 3DSTATE_RASTER/3DSTATE_SF,
3DSTATE_CLIP and 3DSTATE_WM (gen7.x) emission in the dynamic case.

Fixes: f6fa4a80000 ("anv: add support for dynamic primitive topology change")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4924
Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11379>

Revert "ci: disable panfrost t760 jobs"

Both our T760 machines took a dive in beautiful synchronicity last
night, were recovered early this morning.

This reverts commit 854d93f73d6064a13ddc13dddf74c8c760cda1d4.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11555>

iris: take a reference to memobj bo in iris_resource_from_memobj

v2: and remove it from iris_memobj_create_from_handle ... (Nanley)

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4969
Fixes: 772dc50d162 ("iris: hook up resource creation from memory object")
Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11552>

radeonsi: disable ngg culling on llvm < 12

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4874
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
CC: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11362>

anv: bound checks buffer memory binding in debug builds

Validation layers should warn you about this
(VUID-VkBindBufferMemoryInfo-size-01037) but this would be useful for
zink debugging.

Requested by Zmike.

v2: Also check memoryOffset (Jason)

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11542>

intel/fs: Implement Wa_14013745556 on TGL+.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11433>

intel/fs: Fix synchronization of accumulator-clearing W/A move on TGL+.

Right now the accumulator-clearing move emitted by the generator for
Wa_14010017096 inherits the SWSB field from the previous instruction.
This can lead to redundant synchronization, or possibly more serious
issues if the previous instruction had a TGL_SBID_SET SWSB
synchronization mode.  Take the SWSB synchronization information from
the IR.

Fixes: a27542c5ddec8 ("intel/compiler: Clear accumulator register before EOT")
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11433>

intel/fs: Teach IR about EOT instruction writing the accumulator implicitly on TGL+.

This is unlikely to have had any negative side effect on the original
TGL, but will lead to issues on XeHP+ if the software scoreboard pass
isn't able to synchronize the accumulator writes.

Fixes: a27542c5ddec8 ("intel/compiler: Clear accumulator register before EOT")
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11433>

intel/fs: Add SWSB dependency annotations for cross-pipeline WaR data hazards on XeHP+.

In cases where an in-order instruction is overwriting a register
previously read by another in-order instruction, drop the dependency
iff the previous read is guaranteed to have occurred from the same
in-order pipeline.  This should only have an effect on XeHP+ since
previous Xe platforms only had one in-order FPU pipeline.

The previous workaround we were using for this treated all ordered
read dependencies as write dependencies to avoid noise from our
simulation environment.  Relative to our previous workaround this
improves performance of GFXBench5 gl_tess by ~7% on a DG2 system
among other single-digit percentual FPS improvements.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11433>

intel/fs: Implement Wa_22012725308 for cross-pipe accumulator data hazard.

The hardware fails to provide the expected data coherency guarantees
for accumulator registers when accessed from multiple FPU pipelines.
Fix this by tracking implicit accumulator accesses just like we do for
regular GRF registers, but instead of adding synchronization
annotations for any dependency we only do it for dependencies with a
pipeline mismatch, since the hardware should be able to guarantee
proper synchronization for matching pipelines.

Note that this workaround handles RaW and WaW dependencies in addition
to the WaR dependencies described in the hardware bug report even
though cross-pipeline RaW accumulator dependencies should be extremely
rare, since chances are the hardware will also hang if we ever hit
such a condition.  This only affects XeHP+, since all FPU instructions
are executed as a single in-order pipeline on earlier Xe platforms.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11433>

intel/fs: Track single accumulator in scoreboard lowering pass.

This change reduces the precision of the scoreboard data structure for
accumulator registers, because the rules determining the aliasing of
accumulator registers are non-trivial and poorly documented (e.g. acc0
overlaps the storage of acc1 when the former is accessed with an
integer type).  We could implement those rules but it wouldn't have
any practical benefit since we currently only use acc0-1, and for the
most part we can rely on the hardware's accumulator dependency
tracking.  Instead make our lives easier by representing it as a
single register.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11433>

intel/fs/xehp: Assert that the compiler is sending all 3 coords for cubemaps.

As required by HSDES:14013363432.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11433>

freedreno/ci: Use TU_IGNORE_CONFORMANCE_WARNING to reduce warnings

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11543>

tu: Provide a toggle to avoid warnings about unsupported devices

In the CI, we have such devices, and this message is printed many
hundreds of times. This results in a useless spam which makes it
difficult to see real issues.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11543>

meson/crocus: add prefer-crocus option.

This just allows picking crocus without having to set the env var.

Acked-by: Alyssa Rosenzweig <alyssa@collabora.com>
Acked-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11353>

mesa: unreference zombie buffers when creating buffers to lower memory usage

This fixes an issue where one context only creates buffers while another
context only destroys buffers. Only the creating context can release its
buffers and the destroying context only turns them into zombie buffers.

This fix makes the creating context release its zombie buffers.
It's not a plot from an apocalyptic movie.

Fixes: e014e3b6be6 "mesa: don't count buffer references for the context that created them"
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4840

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11514>

crocus: fix batch state bo leak

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11547>

crocus: fix vertex buffer leak on screen end.

this was stopping the screen from being cleaned up as well

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11547>

crocus: free context state properly.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11547>

zink: ci updates

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

zink: remove primconvert

this is no longer used

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

zink: export supported primitive restart types

this is now handled by gallium, so the codepath can be dropped

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

zink: export supported prim types

this is now handled by gallium, so the related codepaths can be dropped

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

zink: export 8bit index buffer support based on extension presence

this is now handled by gallium, so the codepath can be dropped

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

zink: export PIPE_CAP_EMULATE_NONFIXED_PRIMITIVE_RESTART

gallium will now handle unsupported restart index rewriting, so
this codepath can be removed

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

gallium: add a pipe cap for determining driver support for prim type in restarts

this is another prim type bitmask which will trigger automatic draw rewriting
to a direct draw any time a prim-restart draw occurs with a prim type that is
not supported by the driver for prim restart, even if that prim type is supported
for normal drawing

the default is set to all prim types to preserve existing functionality, and PrimitiveRestartForPatches
is now explicitly set to false because no driver supports it

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

gallium: add a pipe cap for performing automatic prim type conversion

drivers can now export a bitmask of the primitive types they support,
and all others will be automatically be rewritten

the default value is set to all primitive types supported to preserve
existing behavior

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

gallium: handle automatic 8bit -> 16bit index buffer rewrites

this is based on PIPE_FORMAT_R8_UINT format support

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

gallium: add a pipe cap to rewrite index buffers for draws using a non-fixed restart index

for drivers that set it, this now automatically handles restart index rewriting
by running draws through primconvert when necessary

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

util/primconvert: handle multidraws in primconvert

once a draw reaches primconvert, it should never be able to reach the driver
until all draw operations have been converted as necessary

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

util/primconvert: handle rewriting of prim-restart draws with unsupported primtype

this is a bit more work, as the primitive restart needs to be rewritten into a multidraw,
then the multidraw converted to the new primitive type and serialized back into a
single draw

detection is handled using a new primconvert config member, which is set to the full
primtype mask by default for compatibility with existing drivers

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

util/primconvert: map index buffer before getting index translator function

also set other local values on init using new util functions

no functional changes

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

util/primconvert: handle indirect draws

this avoids ping-ponging to the driver, which would have to call back
to here anyway with the unwrapped indirect draw

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

util/prim_restart: break out draw rewriting into separate function

this enables doing the draw rewrites without calling draw_vbo

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

aux/draw: add a util function for reading back indirect draw params

the return type of this is a bit clunky because instance values can change,
but it's simpler to just return the full draw info struct than to force the
caller to keep pulling from arrays or whatever

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/10973>

ci: disable panfrost t760 jobs

these are timing out

Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11548>

crocus: Explicitly cast value to uint64_t

Ports 565a80450d28f6daa0ca8b98dad93924e712f94b from iris.

Acked-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11546>

amd/addrlib: remove Meson debug message()

This message pops up in the Meson build logs, but has no context
and doesn't seem to be directed at end-users. Sounds like a leftover
from a debugging session.

Signed-off-by: Simon Ser <contact@emersion.fr>
Fixes: d0767fc045a1 ("amd/addrlib: use cpp.has_argument() to filter compiler arguments")
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11524>

radeon/vcn/enc: Add missing line to HEVC SPS header code

Add previously missing line in HEVC SPS header generation on VCN 1.0
devices, for when an image does not need to be cropped.

Fixes: e62c7e7c6ca ("radeon: Add cropping to encoded H.265 when padding is used")
Signed-off-by: Thong Thai <thong.thai@amd.com>
Reviewed-by: Leo Liu <leo.liu@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11535>

ac/llvm: don't set skip-uniform-regions to fix atomic.cmpswap

This fixes SSBO atomic cmpswap tests and transform feedback order piglit tests.

Fixes: 3bb65c06706 - ac: force enable -structurizecfg-skip-uniform-regions for LLVM 11
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3884

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Acked-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11517>

util: add force_gl_names_reuse for SWKOTOR.

SWKOTOR is expecting texture names generated by glGenTextures() to
be small enough and crashes otherwise.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11527>

i915g: Fix GL_ARB_copy_buffer assertion fails.

The i915_texture() asserts that the resource is not a buffer, so check for
the buffer fallback first.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11540>

nir: use a more fitting index for btd_stack_push_intel

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

nir: drop the btd_resume_intel intrinsic

This is now 100% equivalent to the new rt_resume intrinsic.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

intel/rt: switch to common pass for shader calls lowering

v2: rename for new indices

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: implement caching for ray tracing pipelines

v2: Turn a bunch of pointer checks into checks against NULL (Jason)

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: move trivial return shader to device

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: store more RT shader data in pipeline_stage object

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: cache raytracing trampoline shader

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

blorp: add blorp string in shader keys

Upon looking at caching the raytracing shader (in particular the
trampoline one) I kind of got afraid that some of the keys used for
blorp would end up matching other keys. This is because blorp keys are
fairly simple. There is no SPIRV module hash included.

This change includes a "blorp" string at the beginning of the queue to
ensure we don't collide with other keys.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Implement vkCmdTraceRays and vkCmdTraceRaysIndirect

v2: Fix anv_cmd_state::binding_tables array size (Lionel)

v2: Fix anv_cmd_state::samplers array size (Lionel)

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Make anv_address::offset 64-bit

This allows us to convert a 64-bit address to an anv_address which is
useful for working with device addresses.

v2: switch to int64_t to keep state pool relative relocation working
    on non-softpin platforms

v3: Update assert to reflect relative offsets (Jason)

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Allow _anv_combine_address with a NULL batch

This is required in order to be able to use GenXML pack functions for
structs with addresses when you're not packing into a batch.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Add support for vkCmdSetRayTracingPipelineStackSizeKHR

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Compute scratch sizes for ray-tracing pipelines and shader groups

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Create and return ray-tracing pipeline SBT handles

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

intel/fs: Don't pull CS push constants if uses_inline_data

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Compile trivial return and trampoline shaders

These don't necessarily go in any group but are required for dispatch to
work properly.  The trampoline is a compute shader that is the initial
start point for the trace.  It's in charge of invoking the actual
ray-gen shader.  The trivial return shader is used whenever another
shader is missing and it does no work except the minimum required to do
a stack return.

v2: Rebase on upstream changes (Lionel)

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Compile ray-tracing shaders

This doesn't look too different from other compile functions we have in
anv_pipeline.c.  The primary difference is that ray-tracing pipelines
have this weird two-stage thing where you have "stages" which are
individual shaders and "groups" which are sort of mini pipelines that
are used to handle hits.  For any given ray intersection, only the hit
and intersection shaders from the same group get used together.  You
can't have an intersection shader from group A used with an any-hit from
group B.  This results in a weird two-step compile.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Support fetching descriptor addresses from push constants

Bindless shaders don't have binding tables so they have to get at the
descriptor sets via a different mechanism.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

nir/apply_pipeline_layout: Handle bindless shaders

They don't have binding tables so they have to use A64 descriptor set
access and everything has to be bindless all the time.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Disallow UBO pushing for bindless shaders

They don't really have push constants.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

intel/rt: Use reloc constants for the resume SBT

It's going to be attached to the end of the shader binary, not an
arbitrary table somewhere in memory.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

intel/fs: Add support for compiling bindless shaders with resume shaders

Instead of depending on the driver to compile each resume shader
separately, we compile them all in one go in the back-end and build an
SBT as part of the shader program.  Shader relocs are used to make the
entries in the SBT point point to the correct resume shader.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

intel/compiler: Add a U32 reloc type

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

intel/compiler: Generalize shader relocations a bit

This commit adds a delta to be added to the relocated value as well as
the possibility of multiple types of relocations.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv,iris: Move the SHADER_RELOC enums to brw_compiler.h

They're common between the two drivers and we want to add a couple more
that get emitted from code in src/intel/compiler.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Add support for binding ray-tracing pipelines

This includes not only vkCmdBindPipeline but also descriptor sets and
push constants.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Add a ray-tracing pipeline object

This just adds the core data structure which we'll build on going
forward.

v2: Add VK_EXT_pipeline_creation_cache_control handling (Lionel)

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Get ready for more pipeline stages

This makes a bunch of loops use ARRAY_SIZE instead of MESA_SHADER_STAGES,
extends a few arrays, and adds a bunch of array length asserts.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Add minimal boilerplate for VK_KHR_ray_tracing_pipeline

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Add support for binding acceleration structures

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Add stub support for acceleration structures

This just adds a base struct and trivial implementations of all the
create/destroy/bind functions.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

anv: Add minimal boilerplate for VK_KHR_acceleration_structure

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

spirv: Create acceleration structure and shader record variables

spirv_to_nir now requires NIR variables to be created for everything.

Fixes: 10b3eecd361af "spirv: Don't remove variables used by resource..."
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8637>

zink: don't clear batch resources on fence finish

this eliminates the need for locking here and has no noticeable
increase in overhead, so the reduction of complexity seems worthwhile

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: move some end-of-batch stuff to submit thread

no reason to do this on the driver thread when it could be async

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: flag scanout updates to batch state, not resource

now that submission is serialized better, it's not actually the resource that should be
tagged for scanout sync, it's the batch state, as multiple contexts might reuse the same
resource, thus requiring synchronization on every submit

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: move fence reset to submit thread

try to reduce job queue latency

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: move sparse buffer commit to screen queue

all queue submission must be serialized with the dispatch thread

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: move queue submit thread to screen

this needs to be a screen-based queue in order to serialize properly
with multiple contexts

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: enforce multi-context waiting for unflushed resources on foreign batches

this doesn't seem to be a real issue now that tc doesn't break makeCurrent
anymore, but if such a thing were to once again become a problem, at least
there will be handling for it

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: make init_mem_range() a public function

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: break out offset alignment calculation into helper

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: apply zink_resource_object::offset for memory flush/invalidates

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: queue v3.0

this uses a pointer to a batch state substruct for timeline tracking,
which provides a few nice benefits:
* explicit ability to detect unflushed batches (even on other contexts)
* the context doesn't need to have a "current" timeline id
* timeline (batch) ids can be distributed during submit, not when recording begins
* an abstracted api which can be more easily changed under the hood

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: remove no-longer-used resource helper functions

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: use batch usage api for resource helper function

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>

zink: add and use fencing functions which take batch usage structs

this hides the exact mechanics of fencing based on batch usage and makes
the usage a bit more explicit

Acked-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/11437>