nir/builder: Add nir_replicate helper

Splat a scalar to all components of a vector.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23259>

radeonsi: return kernel queried video capability for HEVC and JPEG

Query and return the values obtained from kernel for VCN_1 and above.
Earlier the HEVC and JPEG capabilities were returned based on
pre-defined values.

Signed-off-by: Veerabadhran Gopalakrishnan <veerabadhran.gopalakrishnan@amd.com>
Reviewed-by: Thong Thai <thong.thai@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23067>

radeonsi: return kernel queried video capability for HEVC and JPEG

Query and return the values obtained from kernel for VCN_1 and above.
Earlier the HEVC and JPEG capabilities were returned based on
pre-defined values.

Signed-off-by: Veerabadhran Gopalakrishnan <veerabadhran.gopalakrishnan@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23067>

gallivm: Fix gather/scatter types for newer llvm

The types changed with opaque pointer support.

Fixes a bunch of lavapipe regressions.
Cc: mesa-stable
Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23293>

docs/calendar: add 23.2 branchpoint and release candidates

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23205>

docs: update calendar for 23.0.4

At this point I'm calling 23.0 done. Please use 23.1 for future updates

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23325>

docs: Add sha256 sum for 23.0.4

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23325>

docs: add release notes for 23.0.4

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23325>

microsoft/compiler: Use image formats to determine texture types

Fixes some tests when bindless is disabled, where the image format is
R32, we do atomics on it, but we didn't set the "typed UAV load with
additional formats" feature bit because when we loaded from it, we
only loaded one component. Since the image format on the DXIL side
was declared as U32x4, the DXIL validator said that we should have.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23266>

spirv2dxil: Assign formats to image vars before lowering to bindless

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23266>

microsoft/compiler: Add a pass to assign image formats based on number of components

For loads/stores without formats, let's guess one based on how it's used.
The actual format doesn't matter, we just want to use it for the number
of components it has.

Also copy image formats from variables to intrinsics, to ensure that
deref-based intrinsics have formats assigned and lowered intrinsics
are up to date.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23266>

microsoft/compiler: Enable emitting type info for textures with <4 comps

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23266>

radv/amdgpu: Report 48-bit VAs in bo logs.

More ergonomic when correlating with page fault addresses.

Signed-off-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23256>

radv: fix gl_SampleMaskIn for sample shading

When sample shading, we need

  gl_SampleMaskIn = SampleCoverage & (PsIterMask << gl_SampleID);

Add a new shader arg, ps_iter_mask, to pass PsIterMask to ps.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23265>

ac, radeonsi: add and use ac_get_ps_iter_mask

It is more natural for ac_get_ps_iter_mask to take sample count.
Replace samplemask_log_ps_iter by ps_iter_smples in
ac_nir_lower_ps_options accordingly.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23265>

aco: fix alignment check in emit_load

align_offset already takes const_offset into consideration.  Remove the
adjustment.

Fixes: 542733dbbf7 ("aco: add emit_load helper")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9097
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23242>

anv: remove unused functions

I thought I removed those, it seems my rebase got screwed up :(

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 64f20cec28 ("anv: prepare image/buffer views for non indirect descriptors")
Acked-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23317>

anv: add support for VK_EXT_dynamic_rendering_unused_attachments

Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23294>

zink: do not lower line-smooth for non-lines

We used to do this correctly, but it seems this accidentally got
dropped.

Fixes: d80a35a7f76 ("zink: unified `zink_set_primitive_emulation_keys` and `zink_create_primitive_emulation_gs`")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23316>

zink: compute correct location for line-smooth gs

The GS and the FS needs to agree on the driver_location. But we just
used the num_outputs variable for the GS instead of matching the logic
from lower_aaline_instr in nir_draw_helpers.c.

This does that, but cleans up our copy slightly to avoid computing the
needless location, as well as using unsigned values.

This used to *mostly* work before, but only because we were lucky and
not too much crazy stuff went on with the inputs / outputs in
smooth-line cases.

Fixes: edecb66b018 ("nir: avoid generating conflicting output variables")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23316>

zink: keep gl46_optimal extensions/features sorted

These are mostly sorted, which makes it easy to know where to insert new
stuff. Let's make them completely sorted.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23316>

zink: explicitly avoid ci errors due to unrecognized extensions in VVL

lavapipe still had this from shader object, but this should always be in
place for all drivers so that old VVL doesn't block merges

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23315>

zink: update profiles schema

We technically need the 251 version, but the schema for that version
is not released yet, so this is the best we can do.

See https://github.com/KhronosGroup/Vulkan-Profiles/issues/437

This gets rid of some JSON schema-validation warnings.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23308>

nir/opt_if: use nir_alu_instr_is_comparison directly

Since 2d6233d0 ("nir: Check all sizes in nir_alu_instr_is_comparison"),
nir_alu_instr_is_comparison already returns true for comparisons with 32bit
result.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23287>

aco: improve printing of s_delay_alu

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23213>

aco: insert s_delay_alu on the linear CFG

fossil-db (gfx1100):
Totals from 10498 (7.87% of 133428) affected shaders:
Instrs: 22274711 -> 22277717 (+0.01%); split: -0.01%, +0.03%
CodeSize: 114557040 -> 114569064 (+0.01%); split: -0.01%, +0.02%
Latency: 236505186 -> 236497338 (-0.00%); split: -0.00%, +0.00%
InvThroughput: 33425052 -> 33423876 (-0.00%); split: -0.00%, +0.00%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23213>

aco: use pass_flags to recover s_delay_alu cycles

This is simpler and more accurate.

fossil-db (gfx1100):
Totals from 11678 (8.75% of 133428) affected shaders:
Instrs: 25448655 -> 25436028 (-0.05%)
CodeSize: 130364728 -> 130314220 (-0.04%)
Latency: 325247603 -> 325231064 (-0.01%); split: -0.01%, +0.00%
InvThroughput: 45901166 -> 45900022 (-0.00%); split: -0.00%, +0.00%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23213>

aco: fix update_alu(clear=true) for exports

For:
   v_mov_b32_e32 v0, 1.0
   exp mrtz v0, off, off, off
we should completely remove the ALU entry before creating the EXP's WaR entry for v0.
Otherwise, the two will be combined into an entry which will wait for
expcnt(0) for later uses of v0.

gen_alu() should also be before gen(), since gen_alu() performs the clear
while gen() creates the WaR entry.

fossil-db (gfx1100):
Totals from 3589 (2.69% of 133428) affected shaders:
Instrs: 5591041 -> 5589047 (-0.04%); split: -0.04%, +0.00%
CodeSize: 28580840 -> 28572864 (-0.03%); split: -0.03%, +0.00%
Latency: 65427923 -> 65427543 (-0.00%); split: -0.00%, +0.00%
InvThroughput: 11109079 -> 11109065 (-0.00%); split: -0.00%, +0.00%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23213>

d3d12: HEVC Encode - Fix num_subregions_per_scanline rounding

num_subregions_per_scanline need to be round up

Signed-off-by: Teng, Jin Chung <jin.chung.teng@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23247>

pvr: Fix page faults in occlusion query tests

This does not fix the tests completely, but does allow them to run to
completion and fail "properly".

Also contains a few trivial bugfixes in the same codepath.

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reported-by: James Glanville <james.glanville@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23100>

pvr: Handle barrier load and store flags.

This commit adds handling for {s,z}loaden and {s,z}storeen to
control loading from and storing to the stencil and depth buffer.

This commit also addressed the FIXMEs around barrier_{load,store}
which control the {s,z}{load,store}en.

Signed-off-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Reviewed-by: Frank Binns <frank.binns@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20487>

pvr: Reorder execution in pvr_cmd_buffer_end_sub_cmd()

This allows sub_cmd->job.run_frag to be setup before calling
pvr_sub_cmd_gfx_requires_split_submit().

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reported-by: James Glanville <james.glanville@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23125>

pvr: Fix out of range stream errors for geometry-only jobs on pvrsrvkm

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reported-by: James Glanville <james.glanville@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23125>

pvr: Do not free deferred pvr_transfer_cmd instances

Deferred pvr_transfer_cmd instances are allocated from a dyn_array on
the owning pvr_cmd_buffer and must not be freed directly.

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reported-by: James Glanville <james.glanville@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23125>

pvr: Rename shadowing loop variable in pvr_add_deferred_rta_clear()

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23125>

pvr: Use correct surface for deferred RTA clear

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reported-by: James Glanville <james.glanville@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23125>

pvr: Correct error flow in pvr_compute_pipeline_compile()

Fixes:
  dEQP-VK.api.object_management.alloc_callback_fail.compute_pipeline

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23130>

pvr: Correct error flow in pvr_graphics_pipeline_compile()

Fixes:
  dEQP-VK.api.object_management.alloc_callback_fail.graphics_pipeline

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23130>

pvr: Fix memory leaks on realloc failure in pvr_pipeline.c

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23130>

pvr: Fix allocation scopes in vkCreateRenderPass2() code path

These previously COMMAND scoped allocations are stored on the
VkRenderPass and must therefore be OBJECT scoped.

Fixes: dEQP-VK.api.object_management.single_alloc_callbacks.render_pass

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Karmjit Mahil <Karmjit.Mahil@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23130>

v3d: only warn about bining sync for indirect draw once

Reviewed-by: Juan A. Suarez <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23306>

anv: override vendorID for Cyberpunk 2077

A recent update to Cyberpunk 2077 enables XeSS code for Intel GPUs
which is causing the game to crash in the XeSS libraries.  As a
temporary work around, stop identifying as Intel for Cyberpunk so
XeSS falls back to the cross-vendor path.

References: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8860
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23271>

anv: enable direct descriptors on platforms with extended bindless offset

Aka. DG2

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: ensure descriptor addresses are used with bindless stages

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: descriptor binding for direct descriptors

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: bring back the max number of sets to 8

Not sure we bumped it to 32 for the right reasons. This generates more
push constant data and because we're not tighly packing our push
constant data this can generate more register pressure.

We could tightly pack things at the cost of some CPU cycles but only
for some stages. RT stages would have to retain the current "sparse"
version of push constants.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: add direct descriptor support to apply_layout

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: track descriptor data size

descriptor_stride includes multiple plane size, this new field tracks
just the data of one plane.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: simplify ycbcr bti computations

To make BTI indexing simpler with ycbcr samplers, stop doing packing
calculations in the apply_layout. We'll insert NULL bindings for the
few ycbcr cases where it's needed.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: implement binding table emission for direct descriptors

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: factor out dynamic buffer bti emission

No functional change. Will reuse in the followup commit.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: handle null surface in the binding table with direct descriptors

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: add helpers to build pipeline bindings

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: add support for direct descriptor in allocation/writes

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: prepare image/buffer views for non indirect descriptors

When in direct descriptor mode, the descriptor pool buffers will hold
surface states directly. We won't allocate surface states in image &
buffer views.

Instead views will hold a packed RENDER_SURFACE_STATE ready to copied
into the descriptor buffers.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: bound load descriptor mem better

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

nir: expose a couple of address format add helpers

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: add a pass to partially lower resource_intel

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: new structure to hold surface states

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: reduce push constant size for descriptor sets

Now that descriptor sets are located a in a 1Gb area, we can avoid
storing the whole address to the descriptor and add the base address
of the area to a 32bit offset.

Replay a bunch of fossils with this and changes not really significant
one way or another :

Totals:
Instrs: 9278246 -> 9277148 (-0.01%); split: -0.01%, +0.00%
Cycles: 3547598421 -> 3547579435 (-0.00%); split: -0.00%, +0.00%

Totals from 353 (1.14% of 31021) affected shaders:
Instrs: 581546 -> 580448 (-0.19%); split: -0.23%, +0.04%
Cycles: 25885422 -> 25866436 (-0.07%); split: -0.31%, +0.24%

No difference on send messages or spills/fills.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: create a pool for indirect descriptors

We'll use the fact that the pool is aligned to 4Gb to limit the amount
of address computations to build the address in the shaders.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: introduce a new descriptor set layout type

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: add an option for using indirect descriptors

This is the default for now. It needs to be part the pipeline hashing
as we will allow this to be tweaked per application.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

docs/anv: some binding table explanations

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: toggle extended bindless surface state on Gfx12.5+

We bump the max surfaces to ~16 million instead of ~1 million on
Gfx9-12. We could do more but that'll come later.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: increase workaround BO so that we can hold a full 4Kb page of 0s

At the beginning of the buffer is located the driver identifier for
error states.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: move pipeline active_stages to common structure

And fill it out for all types of pipelines.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: track pipeline in anv_cmd_pipeline_state

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: bail flush_gfx_state when not gfx push constant is dirty

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: remove incorrect ifdef

This is a leftover from a previous fix attempt. We don't need this.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: fix null descriptor handling with A64 messages

global load/store (or A64 messages) need the NIR bound checking which
is enabled by "robust" behavior even when robust behavior is disabled.

Many thanks to Christopher Snowhill for pointing out the pushed
constant related issue with the initial version of this patch.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: mesa-stable
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: remove unused define

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: try to rematerialize surface computation code

This helps a lot with accessing surface handles in control flow. Our
resource_intel intrinsic has a non_uniform flag, in which case we
cannot apply this optimization. But in uniform cases, this is just a
massive win. We drop all kind of pipeline stalls due to
find_live_channel. We also reduce register pressure by doing the
surface handle computation in a single GRF (instead of 2 or 4).

There are some regressions in max dispatch width but those I think are
only on SIMD32 and due to the current heuristic disabling it after
throughput comparison with SIMD16. We know this heuristic is not
perfect, it should probably be updated in another change.

Here are some stats (all titles seem to have similar gains) :

 PERCENTAGE DELTAS    Shaders   Instrs    Cycles  Subgroup size Send messages Spill count Fill count Scratch Memory Size Max live registers Max dispatch width
 red_dead_redemption2 5860     -36.80%    -5.67%      +0.77%        +0.06%      -81.26%     -79.16%        -70.62%             -8.63%             -6.93%
 ---------------------------------------------------------------------------------------------------------------------------------------------------------------
 All affected         4716     -37.29%    -5.67%      +0.95%        +0.07%      -81.26%     -79.16%        -70.62%             -9.15%             -8.47%
 ---------------------------------------------------------------------------------------------------------------------------------------------------------------
 Total                5860     -36.80%    -5.67%      +0.77%        +0.06%      -81.26%     -79.16%        -70.62%             -8.63%             -6.93%

 PERCENTAGE DELTAS          Shaders   Instrs    Cycles  Subgroup size Send messages Spill count Fill count Scratch Memory Size Max live registers Max dispatch width
 rise_of_the_tomb_raider_g2 12010    -37.19%   -22.12%      +0.01%        +0.00%      -99.01%     -99.14%        -98.65%             -7.62%             -4.96%
 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 All affected               11732    -37.27%   -22.14%      +0.01%        +0.00%      -99.01%     -99.14%        -98.65%             -7.67%             -5.11%
 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Total                      12010    -37.19%   -22.12%      +0.01%        +0.00%      -99.01%     -99.14%        -98.65%             -7.62%             -4.96%

 PERCENTAGE DELTAS    Shaders   Instrs    Cycles  Spill count Fill count Scratch Memory Size Max live registers Max dispatch width
 total_war_warhammer2 462      -27.45%   -12.42%    -82.35%     -88.46%        -66.67%             -5.52%             -5.62%
 -----------------------------------------------------------------------------------------------------------------------------------
 All affected         335      -28.31%   -12.77%    -82.35%     -88.46%        -66.67%             -6.25%             -7.24%
 -----------------------------------------------------------------------------------------------------------------------------------
 Total                462      -27.45%   -12.42%    -82.35%     -88.46%        -66.67%             -5.52%             -5.62%

 PERCENTAGE DELTAS Shaders   Instrs    Cycles  Subgroup size Send messages Spill count Fill count Scratch Memory Size Max live registers Max dispatch width
 witcher_3_dxvk_g2 1049     -36.94%   -57.82%      +0.06%        +0.01%      -98.52%     -97.29%        -98.10%             -7.81%             -1.00%
 ------------------------------------------------------------------------------------------------------------------------------------------------------------
 All affected      693      -41.93%   -58.45%      +0.09%        +0.01%      -98.52%     -97.29%        -98.10%             -10.25%            -1.33%
 ------------------------------------------------------------------------------------------------------------------------------------------------------------
 Total             1049     -36.94%   -57.82%      +0.06%        +0.01%      -98.52%     -97.29%        -98.10%             -7.81%             -1.00%

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: enable uniform block accesses through bindless heap

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: enable bindless sampler state offsets

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: enable extended bindless surface offset

Gives use 4Gb of bindless surface state on Gfx12.5+ instead of 64Mb.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: enable get_buffer_size on bindless heap

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: enable UBO accesses through bindless heap

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: enable SSBO accesses through the bindless heap

Using the information coming from surface_index_intel, we can tell
whether we should use the BTI or bindless heap for a particular SSBO
access.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: keep track of new resource_intel information

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: teach ubo range analysis pass about resource_intel

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: add a pass to move resource_intel closer to user

Non uniform lower can insert read_first_invocation on the result of
resource_intel. We want to keep that intrinsic directly in front of
the user (load_ubo/load_ssbo/load_image/etc...)

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

nir/opt_gcm: allow resource_intel to be moved anywhere

The resouce_intel intrinsic doesn't not result in an actual
instruction, it's just a wrapper around another value, usually a
load_const.

Allowing this intrinsic to be moved anywhere means it's going to be
closer to the value it wraps, enabling opt_gcm to move a load_ubo
using this resource_intel.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

nir: teach nir_chase_binding about resource_intel

It's needed to have opt_gcm work properly.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

nir: add a new intrinsic to describe resources accessed on intel

Intel HW has multiple ways to access resources like UBO/SSBO/images :

   - binding tables : a small ~240 heap of surfaces

   - bindless surfaces : a 64Mb heap of surfaces up to Gfx12+, 4Gb on Gfx12.5+

   - surfaces : a 4Gb heap on Gfx12.5+ (mostly unused at the moment,
     only available through the LSC)

For samplers, we have 2 options since Gfx11+ :

   - samplers indexed from the Dynamic State Heap (4Gb)

   - samplers indexed from the Bindless Sampler Heap (4Gb)

Additionally our whole push constant promotion mechanism is based
around binding table indices. This is problematic if you want to also
promote to push constants things that would be accessed through the
bindless heap.

To solve this issue, we introduce a new intrinsic that will cary a
block index that is not based off the binding table index nor the
bindless table offset.

We will also use this intrinsic to identify whether the buffer/surface
index in load_ubo/load_ssbo/store_ssbo/etc... is relative to the
binding table or the bindless heap.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

nir/lower_shader_calls: add ability to force remat of instructions

Some instruction we would like to keep around because they carry
additional information in their indices.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: lower get_buffer_size like other logical sends

This will also enable the use of the bindless heap.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

intel/fs: reuse descriptor helper

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: fix push range for descriptor offsets

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 379b9bb7b0 ("anv: Support fetching descriptor addresses from push constants")
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv: update internal address space to have 4Gb of dynamic state

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21645>

anv/video: move format properties to outarray.

This should be using the helper code.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23304>

radv: align video images internal width/height inside the driver.

Due to how the decoders work, they will write garbage data into
the padding, and later using the image for sampling with linear
images will use the garbage to create broken results. Let the
user specify the image size and align it up in the driver, so
sampling of the image later has the correct w/h.

cc: mesa-stable

Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

vk/video: add a common function to get block alignments for profiles

This is to be used by drivers for internal image alignments.

This just adds a common profile to alignment helper.

Cc: mesa-stable
Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

radv/video: fix physical device format property count.

This was returning bad values

Fixes: db62c38091a3 ("radv: add vcn h265 decode.")
Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

radv/video: add debug flag to enable dpb image array on newer GPUs.

This is useful to test the paths on newer GPUs that work on older GPUs.

Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

radv/video: fix some whitespace.

this just removes some TABs.

Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

radv/video: rework stream handle generation.

This shouldn't change anything, except move some calcs to an
earlier spot to avoid redoing them

Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

radv/video: add missing offset to the dpb binding.

This doesn't affect anything I've seen yet.

Fixes: 3e2c768aa860 ("radv/vcn: enable dynamic dpb tier 2 for h264/h265 on navi21+")
Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

radv/video: fix h264/265 dpb usage.

This seems to be the best compromise I can come up with so far.

I can't figure out to get the tier2 programming to match between
264 and 265, maybe they are just programmed different here, good
old firmware.

Fixes: 1693c03a3963 ("radv/video: add initial h264 decoder for VCN")
Reviewed-by: Lynne <dev@lynne.ee>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23227>

mesa: fix refcnt imbalance related to _mesa_delete_semaphore_object()

Indeed, the fence reference was not freed.

For instance, this issue is triggered with
"piglit/bin/ext_external_objects-vk-semaphores-2 -auto -fbo"
while setting GALLIUM_REFCNT_LOG=refcnt.log.

Fixes: 7b6cd912a59a ("mesa/st: get rid of ST_CALLOC_STRUCT use CALLOC_STRUCT")
Signed-off-by: Patrick Lerda <patrick9876@free.fr>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23296>