include: Fix the PFN declarations to be pointers as they should

Broken by b5f9820d905a275bc01bbffa9b4927ec11286f8d back in 2016.

Signed-off-by: Sylvain Munaut <tnt@246tNt.com>
Acked-by: Antonio Gomes <antoniospg100@gmail.com>
Reviewed-by: Adam Jackson <ajax@redhat.com>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24573>

egl: bump extension string length

We've actually been over the 1000 char limit for a while, but we didn't
have a driver in CI that enabled enough extensions to notice it.

If all currently supported extensions are enabled, we need 1441 chars.
Let's bump the string to 2048 to have a little bit of margin.

Signed-off-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24781>

agx: Lower fquantize2f16

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24616>

nir: Lower fquantize2f16

Passes dEQP-VK.spirv_assembly.*opquantize*.

Unlike the DXIL lowering, this should correctly handle NaNs. (I belive Dozen has
a bug here that is masked by running constant folding early and poor CTS
coverage.) It is also faster than the DXIL lowering for hardware that supports
f2f16 conversions natively. It is not as good as a backend implementation that
could flush-to-zero in hardware... but for a debug instruction it should be more
than good enough.

It might be slightly better to multiply with 0.0 to get the appropriate zero,
but NIR really likes optimizing that out ...

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24616>

gtest: backport ansi color fix

Fixes ubsan runs for Mesa3D.

Adds prevents from returning
nullptr by choosing default color.

Upstream PR: https://github.com/google/googletest/pull/4322

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9404
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24783>

ci/freedreno: reuse freedreno_gl_file_list instead of re-definining it

Fixes: 9d442b459a43264c2899 ("ci/freedreno: handle disabling farm properly for each FD/Collabora farm")
Signed-off-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24779>

r300: add dEQP baseline for RV370 with forced swtcl

Reviewed-by: Filip Gawin <filip.gawin@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24769>

r300: don't abort on flow control when using draw for vs

It can handle it just fine. Around 250 dEQPs go from Skip to Pass.

Fixes: 1021e2b946b18739b65b575b0770e6158ba05592
Reviewed-by: Filip Gawin <filip.gawin@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24769>

r600: better tracking for vertex buffer emission

Fixes: 76725452 (gallium: move vertex stride to CSO)

Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24754>

r600: store the mask of buffers used by a vertex state

Fixes: 76725452 (gallium: move vertex stride to CSO)

Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24754>

radeonsi/vcn: Fix leaking fences in decode

Unref fence used in destroy.

Don't store the fence reference in picture->fence, instead
keep it in the driver. Because only the last fence will
now be valid, check the fence pointer in get_decoder_fence.

Reviewed-by: Leo Liu <leo.liu@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24677>

nir: Use nir_shader_intrinsic_pass() a few places

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24760>

nir: Fix metadata in nir_lower_is_helper_invocation

It does not preserve everything.  It adds and removes instructions and
even adds a variable.

Fixes: f17b41ab4f01 ("nir: add lowering pass for helperInvocationEXT()")
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24760>

nir: Add nir_shader_intrinsics_pass

Like instructions_pass but specialized to intrinsics. More ergnomic for this
extremely common case, and possibly a bit faster by avoiding the extra function
call on non-intrinsics.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Acked-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24760>

lavapipe: Advertise AMDX_shader_enqueue

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

lavapipe: Implement AMDX_shader_enqueue commands

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

lavapipe: Implement exec graph pipelines

Just a collection of compute shaders that can enqueue each other.

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

lavapipe: Add lvp_pipeline_type

A boolean is not enough to support exec graph pipelines.

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

spirv: Implement SPV_AMDX_shader_enqueue

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

spirv: Update headers and grammer JSON

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

nir: Add shader enqueue data structures and handling

There are two new variable modes:
- nir_var_mem_node_payload
- nir_var_mem_node_payload_in

Also add a few more intrinsics and some shader info.

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

vulkan Add enqueue entrypoint for CmdDispatchGraphAMDX

The generyted one doesn't copy deep enough.

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

vulkan: Allow beta extensions for physical device properties

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

vulkan: Allow beta extensions for physical device features

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

bin: Update spirv sources

Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24512>

radeonsi: modify algorithm of skipping holes of sparse bo

Modify current algorithm of skipping holes of sparse bo to cover the
following using situations:
1. The whole sparse buffer is uncommitted.
2. More than one page that in the tail of sparse buffer are uncommitted.

Signed-off-by: Julia Zhang <julia.zhang@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24535>

radeonsi: limit CP DMA to skip holes in sparse bo

CP DMA on gfx9 can't handle the hole in sparse buffer. The fix skip
sparse bo hole so that arb_sparse_buffer-buffer-data &&
arb_sparse_buffer-commit pass

Signed-off-by: Flora Cui <flora.cui@amd.com>
Signed-off-by: Julia Zhang <julia.zhang@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24535>

anv: emitting 3DSTATE_PRIMITIVE_REPLICATION is required on Gen12+

This change helps fix the following tests on future platforms:
  - func.multiview
  - dEQP-VK.fragment_shading_rate.renderpass2.monolithic.multiviewsrlayered.dynamic.attachment.noshaderrate.keep.replace.1x1.samples1.vs
  - anything else that uses multiview

Signed-off-by: Rohan Garg <rohan.garg@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24746>

dri: Use RGB internal formats for RGBX formats

These formats do not contain alpha channel, so their internal formats
should reflect that.

Fixes: bf576772 ("dri_util: add driImageFormatToSizedInternalGLFormat function")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9429
Signed-off-by: Sviatoslav Peleshko <sviatoslav.peleshko@globallogic.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24580>

ci/freedreno: another batch of Adreno 530 flakes

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24772>

radv: stop copying if VS or TES uses the InvocationID built-in

It's only allowed in TCS or GS which means the src shader stage
value is always FALSE.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24691>

radv: simplify declaring VS specific input SGPRs

stage/previous_stage are actually useless.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24691>

radv: remove unused param from radv_pipeline_init_multisample_state()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24691>

radv: remove radv_cmd_buffer::cached_vertex_formats

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24770>

radv: fix emitting TCS epilogs for GFX6-9

The number of SGPRs need to be adjusted.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24747>

radv: add missing comment about TCS_OFFCHIP_LAYOUT_LSHS_VERTEX_STRIDE

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24745>

radv: reduce TCS_OFFCHIP_LAYOUT_NUM_PATCHES to 6-bits

RADV clamps the number of tess patches to 40.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24745>

nir: Drop nir_push_if_src()

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24765>

nir: Drop nir_instr_rewrite_src()

Replace all its remaining users with nir_src_rewrite().

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Drop most uses if nir_instr_rewrite_src()

Generated by the following semantic patch:

    @@
    expression I, S, D;
    @@

    -nir_instr_rewrite_src(I, S, nir_src_for_ssa(D));
    +nir_src_rewrite(S, D);

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Drop nir_instr_rewrite_src_ssa()

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Drop most uses of nir_instr_rewrite_src_ssa()

Generated with the following semantic patch:

    @@
    expression I, S, D;
    @@

    -nir_instr_rewrite_src_ssa(I, S, D);
    +nir_src_rewrite(S, D);

Acked-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Drop nir_if_rewrite_condition()

Use nir_src_rewrite() instead.  In a couple of cases, we can even drop a
switch on whether or not it's an if source.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Add and use a nir_instr_init_src() helper

This helper exists for a very tiny set of use-cases but it's better to
have the helper live in nir.c than hand-roll it elsewhere.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Add a nir_instr_clear_src() helper and use it

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir/opt_undef: Don't rewrite a bcsel to mov

Technically, it's possible because bcsel has more sources than mov.
However, it's not worth the pain of trying to get it right.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Take a nir_def * in nir_phi_instr_add_src()

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

nir: Take a nir_def * in nir_tex_instr_add_src()

NIR bits were hand-typed.  Driver updates done through the following
semantic patch:

    @@
    expression T, ST, D;
    @@

    -nir_tex_instr_add_src(T, ST, nir_src_for_ssa(D));
    +nir_tex_instr_add_src(T, ST, D);

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24729>

ci/etnaviv: update ci expectation

Signed-off-by: Christian Gmeiner <cgmeiner@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24751>

nv50: limit max code uploads to 0x8000

I have no idea why a bigger size doesn't work, the hardware doesn't
complain, but it turns out that uploading big shaders still causes issues
with the old limit. *shrug*

Fixes: 7f63d2ebdbc ("nv50: fix code uploads bigger than 0x10000 bytes")
Signed-off-by: Karol Herbst <git@karolherbst.de>
Reviewed-by: M Henning <drawoc@darkrefraction.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24758>

iris,crocus: drop unnecessary DEBUG_NO_CCS/NO_HIZ checks

Now isl_surf_supports_ccs helper handles DEBUG_NO_CCS check and
isl_surf_get_hiz_surf handles DEBUG_NO_HIZ, so we don't
need to check it everywhere.

Signed-off-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24731>

anv,hasvk: drop unnecessary DEBUG_NO_CCS/NO_HIZ checks

Now isl_surf_supports_ccs helper handles DEBUG_NO_CCS check and
isl_surf_get_hiz_surf handles DEBUG_NO_HIZ, so we don't
need to check it everywhere.

Signed-off-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24731>

intel/isl: Enable INTEL_DEBUG=noccs/nohiz in ISL helpers

Let's enable INTEL_DEBUG=noccs in isl_surf_supports_ccs helper and
INTEL_DEBUG=nohiz in isl_surf_get_hiz_surf helper.

Signed-off-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24731>

disk_cache: Disable the "List" test for RO disk cache.

It uses a poll function that waits for a second hoping for another thread
to catch up, which is not a reliable way to do synchronization.  The test
has been spuriously failing merges on a regular basis recently.

This is issue #9222, which I'm leaving open until the author can fix the test.

Fixes: 3b69b67545b6 ("util/fossilize_db: add runtime RO foz db loading via FOZ_DBS_DYNAMIC_LIST")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24755>

zink: Fix one addicional case when running a compositor

Covers the case where the `dri2_init_screen` calls
`pipe_loader_create_screen_vk` directly and not sets
the device major and minor.

Signed-off-by: Igor Torrente <igor.torrente@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24678>

intel/compiler: Respect NIR_DEBUG_PRINT_INTERNAL for DEBUG_OPTIMIZER

If the NIR_DEBUG_PRINT_INTERNAL flag is not set, don't print debugging
information for internal shaders in INTEL_DEBUG=optimizer dumps.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24684>

ci/turnip: Add a660 VK coverage.

1/2 run pre-merge, and a half-hour full run for nightly.  Test status
looks very stable so far.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24737>

anv: refactor batch_set_preemption to use batch_emit_pipe_control

This makes it easier to hook workarounds for this pipe control.

Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Rohan Garg <rohan.garg@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24690>

radeonsi: don't use threadID.yz/blockID.yz for compute_blit if they're always 0

This can improve performance because fewer VGPRs and SGPRs need to be
initialized.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: don't use threadID.yz/blockID.yz for copy_image if those are always 0

This can improve performance because fewer VGPRs and SGPRs need to be
initialized.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: don't abort for descriptor failures, let the winsys handle it

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeon_winsys: add a ctx_set_sw_reset_status callback

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

winsys/amdgpu: rework how SW reset status is generated and reported

This adds a new helper amdgpu_ctx_set_sw_reset_status that sets the SW
status. The logic of which CS is reported as rejected is also changed
slightly, i.e. other contexts no longer affect the current context.

The helper will be exposed to radeonsi to allow reporting non-recoverable
allocation failures and skipped draws.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeon_winsys: move allow_context_lost from cs_create to ctx_create

to apply it to all command streams of each context.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: set PIPE_CONTEXT_LOSE_CONTEXT_ON_RESET on aux_context explicitly

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: rewrite how occlusion query precision is determined for performance

The precision of occlusion queries is determined from active queries.
Then the register programming is determined from the precision and other
states.

This has the effect that we no longer set PERFECT_ZPASS_COUNTS
for PIPE_QUERY_OCCLUSION_PREDICATE in some cases, resulting in higher
performance.

This also disables conservative occlusion queries for gfx11 because it's
not recommended with late Z, but detecting late Z vs early Z would be
more complicated, so just never use it, which results in better performance
with late Z.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: enable shader culling by default because it helps Viewperf

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: use num_patches_per_workgroup directly in si_get_ia_multi_vgt_param

We don't need to pass it via parameters.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: move GE_CNTL emission from si_draw into si_emit_vgt_pipeline_state

It doesn't depend on pipe_draw_info since pipe_context::set_patch_vertices
was added.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi/ci: update gfx11 failures

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: convert si_gfx_resources_add_all_to_bo_list to a state atom

We can do this as part of the state emit loop instead of having a separate
call in si_draw.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: merge si_upload_*_descriptors into si_emit_*_shader_pointers

This removes calling si_upload_graphics_shader_descriptors from si_draw
by moving the uploading into si_emit_graphics_shader_pointers.

Similar for compute.

si_upload_shader_descriptors used to set sctx->shader_pointers_dirty to
pass the mask to the emit function. Now, shader_pointers_dirty is both set
and consumed in si_emit_graphics_shader_pointers and si_emit_compute_-
shader_pointers, so the mask is passed via a local variable.

All places that set descriptors_dirty must now also dirty
the gfx_shader_pointers state for the descriptors to be uploaded.

All places that set bindless_descriptors_dirty must do the same and also
make the cache_flush state dirty because si_emit_graphics_shader_pointers
can now set cache flush flags (through si_upload_bindless_descriptors).

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: rename shader_pointers state -> gfx_shader_pointers

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: abort when failing to upload descriptors instead of skipping draws

This removes a jump from si_draw, and it's a prerequisite for the next
change, which will move uploading descriptors into a state emit function.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: remove render condition logic from si_draw by reordering atoms

If we reorder state atoms to emit the render condition after cache flushes,
it will automatically give us the behavior we want in si_draw.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: handle deferred cache flushes as a state (si_atom)

This allows us to remove a little bit of code from si_draw, and enable
removing more code in the future.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: add a simple version of si_pm4_emit_state for non-shader states

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: merge pm4 state and atom emit loops into one

This merges both loops in si_draw by tracking which pm4 states are dirty
using the state atom mechanism used for other states. pm4 states now have
to set their own emit function.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: move code around si_pm4_emit_state into si_pm4_emit_state

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: split direct pm4 emission from si_pm4_emit

si_pm4_emit_state will be changed.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: add index parameter into si_atom::emit

si_pm4_state will use si_atom, and both loops in si_emit_all_states will
be merged. This is a preparation for that because si_pm4_emit needs to know
the state index.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: specialize si_draw_rectangle using a C++ template

We have only 1 variant per gfx version except gfx10+, which have 2.
The motivation is to remove instructions from si_draw_vbo.

Code size before this commit:
    si_draw_vbo<GFX11, no tess, no GS, has NGG, has pairs>: 8616 bytes
    si_draw_rectangle: 272 bytes

Code size after this commit:
    si_draw_vbo<GFX11, no tess, no GS, has NGG, has pairs>: 8534 bytes
    si_draw_rectangle<GFX11, has NGG, has pairs>: 2295 bytes

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: always inline si_prefetch_shaders

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: remove the draw counter with primitive restart from the HUD

not used

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: remove unused check_mem parameter from si_sampler_view_add_buffer

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: add padding to si_resource to fix Viewperf2020/catiav5test1 perf

This is needed after the previous commit.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: remove splitting IBs that use too much memory

It was needed for r300, not so much for GCN/RDNA.
This reduces draw overhead.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: move si_emit_rasterizer_prim_state out of si_emit_all_states

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: move si_emit_spi_map into si_state_shaders.cpp

to reduce the amount of code in si_state_draw.cpp.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: move si_update/emit_tess_io_layout_state into si_state_shaders.cpp

to reduce the amount of code in si_state_draw.cpp.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: remove si_compute.h, move the contents into si_pipe.h

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: update obsolete comments about compiler queues

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: handle draw user SGPRs as tracked registers

instead of this custom code doing the same thing. This tracks changes to LS,
ES, and VS user SGPRs separately, so that we can skip more redundant register
changes when enabling/disabling GS and tess. The perf impact should be neutral.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: cosmetic changes to radeon_opt_* macros

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: restructure the loop for non-indexed multi draws

Have one loop for increment_draw_id and another loop without it.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

radeonsi: turn sh_base[PIPE_SHADER_VERTEX] into a constant in emit_draw_packets

HAS_TESS will also be used in the next commit

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24732>

spirv: Re-emit constants at their uses

Right now, spirv_to_nir places all constants at the top of the function
and has a hash table to de-duplicate them.  This change drops the hash
table and starts re-emitting constants more-or-less at their uses.  This
is more consistent with what we do in GLSL -> NIR translation.  It is,
however, a change to SPIR-V -> NIR translation which will likely affect
other optimizations in unexpected ways so it should be evaluated
separately.

This gives some good saves for spills/fills for Intel, without causing
any significant regressions on RADV, because it is using the
nir_opt_reuse_constants() pass.  In the long run that should be superseded
by some form of GCM.

```
Intel TGL results for Intel fossils:

Totals:
Instrs: 183287977 -> 183269431 (-0.01%); split: -0.07%, +0.06%
Cycles: 18224600804 -> 18223114114 (-0.01%); split: -0.16%, +0.15%
Spill count: 111031 -> 108377 (-2.39%); split: -2.45%, +0.06%
Fill count: 221781 -> 216479 (-2.39%); split: -2.39%, +0.00%
Scratch Memory Size: 4355072 -> 4180992 (-4.00%); split: -5.31%, +1.32%

Totals from 43684 (6.54% of 667704) affected shaders:
Instrs: 38289482 -> 38270936 (-0.05%); split: -0.33%, +0.28%
Cycles: 7166415272 -> 7164928582 (-0.02%); split: -0.40%, +0.38%
Spill count: 93747 -> 91093 (-2.83%); split: -2.90%, +0.07%
Fill count: 190943 -> 185641 (-2.78%); split: -2.78%, +0.00%
Scratch Memory Size: 3127296 -> 2953216 (-5.57%); split: -7.40%, +1.83%
```

```
RADV GFX1100 results for radv fossils:

Totals:
Instrs: 71623708 -> 71624667 (+0.00%); split: -0.00%, +0.01%
CodeSize: 369324312 -> 369334744 (+0.00%); split: -0.00%, +0.01%
SpillSGPRs: 13586 -> 13582 (-0.03%)
SpillVGPRs: 911 -> 910 (-0.11%); split: -0.55%, +0.44%
Latency: 632887831 -> 632880378 (-0.00%); split: -0.00%, +0.00%
InvThroughput: 81674859 -> 81676684 (+0.00%); split: -0.00%, +0.01%
VClause: 1273752 -> 1273727 (-0.00%); split: -0.00%, +0.00%
SClause: 2409593 -> 2409078 (-0.02%); split: -0.02%, +0.00%
Copies: 4063579 -> 4064425 (+0.02%); split: -0.05%, +0.07%
Branches: 1196723 -> 1196720 (-0.00%); split: -0.00%, +0.00%

Totals from 16244 (12.17% of 133461) affected shaders:
Instrs: 31116807 -> 31117766 (+0.00%); split: -0.01%, +0.01%
CodeSize: 160316656 -> 160327088 (+0.01%); split: -0.01%, +0.01%
SpillSGPRs: 12270 -> 12266 (-0.03%)
SpillVGPRs: 835 -> 834 (-0.12%); split: -0.60%, +0.48%
Latency: 344388549 -> 344381096 (-0.00%); split: -0.01%, +0.00%
InvThroughput: 43043761 -> 43045586 (+0.00%); split: -0.01%, +0.01%
VClause: 433221 -> 433196 (-0.01%); split: -0.01%, +0.00%
SClause: 900825 -> 900310 (-0.06%); split: -0.06%, +0.00%
Copies: 1989000 -> 1989846 (+0.04%); split: -0.11%, +0.15%
Branches: 676625 -> 676622 (-0.00%); split: -0.00%, +0.00%
```

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5282>

radv: Use nir_opt_reuse_constants()

Right now, this won't change much the shaders since most of the NIR constants
are reused in the NIR generated by SPIR-V, but will make radv resilient to
when that behavior change.

```
RADV GFX1100 results for radv fossils:

Totals:
Instrs: 71623585 -> 71623708 (+0.00%); split: -0.00%, +0.00%
CodeSize: 369326156 -> 369324312 (-0.00%); split: -0.00%, +0.00%
SpillSGPRs: 13576 -> 13586 (+0.07%)
Latency: 632889681 -> 632887831 (-0.00%); split: -0.00%, +0.00%
InvThroughput: 81674616 -> 81674859 (+0.00%); split: -0.00%, +0.00%
SClause: 2409601 -> 2409593 (-0.00%); split: -0.00%, +0.00%
Copies: 4063438 -> 4063579 (+0.00%); split: -0.00%, +0.01%
Branches: 1196703 -> 1196723 (+0.00%)
PreSGPRs: 4242897 -> 4243061 (+0.00%); split: -0.00%, +0.00%
PreVGPRs: 3926739 -> 3926742 (+0.00%)

Totals from 217 (0.16% of 133461) affected shaders:
Instrs: 353567 -> 353690 (+0.03%); split: -0.04%, +0.07%
CodeSize: 1790200 -> 1788356 (-0.10%); split: -0.15%, +0.04%
SpillSGPRs: 8 -> 18 (+125.00%)
Latency: 5152817 -> 5150967 (-0.04%); split: -0.05%, +0.01%
InvThroughput: 664273 -> 664516 (+0.04%); split: -0.03%, +0.06%
SClause: 10164 -> 10156 (-0.08%); split: -0.10%, +0.02%
Copies: 24225 -> 24366 (+0.58%); split: -0.32%, +0.90%
Branches: 7116 -> 7136 (+0.28%)
PreSGPRs: 13351 -> 13515 (+1.23%); split: -0.16%, +1.39%
PreVGPRs: 11583 -> 11586 (+0.03%)
```

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5282>

nir: Add nir_opt_reuse_constants()

Currently SPIR-V does pull all the NIR constants it creates into the
first block, but we plan to change that behavior to let those constants
be defined as they are used.

This pass was written to provide a fallback to the old behavior, it will
be used for radv to avoid regressions when performing the SPIR-V change.

Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5282>

r600: use correct cso pointer for fetch shader

Fixes: 76725452 (gallium: move vertex stride to CSO)

Related: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9567

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24728>