radv: emulate GEOMETRY_SHADER_INVOCATIONS query on RDNA1-2

The number of geometry shader invocations is correctly counted by the
hardware for both NGG and the legacy GS path but it increments for
NGG VS/TES because they are merged with GS, but it shouldn't. Fix this
by emulating the number of geometry shader invocations.

This fixes piglit/bin/arb_query_buffer_object-qbo and recent
dEQP-VK.query_pool.statistics_query.gs_invocations_no_gs.* failures
with NGG.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24231>

radv: implement nir_intrinsic_atomic_add_gs_invocation_count_amd

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24231>

radv: rename RADV_SHADER_QUERY_PIPELINE_STAT_OFFSET

To RADV_SHADER_QUERY_GS_PRIM_EMIT_OFFSET.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24231>

radv: simplify the NGG vs legacy pipelinestat query path

NGG is enabled by default on RDNA1-2 but the driver might fallback to
legacy GS for some reasons, like XFB. On these generations, the number
of generated primitives by GS needs to be emulated from the NGG shader
because the hw doesn't increment the related pipelinestat counter.

In order to support NGG and legacy GS with that query (remember that
we can't know pipelines when starting/ending queries), we used to
reserve 2x 64-bit counters to store the GDS results, and the results
were accumulated.

Now that legacy GS also uses GDS counters, we can simplify this path
and overwrite the pipelinestat counter directly instead of having two
separate counters.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24231>

radv: enable pipelinestat query emulation for legacy GS

This will allow us to fix a bug with the number of geometry shader
invocations which increase for NGG VS or TES but shouldn't. And also
for simplifying the NGG vs legacy query path.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24231>

radv: declare the shader query user SGPR for emulating GS counters

This user SGPR is only declared on chips that support NGG but might
fallback to legacy GS for some reasons, like XFB. It will be used to
emulate GS counters from shaders.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24231>

radv: rename NGG query state to be more generic

To use emulated GS counters for legacy GS.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24231>

freedreno/regs: python does not need ';'

Signed-off-by: Christian Gmeiner <cgmeiner@igalia.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24330>

freedreno/regs: remove dead code

Signed-off-by: Christian Gmeiner <cgmeiner@igalia.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24330>

freedreno/regs: remove not used variable

Signed-off-by: Christian Gmeiner <cgmeiner@igalia.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24330>

freedreno/regs: remove double assignment of self.current_domain

Signed-off-by: Christian Gmeiner <cgmeiner@igalia.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24330>

meson: report with_glvnd in summary

GLVND *really* kinda belongs in *both* GLX and EGL, but it feels silly
to repeat the same setting. So let's just report it under the GL
section, as that's generic enough to apply to both of other sections.

Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24317>

anv: wire image sparse loads

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23882>

intel/nir: add lower for sparse images & textures

We have to lower images into image load + sampler residency.

There is also a restriction on sampler access with a compare, lower
those as 2 sampler instructions to meet the restriction.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23882>

intel/nir: handle image_sparse_load in storage format lowering

The last component of sparse load is the residency data. We don't want
to touch/convert that value with the format lowering.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23882>

intel/fs: add support for sparse accesses

Purely from the backend point of view it's just an additional
parameter to sampler messages.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23882>

gallium: unbreak kmsro/freedreno case

In case the case of kmsro and freedreno driver, freedreno will fail to
with the xmlconfig errors as the kmsro declaration doesn't have defaults
for the freedreno options. Instead of directly using v3d_driconf for
kmsro, add native kmsro_driconf, which optionally includes v3d and
freedreno options.

Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24218>

gallium: move kmsro definition to the bottom of the file

The kmsro (in theory) can be using any other driver. In order to
simplify handling of driver public headers, move kmsro definition to the
bottom of the drm_helper.h

Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@linaro.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24218>

ci/kernel: add amd patch to prevent crashes when starting X

See: https://gitlab.freedesktop.org/drm/amd/-/issues/2669

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9402

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24338>

nir: Add a backend_flags field to nir_tex_instr

In 9ffd00bcf133 ("nir_to_tgsi: Pack our tex coords into vec4
nir_tex_src_backend[12]"), Emma added a pair of back-end sources to
nir_tex_instr to allow complex lowering to be done in NIR.  This adds a
tiny bit more hw-specific back-end information that a NIR lowering pass
can communicate to the back-end compiler.

While the opcode contains most of the information needed, some thing
such as the presence of offsets is currently only communicated via the
presence of specific source types in the source list.  This information
is gone when the texture instruction is lowered to back-end sources.
Adding a backend_flags field fixes this by allowing the lowering pass to
communicate a small amount of side-band information if needed.

Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22303>

r600: don't use sb disasm to disassamble copy shader

We will remove sb at one point, so drop its use.

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600: don't check possible size of ALU CF

The scheduler and sfn assembler already o this.

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600: Assert when backend wants to create a new ALU CF

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600: Explicitly force new CF in gs copy shader

With that we can assert on ALU CF mission in the assembler

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: Schedule AR uses befor possible groups

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: rework checks for ALU CF emission

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: Splizt ALU blocks in scheduler to fit into 128 slots

With that the backend doesn't split these ALU CFs any more.

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: on Cayman loading an index register needs only one slot

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: make remaining slots a signed value

Needed to check the limits.

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: Add flags to check whether a group starts CF and can do that

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: Add method to convert to AluGroup directly

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: override slot count for IfInstr

We have one slot for the predicate (todo fix this value)

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: Fix typo with block type

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: set  block sizes based on chip class

Be conservative with the ALU slots and the VTX slots.

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: Always check arrays writes before allowing copy propagation

Also propaate extra dependencies when an indirect load is propagated

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

r600/sfn: Fix use of multiple IDX with kcache

Currently we don't properly support using he two IDX registers in the
same ALU CF, so work around this by enforcing a new CF if both indices
are used.

Fixes: d21054b4bc92a1a9240841dca719f81a142fd5cc
    r600/sfn: Add pass to split addess and index register loads

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24297>

Revert "ci/farms: always compare the code against main repository"

Unsustainable solution for the fork due to need keep main repository
up-date there.

This reverts commit 7b29ae557ab9e86d75952c580afd87b6cb985373.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24337>

tu: Disable transformFeedbackPreservesProvokingVertex

Our implementation only preserves the provoking vertex if it is last,
and always preserving it would be more draw-time validation because the
vertex ordering is tied up with the tessellation domain origin. It turns
out we didn't notice this because the tests aren't good enough to catch
the issue.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24304>

ir3: Handle GS stream "mixing" with non-point output primitives

This fixes some new Vulkan CTS tests that do this.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24304>

aco: insert s_nop before VGPR deallocation

A hazard requires this, apparently.

https://reviews.llvm.org/D155681

fossil-db (gfx1100):
Totals from 23175 (17.36% of 133461) affected shaders:
Instrs: 41240100 -> 41263275 (+0.06%)
CodeSize: 211858524 -> 211951224 (+0.04%)
Latency: 469738570 -> 469738576 (+0.00%)
InvThroughput: 58013998 -> 58013999 (+0.00%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24320>

ci/farms: always compare the code against main repository

`rules:changes:compare_to` resolved firstly pushed branch pipelines,
which always evaluated `rules:changes` as true which breaks the workflow

Since we now explicitely say, that we compare against `main` repository,
GitLab can evaluate against real changes.

Fixes: 79f7882fc604 ("ci: add quirk for GitLab assuming changes is always true for scheduled runs")

Reviewed-by: Eric Engestrom <eric@igalia.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24305>

anv: simplify buffer address+size loads from descriptor buffer

Only found a couple titles that have been helped by this :

 PERCENTAGE DELTAS Shaders   Instrs    Cycles
 cyberpunk_2077    10388     -0.00%    -0.00%
 -----------------------------------------------
 All affected      1         -2.24%    -0.39%
 -----------------------------------------------
 Total             10388     -0.00%    -0.00%

 PERCENTAGE DELTAS    Shaders   Instrs    Cycles
 red_dead_redemption2 5949      -0.10%    -0.00%
 --------------------------------------------------
 All affected         111       -0.74%    -0.14%
 --------------------------------------------------
 Total                5949      -0.10%    -0.00%

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23318>

isl: add ability to store buffer size in unused RENDER_SURFACE_STATE fields

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23318>

intel/fs: add more UNDEFs around SEND messages

lower_find_live_channel() in particular is used a lot in control flow
to find the live channel for the surface/sampler handle. Adding UNDEFs
on the temporary registers used for finding the live channels helps
reduce the liveness of those temporary registers, especially in loops.

Some titles affected :

Rise Of The Tomb Raider:
Totals from 2780 (22.58% of 12311) affected shaders:
Instrs: 1294455 -> 1294592 (+0.01%); split: -0.15%, +0.16%
Cycles: 1473136441 -> 1471302617 (-0.12%); split: -1.52%, +1.40%
Max live registers: 144282 -> 143595 (-0.48%)
Max dispatch width: 22200 -> 22232 (+0.14%)

Red Dead Redemption 2:
Totals from 435 (7.28% of 5972) affected shaders:
Instrs: 488472 -> 487594 (-0.18%); split: -0.31%, +0.14%
Cycles: 11354732 -> 11384928 (+0.27%); split: -0.44%, +0.71%
Spill count: 1217 -> 1172 (-3.70%)
Fill count: 3521 -> 3447 (-2.10%)
Scratch Memory Size: 64512 -> 62464 (-3.17%)
Max live registers: 35997 -> 35798 (-0.55%)

Fallout 4:
Totals from 8 (0.49% of 1638) affected shaders:
Instrs: 41908 -> 40509 (-3.34%)
Cycles: 3638464 -> 3555680 (-2.28%); split: -2.67%, +0.39%
Spill count: 717 -> 665 (-7.25%)
Fill count: 2542 -> 2438 (-4.09%)
Scratch Memory Size: 32768 -> 16384 (-50.00%)
Max live registers: 567 -> 534 (-5.82%)

Cyberpunk 2077:
Totals from 2984 (28.97% of 10301) affected shaders:
Instrs: 3888874 -> 3891600 (+0.07%); split: -0.20%, +0.27%
Cycles: 67906489 -> 67767721 (-0.20%); split: -0.68%, +0.47%
Spill count: 200 -> 98 (-51.00%)
Fill count: 237 -> 90 (-62.03%)
Scratch Memory Size: 10240 -> 8192 (-20.00%)
Max live registers: 215715 -> 212727 (-1.39%)

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24282>

intel/fs: consider UNDEF as non-partial write

A few titles show max live register reductions, but nothing
significant in instruction count or other stats.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24282>

radv: rename radv_pipeline_stage to radv_shader_stage

It's more generic and it will fit shader object just well.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: cleanup pipeline compute emit helpers

Merge both functions together and rename the function.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: rework determining the NGG stage without a graphics pipeline

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: simplify lowering NGG GS intrinsics

The is_ngg field is already set correctly for GS.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: rename graphics pipeline linking helpers

There is no pipeline dependency.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: move removing all varyings when the FS is a noop

This allows us to remove one more pipeline dependency.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: stop passing radv_graphics_pipeline to radv_fill_shader_info()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: rework considering force VRS without relying on graphics pipeline

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: set next_stage to MESA_SHADER_NONE if there is no FS

This follows the same convention as shader object where the last stage
would have nextStage to 0. This will allow more refactoring.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

radv: initialize stage/next_stage earlier

This will allow more refactoring.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24313>

intel/vec4: fix log_data pointer

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 3384f029be ("intel/compiler: rework input parameters")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9421
Acked-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24307>

ac: Switch to use nir_foreach_function_impl in function analyze_shader_before_culling

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23940>

radeonsi: Convert to use nir_foreach_function_impl

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23940>

microsoft/clc/compiler: Convert to use nir_foreach_function_impl when possible

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23940>

microsoft/compiler: convert to use nir_foreach_function_with_impl in function emit_module

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23940>

vulkan/runtime: Add helper functions for VK_EXT_host_image_copy

Add helper functions vk_memory_to_image_copy_layout() and
vk_image_to_memory_copy_layout(), which will be useful in
VK_EXT_host_image_copy implementations.

vk_memory_to_image_copy_layout() is similar to
vk_image_buffer_copy_layout(), except the second parameter is
VkMemoryToImageCopyEXT instead of VkBufferImageCopy2.

vk_image_to_memory_copy_layout() is similar to
vk_image_buffer_copy_layout(), except the second parameter is
VkImageToMemoryCopyEXT instead of VkBufferImageCopy2.

Signed-off-by: Rebecca Mckeever <rebecca.mckeever@collabora.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24290>

gm107/ir: fix SULDP for loads without a known format

Signed-off-by: Karol Herbst <git@karolherbst.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24312>

intel/compiler: call brw_nir_adjust_payload from brw_postprocess_nir

Calling anything after nir_trivialize_registers() risks undoing some of
its work.
In this case, brw_nir_adjust_payload() will do a constant folding pass
if any payload adjusting happened, and that can turn a bunch of
@store_regs into basically noops.

Fixes dEQP-VK.subgroups.*task

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24325>

intel/fs: Constant fold OR and AND

The path taken in fs_visitor::swizzle_nir_scratch_addr for DG2 generates
some AND and OR instructions before the SHL. This commit folds those so
the whold calculation becomes a constant (like on older platforms).

v2: Fix return type of src_as_uint. Noticed by Marcin.

shader-db results:

DG2
total instructions in shared programs: 23190475 -> 23179540 (-0.05%)
instructions in affected programs: 36026 -> 25091 (-30.35%)
helped: 7 / HURT: 0

total cycles in shared programs: 841196807 -> 841142563 (<.01%)
cycles in affected programs: 1660670 -> 1606426 (-3.27%)
helped: 7 / HURT: 0

No shader-db changes on any older Intel platforms.

fossil-db results:

DG2
Totals:
Instrs: 197780372 -> 197773966 (-0.00%)
Cycles: 14066410782 -> 14066399378 (-0.00%); split: -0.00%, +0.00%
Subgroup size: 8438104 -> 8438112 (+0.00%)
Send messages: 8049445 -> 8049446 (+0.00%)
Scratch Memory Size: 14263296 -> 14264320 (+0.01%)

Totals from 9 (0.00% of 668055) affected shaders:
Instrs: 24547 -> 18141 (-26.10%)
Cycles: 1984791 -> 1973387 (-0.57%); split: -0.98%, +0.40%
Subgroup size: 88 -> 96 (+9.09%)
Send messages: 867 -> 868 (+0.12%)
Scratch Memory Size: 69632 -> 70656 (+1.47%)

No fossil-db changes on any older Intel platforms.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23884>

intel/fs: Constant fold SHL

This is a modified version of a commit originally in !7698. This version
add the changes to brw_fs_copy_propagation. If the address passed to
fs_visitor::swizzle_nir_scratch_addr is a constant, that function will
generate SHL with two constant sources.

DG2 uses a different path to generate those addresses, so the constant
folding can't occur there yet. That will be addressed in the next
commit.

What follows is the commit change history from that older MR.

v2: Previously this commit was after `intel/fs: Combine constants for
integer instructions too`.  However, this commit can create invalid
instructions that are only cleaned up by `intel/fs: Combine constants
for integer instructions too`.  That would potentially affect the
shader-db results of each commit, but I did not collect new data for
the reordering.

v3: Fix masking for W/UW and for Q/UQ types. Add an assertion for
!saturate. Both suggested by Ken. Also add an assertion that B/UB types
don't matically come back.

v4: Fix sources count. See also ed3c2f73dbb ("intel/fs: fixup sources
number from opt_algebraic").

v5: Fix typo in comment added in v3. Noticed by Marcin. Fix a typo in a
comment added when pulling this commit out of !7698. Noticed by Ken.

shader-db results:

DG2
No changes.

Tiger Lake, Ice Lake, and Skylake had similar results (Ice Lake shown)
total instructions in shared programs: 20655696 -> 20651648 (-0.02%)
instructions in affected programs: 23125 -> 19077 (-17.50%)
helped: 7 / HURT: 0

total cycles in shared programs: 858436639 -> 858407749 (<.01%)
cycles in affected programs: 8990532 -> 8961642 (-0.32%)
helped: 7 / HURT: 0

Broadwell and Haswell had similar results. (Broadwell shown)
total instructions in shared programs: 18500780 -> 18496630 (-0.02%)
instructions in affected programs: 24715 -> 20565 (-16.79%)
helped: 7 / HURT: 0

total cycles in shared programs: 946100660 -> 946087688 (<.01%)
cycles in affected programs: 5838252 -> 5825280 (-0.22%)
helped: 7 / HURT: 0

total spills in shared programs: 17588 -> 17572 (-0.09%)
spills in affected programs: 1206 -> 1190 (-1.33%)
helped: 2 / HURT: 0

total fills in shared programs: 25192 -> 25156 (-0.14%)
fills in affected programs: 156 -> 120 (-23.08%)
helped: 2 / HURT: 0

No shader-db changes on any older Intel platforms.

fossil-db results:

DG2
Totals:
Instrs: 197780415 -> 197780372 (-0.00%); split: -0.00%, +0.00%
Cycles: 14066412266 -> 14066410782 (-0.00%); split: -0.00%, +0.00%

Totals from 16 (0.00% of 668055) affected shaders:
Instrs: 16420 -> 16377 (-0.26%); split: -0.43%, +0.17%
Cycles: 220133 -> 218649 (-0.67%); split: -0.69%, +0.01%

Tiger Lake, Ice Lake and Skylake had similar results. (Ice Lake shown)
Totals:
Instrs: 153425977 -> 153423678 (-0.00%)
Cycles: 14747928947 -> 14747929547 (+0.00%); split: -0.00%, +0.00%
Subgroup size: 8535968 -> 8535976 (+0.00%)
Send messages: 7697606 -> 7697607 (+0.00%)
Scratch Memory Size: 4380672 -> 4381696 (+0.02%)

Totals from 6 (0.00% of 662749) affected shaders:
Instrs: 13893 -> 11594 (-16.55%)
Cycles: 5386074 -> 5386674 (+0.01%); split: -0.42%, +0.43%
Subgroup size: 80 -> 88 (+10.00%)
Send messages: 675 -> 676 (+0.15%)
Scratch Memory Size: 91136 -> 92160 (+1.12%)

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23884>

intel/fs: Always do opt_algebraic after opt_copy_propagation makes progress

opt_copy_propagation can create invalid instructions like

    shl(8) vgrf96:UD, 2d, 8u

These instructions will be cleaned up by opt_algebraic.  The irony is
opt_algebraic converts these to simple mov instructions that
opt_copy_propagation should clean up.  I don't think we want a loop like

   do {
      progress = false;
      if (OPT(opt_copy_propagation)) {
         OPT(opt_algebraic);
         OPT(dead_code_eliminate);
      }
   } while (progress);

But maybe we do?

Maybe this would be sufficient:

   while (OPT(opt_copy_propagation))
      OPT(opt_algebraic);
   OPT(dead_code_eliminate);

No shader-db or fossil-db changes (yet) on any Intel platform.  This is
expected.

v2: Do opt_algebraic immediately after every call to
opt_copy_propagation instead of being clever. Suggested by Lionel.

Tested-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23884>

ci/a5xx: Add another GPU hanging piglit test to the skips.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23485>

Uprev Piglit to ed58dfbd12be34fa3dab97a7a2987b890e0637f1

https://gitlab.freedesktop.org/mesa/piglit/-/compare/5036601c43fff63f7be5cd8ad7b319a5c1f6652c...ed58dfbd12be34fa3dab97a7a2987b890e0637f1

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23485>

tu: Fix data race in userspace VMA management.

The sequence was two threads A and B on a shared VkDevice:

A: move a BO to zombie VMA list
A: drop the BO VMA lock
B: prepare to allocate a BO
B:   Lock BO VMA lock
B:     call tu_free_zombie_vma_locked()
B:       close the gem handle from the VMA list
B:   Drop BO VMA lock
B: allocate a BO, getting the recently-closed handle back.
B: initialize the BO struct for the new handle.
A: memset the BO struct to 0.

Multithreading in C is the worst.

Closes: #9049, #9247
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24324>

iris: Request Xe KMD to place BOs to CPU visible VRAM when required

This is required to support discrete GPUs placed in systems with large
PCI bar or resizeble PCI bar not available or disabled.

Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23781>

anv: Request Xe KMD to place BOs to CPU visible VRAM when required

This is required to support discrete GPUs placed in systems with large
PCI bar or resizeble PCI bar not available or disabled.

Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23781>

intel/dev/xe: Add support for small-bar setups

This adds support for discrete GPUs placed in systems with large PCI
bar or resizeble PCI bar not available or disabled.

Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23781>

intel: Sync xe_drm.h

Sync with commit aef50195664a ("drm/xe/uapi: add the userspace bits for small-bar")

Link: https://patchwork.freedesktop.org/series/115515/
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23781>

ci/tu: Drop some xfails for !24086

Fixes: 99e58460efb9 ("tu: Fix zombie VMAs array not initialized when first BOs may be freed")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24322>

ci/tu: Mark descriptor_buffer.basic.limits as failing in gmem too.

Noticed in a full run.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24322>

ci/tu: Add more crash cases for the multithreading bugs caught on a630.

Weirdly, we don't see this group on a618.  Different CPU timings/core
counts just getting unlucky?

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24322>

nir/lower_blend: Optimize out PIPE_LOGICOP_NOOP

Just drop the store. Written while debugging
dEQP-VK.pipeline.monolithic.logic_op.r8_uint.no_op.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Italo Nicola <italonicola@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24252>

nir/lower_blend: Fix 32-bit logicops

nir_const_value_for_int asserts signed bounds on the input, but we pass in an
unsigned value that would be out-of-bounds for 32-bit channels, causing the
assert to fail for 32-bit channel formats.

Fixes dEQP-VK.pipeline.monolithic.logic_op.r32_uint.* on AGXV (and probably
PanVK).

Fixes: dbd0615e7ad ("nir/lower_blend: Avoid useless iand with logic ops")
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Italo Nicola <italonicola@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24252>

panfrost: Disable blending for no-op logic ops

Prevents regression from the series, since we don't support empty blend
shaders. This could be fixed more generically but I'm not inclined to compile
more blend shaders than needed so shrug.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Italo Nicola <italonicola@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24252>

rusticl: fix warnings with newer rustc

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24315>

intel/fs: Assume NIR is in SSA form

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24310>

intel/fs: Rework the overlapping mov/vec case

Now that we're using load/store_reg intrinsics, the previous checks for
registers aren't what we want.  Instead, we need to be looking for a mov
or vec where both the destination and a source are load/store_reg with
matching decl_reg.

Fixes: b8209d69ffdb ("intel/fs: Add support for new-style registers")
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24310>

intel/fs: Use write masks from store_reg intrinsics

Fixes: b8209d69ffdb ("intel/fs: Add support for new-style registers")
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24310>

broadcom/compiler: Convert to new-style NIR registers

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

nir/schedule: Support load/store_reg

These are tracked the same way as register reads and writes, allowing
them to be re-arranged as long as they respect dependencies within the
same reg.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

vc4: Convert to new-style NIR registers

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

nir/trivialize: Move decl_reg to the start of the block

This makes it so we never find a reg_decl in between a reg_store and the def
for its value, which helps avid inserting copy movs.

Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

nir/trivialize: Handle more RaW hazards

Consider the snippet of NIR:

   div 32    %447 = @load_reg (%442) (base=0, legacy_fabs=0, legacy_fneg=0)
   div 32    %463 = @load_reg (%442) (base=0, legacy_fabs=0, legacy_fneg=0)
   con 32    %409 = iadd %17 (0x3), %447
                    @store_output (%182 (0x601), %463) (base=0, wrmask=x, component=0, src_type=invalid...
                    @store_reg (%409, %442) (base=0, wrmask=x, legacy_fsat=0)

The load_reg's are trivial, so the %442 read will get folded into store_output.
But under the old definition, the store_reg is also trivial so it gets folded
into the iadd... causing a read-after-write hazard and invalid code generation.

The fix is to amend our definition of store_reg triviality to account for loads
getting folded in. It's not good enough that there's no intervening load_reg,
there can also be no intervening source that gets chased to a load_reg. Handle
that case as well.

Identified in dEQP-VK.geometry.input.basic_primitive.triangles_adjacency on
V3DV.

Fixes: d313eba94ef0 ("nir: Add pass for trivializing register access")
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reported-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

nir/trivialize: Trivialize cross-block loads

In order for a register load to be trivial, it cannot be used in any
block other than the one in which it is loaded.  We're not currently
explicitly doing anything to ensure this invariant holds.  It may be
that it holds regardless but I couldn't find any documented reason why
it should so let's explicitly handle that case.  Worst case, the newly
added code does nothing.

Fixes: d313eba94ef0 ("nir: Add pass for trivializing register access")
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

nir/trivialize: Maintain divergence information

Because this pass is intended to be run after out-of-SSA and directly
before injesting the NIR into the back-end, it may come after divergence
analysis and needs to preserve the divergence information. Fortunately,
since all we ever do is insert nir_op_mov, this is easy.

Fixes: d313eba94ef0 ("nir: Add pass for trivializing register access")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

nir: Properly handle divergence for load_reg

This commit makes three changes:

 1. Default all newly created registers divergent because this is the
    safer default.

 2. Make divergence analysis do something sane with register divergence.
    It's not perfect because divergence analysis isn't able to prove
    registers divergent based on stores but at least if someone uses
    registers a bit they'll end up with safe defaults.  This matches
    what they'd get with nir_ssa_def_init().

 3. Make the load_reg() helper automatically propagate divergence from
    the register.  Because the defaults for both nir_ssa_def_init() and
    nir_decl_reg() are to mark everything divergent, this only means
    that nir_load_reg() of a uniform reg is now uniform.

Putting all these together, nir_from_ssa should now be producing
load_reg intrinsics with the proper uniform information.

Fixes: 7229bffcb133 ("nir: Add intrinsics for register access")
Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

pan/bi: Remove leftover include

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24153>

intel/compiler/test: fix crashes when TEST_DEBUG is set

Dumping instructions requires that ISA info is not empty.

Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24274>

lavapipe: fixes indent of function lvp_inline_uniforms

The indent fixes are in separate patch is for easier to review

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24316>

lavapipe: Convert to use nir_foreach_function_impl

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24316>

zink: Convert to use nir_foreach_function_impl when possible

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24316>

freedreno: Switch to use nir_foreach_function_impl in tu_shader.cc

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24316>

lima: Convert to use nir_foreach_function_impl when possible

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Erico Nunes <nunes.erico@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24316>

rusticl/core: Make convert_spirv_to_nir output pair (KernelInfo, NirShader)

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23898>

rusticl/core: Delete KernelDevState and KernelDevStateInner

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/23898>