radv: move ac_perfcounters to physical_device.

Going to need it there for vkGetPhysicalDeviceQueueFamilyPerformanceQueryPassesKHR.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Add performance counter reg write.

Needed for reliably writing performance counter selectors.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

docs: briefly document Gallium Nine

Credits: Axel Davy, lorn10, Pavel Ondračka

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4937

Acked-by: Filip Gawin <filip@gawin.net>
Acked-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17154>

intel/compiler: Don't set SBID on EOT send messages

The send message with EOT is the last instruction so there's no need to
set any SWSB annotations for future instructions.

Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17202>

nir/serialize: restore ray query variables

The ray query status of a variable is tracked in the
nir_variable::data. We need to store it in the serialization otherwise
restoring NIR from a cache will drop the annotation.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 5a9cdab17050 ("nir: track variables representing ray queries")
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16059>

ci: Remove GIT_STRATEGY from .test-manual-mr.

Since .test-manual-mr is just a modifier of an existing test job's deps,
it should already have this var set appropriately.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Fix non-freedreno performance jobs running during Marge merges.

I mistakenly applied .gl-rules to the non-freedreno perf jobs, which
caused them to be incorrectly run pre-merge when core GL files changed.
Pull the freedreno core GL performance job rules out, explain a bit more
what is going on, and use it from iris and virgl performance testing.

This also drops running freedreno performance when core vulkan files
change -- freedreno perf testing doesn't have any turnip usage, nor does
it watch for turnip file changes.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Remove .build-rules from core test job definitions.

If you accidentally re-included your test job core definition after your
driver-specific ruleset, you'd end up running the driver job on every
source code change.  This had happened with a630_gles_asan: it included
.baremetal-test-arm64-asan (and thus .baremetal-test) after including
.a630-test, to override .baremetal-test-arm64's depednencies to use asan
artifacts instead.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Drop .build-rules from container jobs.

The rules: in this job overrides the .build-rules.  This was a leftover
from retry: being the former definition of .build-rules.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Rename .ci-run-policy rules to .build-rules.

... and explain what they're doing, compared to the test rules in
test-source-dep.yml.

Unfortunately, we can't really pull them into test-source-dep.yml with
other source deps, because of various '&'-'*' references.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Make the retry policy default for all jobs.

We had to make sure to enable .ci-run-policy from every job to get the
retry, but we can just put it in the default section.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

anv: Implement VK_EXT_shader_module_identifier

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

intel,anv,iris,crocus: Drop subgroup size from the shader key

Use nir->info.subgroup_size instead.

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Use vk_pipeline_shader_stage_is_null()

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

vulkan: Add a vk_pipeline_shader_stage_is_null() helper

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Use vk_pipeline_shader_stage_to_nir

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Put a VkPipelineShaderStageCreateInfo* in anv_pipeline_stage

It's an entirely temporary struct used by the compile process and never
escapes vkCreate*Pipelines so it's safe to just stuff the pointer in
there.  This makes it easier to use some of our new helpers.

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Drop unnecessary parameters to anv_pipeline_compile_cs

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

mesa,glsl,ttn: Set subgroup_size to UNIFORM

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

spirv,vulkan: Set shader_info::subgroup_size

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

vulkan: Constify vk_spirv_version

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

vulkan: Re-order pipeline hashing

Match the order in vkPipelineShaderStageCreateInfo

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

shader_info: Move subgroup_size out of cs and make it an enum

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

d3d12: add more formats to supported conversions

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4761
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17435>

vulkan/wsi: Fix structure chaining in wsi_create_buffer_image_mem

First, because we're using __vk_append_struct which attacks it on the
end, memory_wsi_info is modified even though it's const.  Make things
non-const so we aren't silently violating assumptions.  Also, we set a
pNext in memory_export_info which causes a loop in the pNext chain in
the handle_types != 0 case.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6826
Fixes: 124848bf9efa ("vulkan/wsi: Support tiled CPU images")
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17434>

vulkan/wsi/wayland: Only memcpy if the swapchain is actually software

Otherwise, we'll segfault. :-(

Fixes: aca545d616d8 ("vulkan/wsi/wayland: Use host pointer import when available")
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17434>

panfrost/ci: Disable T760 jobs

These keep timing out due to abusive jobs.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17433>

freedreno/a6xx: Do clip-plane lowering in backend

Our GS-lowered-to-quasi-VS confuses core nir passes, so handle clip-
plane lowering ourself.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: Handle driver-params in GS/DS

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno: rename ir3_emit_driver_params()

Driver-params are not VS specific, rename helper to reflect this.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Fix GS clip-plane lowering

And also handle tess.  In all cases, we want to use the VS lowering pass
on the last geometry stage.  We don't make a special exception for GS
like other drivers, because GS gets lowered into a quasi-VS.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: Fix VS const packet size

Need to account for the PKT7 header.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: Fix indentation

Another victim of automated re-indenting being unaware of the semantics.
Re-indent this to put each dword of the packet on it's own line to make
the size of the packet more clear.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Add more tess varying slots

Fixes some piglits that I stumbled across by mistake.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Copy vars if needed on EndPrimitive()

If we didn't EmitPrimitive() then the shadow (old) outputs would not
get copied to the emit temps (to eventually be copied back to the real
outputs.  This isn't so bad except that means the realy vertex_flags
output has an undefined value.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Add copy_vars() helper

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Don't lower_gs multiple times

At least with gallium, this can be called multiple times via
pipe_screen::finalize_nir().  But it is not designed to be called
multiple times, and can result in vertex_flags getting 'optimized'
away.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6720
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

turnip: assert valid vertex_flag reg

If this somehow gets optimized out, the GS will run forever.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: assert valid vertex_flags reg

If this somehow gets optimized out, the GS will run forever.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

intel/fs: Remove non-_LOGICAL URB messages

The _LOGICAL versions are lowered direct to SEND, so nothing can ever
generate these messages.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/fs: Lower URB messages to SEND

Before rebasing on top of Ken's split-SEND optimization (see !17018),
this commit just caused some scheduling changes in various tessellation
and geometry shaders.  These changes were caused by the addition of real
latency information for the URB messages.

With the addition of the split-SEND optimization, the changes
are... staggering.  All of the shaders helped for spills and fills are
vertex shaders from Batman Arkham Origins.  What surprises me is that
these shaders account for such a high percentage of the spills and fills
in fossil-db.  85%?!?

v2: Use FIXED_GRF instead of BRW_GENERAL_REGISTER_FILE in an assertion.
Suggested by Ken.

Tiger Lake, Ice Lake, and Skylake had similar results. (Ice Lake shown)
total instructions in shared programs: 20013625 -> 19954020 (-0.30%)
instructions in affected programs: 4007157 -> 3947552 (-1.49%)
helped: 31161
HURT: 0
helped stats (abs) min: 1 max: 400 x̄: 1.91 x̃: 2
helped stats (rel) min: 0.08% max: 59.70% x̄: 2.20% x̃: 1.83%
95% mean confidence interval for instructions value: -1.97 -1.86
95% mean confidence interval for instructions %-change: -2.22% -2.18%
Instructions are helped.

total cycles in shared programs: 859337569 -> 858636788 (-0.08%)
cycles in affected programs: 74168298 -> 73467517 (-0.94%)
helped: 13812
HURT: 16846
helped stats (abs) min: 1 max: 291078 x̄: 82.83 x̃: 4
helped stats (rel) min: <.01% max: 37.09% x̄: 3.47% x̃: 2.02%
HURT stats (abs)   min: 1 max: 1543 x̄: 26.31 x̃: 14
HURT stats (rel)   min: <.01% max: 77.97% x̄: 4.11% x̃: 2.58%
95% mean confidence interval for cycles value: -55.10 9.39
95% mean confidence interval for cycles %-change: 0.62% 0.77%
Inconclusive result (value mean confidence interval includes 0).

Broadwell
total cycles in shared programs: 904844939 -> 904832320 (<.01%)
cycles in affected programs: 525360 -> 512741 (-2.40%)
helped: 215
HURT: 4
helped stats (abs) min: 4 max: 1018 x̄: 60.16 x̃: 39
helped stats (rel) min: 0.14% max: 15.85% x̄: 2.16% x̃: 2.04%
HURT stats (abs)   min: 79 max: 79 x̄: 79.00 x̃: 79
HURT stats (rel)   min: 1.31% max: 1.57% x̄: 1.43% x̃: 1.43%
95% mean confidence interval for cycles value: -75.02 -40.22
95% mean confidence interval for cycles %-change: -2.37% -1.81%
Cycles are helped.

No shader-db changes on any older Intel platforms.

Tiger Lake, Ice Lake, and Skylake had similar results. (Ice Lake shown)
Instructions in all programs: 142622800 -> 141461114 (-0.8%)
Instructions helped: 197186

Cycles in all programs: 9101223846 -> 9099440025 (-0.0%)
Cycles helped: 37963
Cycles hurt: 151233

Spills in all programs: 98829 -> 13695 (-86.1%)
Spills helped: 2159

Fills in all programs: 128142 -> 18400 (-85.6%)
Fills helped: 2159

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/fs: Add _LOGICAL versions of URB messages

The lowering is currently fake.  It just changes the opcode from the
_LOGICAL version to the non-_LOGICAL version.

v2: Remove some rebase cruft.  's/gfx8_//;s/simd8_/' in
brw_instruction_name.  Both suggested by Ken.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/compiler: Move logical-send lowering to a separate file

brw_fs.cpp was 10kloc.  Now it's only 7.5kloc.  Ugh.

v2: Rebase on 9680e0e4a2d.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/eu: Validate some aspects of URB messages

If these checks had been in place previously, some bugs
that... eh-hem... practically took down the Intel CI would have been
caught earlier. *blush*

v2: Update to account for split sends.

v3: Add some more Gfx version checks.  Remove the redundant "src0 is a
GRF" check.  Both suggested by Ken.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/compiler: Rename vec4 state URB opcodes to have VEC4_ prefix

An argument could be made that all stage-specific opcodes for vec4
stages should be prefixed with VEC4_ like the stage-agnostic opcodes.
I'll leave those additional sed jobs for another day.

    egrep -lr '(VS|GS|TCS)_OPCODE_URB_WRITE' src |\
    while read f; do
        sed --in-place 's/\(VS\|GS\|TCS\)_OPCODE_URB_WRITE/VEC4_\1_OPCODE_URB_WRITE/g' $f
    done

    egrep -lr 'T.S_OPCODE[_A-Z]*URB_OFFSETS' src |\
    while read f; do
        sed --in-place 's/\(T.S_OPCODE[_A-Z]*URB_OFFSETS\)/VEC4_\1/g' $f
    done

Suggested-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

dzn: Add for condition to break nested loop

Fixes: d132ec92 ("dzn: Support native image copies when formats are compatible")
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17377>

dzn: Fix incompatible pointer type error affecting MSYS2 MINGW32
Suggested-by: Yonggang Luo <luoyonggang@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6807

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17414>

ci/traces: piglit, be more verbose

Print more information about traces testing progress.

Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17416>

radv/ci: enable fossils testing for GFX1100

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16447>

aco: use scratch_* for VGPR spill/reload on GFX9+

fossil-db (navi21):
Totals from 12 (0.01% of 162293) affected shaders:
Instrs: 122808 -> 122782 (-0.02%); split: -0.11%, +0.09%
CodeSize: 711248 -> 710788 (-0.06%); split: -0.16%, +0.10%
SpillSGPRs: 928 -> 831 (-10.45%)
SpillVGPRs: 1626 -> 1624 (-0.12%)
Latency: 4960285 -> 4932547 (-0.56%)
InvThroughput: 2574083 -> 2559953 (-0.55%)
VClause: 3404 -> 3402 (-0.06%)
Copies: 36992 -> 37181 (+0.51%); split: -0.05%, +0.56%
Branches: 3582 -> 3585 (+0.08%)
PreVGPRs: 3055 -> 3057 (+0.07%)

fossil-db (vega10):
Totals from 12 (0.01% of 161355) affected shaders:
Instrs: 124817 -> 124383 (-0.35%); split: -0.46%, +0.12%
CodeSize: 705116 -> 703664 (-0.21%); split: -0.44%, +0.23%
SpillSGPRs: 1012 -> 898 (-11.26%)
SpillVGPRs: 1632 -> 1624 (-0.49%)
Scratch: 201728 -> 200704 (-0.51%)
Latency: 6160115 -> 6266025 (+1.72%); split: -0.34%, +2.06%
InvThroughput: 6440203 -> 6544595 (+1.62%); split: -0.35%, +1.97%
VClause: 3409 -> 3423 (+0.41%)
Copies: 37929 -> 37748 (-0.48%); split: -1.16%, +0.69%
Branches: 3851 -> 3855 (+0.10%); split: -0.13%, +0.23%
PreVGPRs: 3053 -> 3055 (+0.07%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: use scratch_* for scratch load/store on GFX9+

fossil-db (navi21):
Totals from 52 (0.03% of 162293) affected shaders:
Instrs: 83190 -> 82145 (-1.26%)
CodeSize: 454892 -> 447260 (-1.68%); split: -1.68%, +0.00%
VGPRs: 4768 -> 4672 (-2.01%)
Latency: 1490887 -> 1487170 (-0.25%); split: -0.68%, +0.43%
InvThroughput: 935500 -> 933060 (-0.26%); split: -0.72%, +0.46%
VClause: 2715 -> 2632 (-3.06%); split: -4.53%, +1.47%
SClause: 1902 -> 1883 (-1.00%)
Copies: 8839 -> 8496 (-3.88%)
PreSGPRs: 2012 -> 1807 (-10.19%)
PreVGPRs: 3282 -> 3192 (-2.74%)

fossil-db (vega10):
Totals from 41 (0.03% of 161355) affected shaders:
Instrs: 35772 -> 35699 (-0.20%)
CodeSize: 187040 -> 186584 (-0.24%)
VGPRs: 4044 -> 4072 (+0.69%)
Latency: 243088 -> 242379 (-0.29%)
InvThroughput: 180301 -> 179783 (-0.29%)
VClause: 1204 -> 1216 (+1.00%)
SClause: 653 -> 637 (-2.45%)
Copies: 3736 -> 3704 (-0.86%); split: -0.88%, +0.03%
PreSGPRs: 1331 -> 1207 (-9.32%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: initialize scratch base registers on GFX9-GFX10.3

fossil-db (navi21):
Totals from 1142 (0.70% of 162293) affected shaders:
Instrs: 271636 -> 271974 (+0.12%)
CodeSize: 1532020 -> 1533792 (+0.12%)
Latency: 7484066 -> 7485698 (+0.02%)
InvThroughput: 4048824 -> 4049579 (+0.02%)
SClause: 4171 -> 4212 (+0.98%)
PreSGPRs: 11203 -> 12276 (+9.58%)

fossil-db (vega10):
Totals from 3327 (2.06% of 161355) affected shaders:
Instrs: 257413 -> 257601 (+0.07%)
CodeSize: 1424244 -> 1425372 (+0.08%)
Latency: 8598402 -> 8600466 (+0.02%)
InvThroughput: 7906335 -> 7908234 (+0.02%)
SClause: 4932 -> 4973 (+0.83%)
PreSGPRs: 22010 -> 25405 (+15.42%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: treat flat-like as vmem in some scheduling heuristics

fossil-db (navi21):
Totals from 12 (0.01% of 162293) affected shaders:
Instrs: 48754 -> 48762 (+0.02%)
CodeSize: 267092 -> 267124 (+0.01%)
Latency: 1293798 -> 1292303 (-0.12%); split: -0.12%, +0.00%
InvThroughput: 854599 -> 853578 (-0.12%)
VClause: 1623 -> 1619 (-0.25%)
SClause: 1187 -> 1188 (+0.08%); split: -0.08%, +0.17%

fossil-db (vega10):
Totals from 1 (0.00% of 161355) affected shaders:
Latency: 18720 -> 18848 (+0.68%)
InvThroughput: 5775 -> 5776 (+0.02%)
SClause: 12 -> 11 (-8.33%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: include scratch/global in VMEM WAW optimization

fossil-db (navi21):
Totals from 2 (0.00% of 162293) affected shaders:
Instrs: 4788 -> 4785 (-0.06%)
CodeSize: 25884 -> 25872 (-0.05%)
Latency: 255008 -> 252950 (-0.81%)
InvThroughput: 170005 -> 168633 (-0.81%)
VClause: 206 -> 205 (-0.49%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: avoid WAW hazard with BVH MIMG and other VMEM

According to LLVM, image_bvh64_intersect_ray does not write results in
order with other VMEM instructions.

fossil-db (navi21):
Totals from 7 (0.00% of 162293) affected shaders:
Instrs: 39978 -> 39985 (+0.02%)
CodeSize: 219356 -> 219384 (+0.01%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: refactor VGPR spill/reload lowering

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: handle subtractions in parse_base_offset

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: combine additions and constants into scratch load/store

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: improve support for scratch_* instructions

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: make FLAT_instruction::offset signed

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: include flat-like in vmem clause statistics

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: make flat access latency match mtbuf/mubuf/mimg

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

virgl: Only progagate the uniform numbers if the numbers are actually right

When the field was first introduces, the numbers were reporting the number of
vec4 instead of the number of float. Do not propagate them if they are wrong.

Fixes: d92c1ca01b326d8f0ff210828830d6542f9e67f7

Signed-off-by: Corentin Noël <corentin.noel@collabora.com>
Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17415>

ci/lava: Add canceled job status

We should be explicit that we are cancelling jobs once the script finds
some log messages that are linked with known issues. That means the
script preemptively retried the job without giving chances to recover.

Adds magenta color to cancelled jobs.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>

ci/lava: Add `slow` pytest marker

Mark test_full_yaml_log with this new marker to be easily run by the
developers.
Make `debian-testing` skip this test with `not slow` marker hint.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>

ci/lava: Color red for fatal and yellow for warning

Fatal errors now have red foreground color and retry messages yellow
one.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>

ci/lava: Make hung job status yellow

It will help to know what happened to a non-successful job.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>

ci/lava: Detect R8152 issues preemptively and retry

Implement a log-based retry hint for R8152 issue described in #6681,
which is based on detecting these two consecutive lines:

```
r8152 <USB> eth0: Tx status -71
nfs: server <IP> not responding, still trying
```

Where <IP> and <USB> could be any IP and USB addresses, respectfully.

This commit is a temporary fix since it requires a section-aware log
follower, implemented in !16323. When the cited MR is merged, one will
make a proper fix on top of that.

Closes: #6681

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>

ci/lava: Split lava_log into modules

This script is getting too big, it been hard to extend it.

Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17389>

zink: flush pending clears for fb texture barriers

if a texture barrier occurs while clears are pending, these clears should
show up if the fb attachments are read in shaders, so trigger a renderpass
to flush out the clears

cc: mesa-stable

fixes #6766

fixes (radv):
dEQP-GLES3.functional.draw_buffers_indexed.overwrite_common.common_advanced_blend_eq_buffer_advanced_blend_eq
dEQP-GLES3.functional.draw_buffers_indexed.overwrite_common.common_blend_eq_buffer_advanced_blend_eq
dEQP-GLES3.functional.draw_buffers_indexed.overwrite_common.common_separate_blend_eq_buffer_advanced_blend_eq
dEQP-GLES3.functional.draw_buffers_indexed.overwrite_indexed.common_advanced_blend_eq_buffer_advanced_blend_eq
dEQP-GLES3.functional.draw_buffers_indexed.overwrite_indexed.common_advanced_blend_eq_buffer_blend_eq
dEQP-GLES3.functional.draw_buffers_indexed.overwrite_indexed.common_advanced_blend_eq_buffer_separate_blend_eq

Reviewed-By: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17363>

radv: fix dumping VS prologs assembly

This got removed by mistake and broke
RADV_DEBUG=shaders,nocache,prologs.

Fixes: 9fe2b6b7480 ("aco/radv: provide a vs prolog callback from aco to radv.")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17413>

radv: Fix vkCmdCopyQueryResults -> vkCmdResetPool hazard.

The Vulkan specification states:

> Query commands, for the same query and submitted to the same queue,
> execute in their entirety in submission order, relative to each other. In
> effect there is an implicit execution dependency from each such query
> command to all query commands previously submitted to the same queue.

Fixes dEQP-VK.query_pool.statistics_query.reset_after_copy.*

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17400>

aco/assembler: Fix s_bitreplicate_b64_b32 on GFX9.

This seems to be a relic from before aco added per generation opcodes.

Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17405>

aco: Fix swapping sources in SOPC -> SOPK optimization.

Fixes: 2d6b0a4177b ("aco/optimizer: Optimize SOPC with literal to SOPK.")

Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17407>

r600/sfn: Add missing std::array include.

Fixes: 79ca456b483 ("r600/sfn: rewrite NIR backend")
Closes https://gitlab.freedesktop.org/mesa/mesa/-/issues/6824

Signed-off-by: Georg Lehmann <dadschoorse@gmail.com>
Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17412>

radeonsi: use LLVMBuildLoad2 for inter-stage outputs loads

The PS case was covered by the previous commit, so we can use f32
everywhere.

Reviewed-by: Mihai Preda <mhpreda@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17361>

radeonsi: use LLVMBuildLoad2 in llvm PS

PS is the only shader type where unpacked 16-bit outputs can be used,
so use ac_shader_abi::is_16bit to pass the proper type to LLVMBuildLoad2.

Reviewed-by: Mihai Preda <mhpreda@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17361>

ac/llvm: use LLVMBuildLoad2 in visit_load

Only FS can have f16 outputs, so always use f32 here.

Reviewed-by: Mihai Preda <mhpreda@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17361>

ac/llvm: handle opaque pointers in visit_store_output

Outputs are always f32 or f16.

Reviewed-by: Mihai Preda <mhpreda@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17361>

ac: add per output is_16bit flag to ac_shader_abi

Outputs are always f32 except for FS that may use unpacked f16.
Store this information here to make it available to later processing.

Reviewed-by: Mihai Preda <mhpreda@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17361>

radeonsi: use LLVMBuildLoad2 where possible

This commit replaces LLVMBuildLoad usage by LLVMBuildLoad2
where possible (= where the pointee type is known).

Reviewed-by: Mihai Preda <mhpreda@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17361>

ac: use LLVMContextSetOpaquePointers if available

Disabling opaque pointers in LLVM doesn't fix all the issues but
it makes pointers non-opaque by default (eg LLVMPointerType()
returns a typed pointer).

Reviewed-by: Mihai Preda <mhpreda@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17361>

zink: re-enable EXT_primitives_generated_query for Turnip

https://gitlab.freedesktop.org/mesa/mesa/-/issues/6602 is resolved
so the extension could be re-enabled.

Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17163>

tu: Fix prim gen query and pipeline stats query interaction

Fixed:
- VK_QUERY_TYPE_PRIMITIVES_GENERATED_EXT was able to stop prim counter
  when pipeline stats query is running.
  - This may have happened when prim gen query was in secondary cmdbuf.
- VK_QUERY_TYPE_PRIMITIVES_GENERATED_EXT counting geometry in each tile.
- VK_QUERY_TYPE_PRIMITIVES_GENERATED_EXT counting geometry in each tile
  when pipeline stats query is started inside prim gen query and inside
  a renderpass.

The matter of VK_QUERY_TYPE_PRIMITIVES_GENERATED_EXT and pipeline stats
interaction is solved by tracking whether pipeline stats query is
running both on CPU (for non secondary cmdbuf case) and on GPU (for
secondary cmdbuf).

Note, prim gen query is not allowed with secondary command buffers, so
only pipeline stats query is tracked on gpu.
See https://gitlab.khronos.org/vulkan/vulkan/-/issues/3142

Counting geometry per each tile is solved by:
- Conditionally executing START/STOP_PRIMITIVE_CTRS to not run in tiling
  pass. Solves the case when prim gen query is inside a renderpass.
- Stop prim counters before executing `draw_cs` and restarting them
  afterwards. Solves prim gen query being outside a renderpass.

Fixes GL CTS tests with Zink + `TU_DEBUG=gmem`:
 GTF-GL46.gtf30.GL3Tests.transform_feedback.transform_feedback_max_separate
 GTF-GL46.gtf40.GL3Tests.transform_feedback2.transform_feedback2_basic
 GTF-GL46.gtf40.GL3Tests.transform_feedback2.transform_feedback2_framebuffer
 GTF-GL46.gtf40.GL3Tests.transform_feedback3.transform_feedback3_streams_overflow
 GTF-GL46.gtf40.GL3Tests.transform_feedback3.transform_feedback3_streams_queried
 GTF-GL46.gtf40.GL3Tests.transform_feedback2.transform_feedback2_states

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6602

Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17163>

tu,freedreno: Refactored START/STOP events for pipeline stats

For a5xx+ renamed:
- RST_PIX_CNT -> START_FRAGMENT_CTRS
- RST_VTX_CNT -> STOP_FRAGMENT_CTRS
- TILE_FLUSH  -> START_COMPUTE_CTRS
- STAT_EVENT  -> STOP_COMPUTE_CTRS
I'm not sure about a5xx itself but I'll take a chance of it being
similar to a6xx in this regard.

Knowing this emit_begin_stat_query/emit_end_stat_query can now emit
only events that are needed for the pool's flags.

Also primitive generated query clearly doesn't need fragment and
compute counters.

Passes tests:
 dEQP-VK.query_pool.statistics_query.*
 dEQP-VK.transform_feedback.primitives_generated_query.*

Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17163>

aco: fix load_barycentric_at_sample without MSAA

It's legal to use this instruction in a fragment shader, even if the
graphics pipeline doesn't use MSAA.

Fixes
dEQP-GLES31.functional.shaders.multisample_interpolation.interpolate_at_sample.non_multisample_buffer.sample_n_*.

Cc: mesa-stable
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17398>

nir/serialize: fix missing divergence info after deserialization

Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17397>

venus: Use maintenance4 to get max_size_buffer

This should help speedup the device initalization process.

Signed-off-by: Igor Torrente <igor.torrente@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17399>

venus: Add support to VK_KHR_maintenance4 extension

Signed-off-by: Igor Torrente <igor.torrente@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17399>

dzn: Only support high/normal queue priorities

D3D uses an int which seems like it'd support value between 0 and 100,
but in reality it only accepts values of exactly 0, or 100. The space
is left in case future values were to be added, so that comparisons
would work (e.g. MEDIUM_HIGH < HIGH).

Treat higher than 0.5 to be HIGH, and anything less to be NORMAL.

Reviewed-by: Bill Kristiansen <billkris@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17406>

panfrost: Clear with a quad to avoid flushing

Flushing the batch midframe (splitting a renderpass) is expensive on a tiler, as
it requires the GPU to flush the framebuffer contents to main memory and read
them back. Clearing the framebuffer should not trigger a flush. Apps expect
clears to be (almost) free, flushing for a clear is at the very least unexpected
behaviour.

The only reason we previously flushed is to ensure we could always use a "fast"
clear. But a slow clear is a heck of a lot faster than a flush ;-) Instead of
flushing, we should clear with a draw (via u_blitter) in case a fast clear isn't
possible.

This fixes pathological performance for applications that rely on partial clears
within a frame. This issue was identified with Inochi2D, which repeatedly clears
the stencil buffer midframe, in order to implement masking efficiently with the
stencil buffer. In total, the all-important workload of rendering Asahi Lina is
improved from 17fps to 29fps on a panfrost device.

Fixes: c138ca80d23 ("panfrost: Make sure a clear does not re-use a pre-existing batch")
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17112>

panfrost/ci: Disable 0ad trace on T860

The last few frames of the trace are expensive (in terms of GPU time) and are
close to hitting the timeout. With the next commit, they do hit the timeout due
to using a larger batch. Nevertheless the next commit should be an overall perf
improvement on average, so this remove to unblock CI.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Suggested-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17112>

pan/decode: Change indent when decoding resources

Make the separation between entries in the resource table more
obvious.

Increase the indent by two levels to keep descriptors distinct from
the resource entry itself.

Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17371>

pan/decode: Use tag bits for resource entry count

Fixes crashes when decoding the blob, which sometimes uses fewer than
9 entries.

Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17371>

pan/decode: fflush buffers after dumping and before aborts

Otherwise trace files or other files being written (dEQP TestResults?)
might be truncated.

Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17371>

pan/va: Use the _safe iterator when adding blend shader calls

Otherwise the list 'next' changing will cause the assertion in
list_for_each_entry to be hit.

This was not hit before because list_assert is defined for debug
builds but not debugoptimized.

Fixes: 5067a26f443 ("pan/bi: Use flow control lowering on Valhall")
Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17371>

panfrost: Enable QUAD_STRIP and POLYGON on v6

I wrote fiction about dreaming that these were supported but after
waking finding that they were not. Somehow I came to consider that
fiction as fact and I never thought to test if they did work.

While reverse engineering the polygon list format, I found that these
were supported after all.

Reviewed-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17371>

panfrost: Fix PIPE_COMPUTE_CAP_SUBGROUP_SIZE

Use the new helper to implement the CAP, correctly handling Midgard and Valhall.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17265>

panfrost: Fix PIPE_COMPUTE_CAP_MAX_COMPUTE_UNITS

This maps to CL_DEVICE_MAX_COMPUTE_UNITS, which is defined as:

   The number of parallel compute cores on the OpenCL device.

Since all supported Malis are unified shader cores, the number of parallel
compute cores is simply the number of shader cores.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17265>

panfrost: Move bifrost_lanes_per_warp to common

Whereas the compiler needs to know the warp size for lowering divergent
indirects, the driver needs to know it to report the subgroup size. Move the
Bifrost-specific helper to common and add the trivial implementation for
Midgard.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17265>