pvr: Make pvr_cmd_pack() macro clearly internal

Add a double-underscore prefix to mark this macro as internal-only. It
should not be used directly, and only exists as a helper for the
pvr_csb_*() macros.

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Frank Binns <frank.binns@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17359>

pvr: Remove pvr_cmd_struct() macro

Motivated by comments on !17252; this macro adds nothing of value over
just using PVRX() directly.

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Frank Binns <frank.binns@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17359>

pvr: Add a const qualifier to inline pvr_csb_*() helpers

These do not modify the input struct pvr_csb at all, so the function
signature should reflect this.

Signed-off-by: Matt Coster <matt.coster@imgtec.com>
Reviewed-by: Frank Binns <frank.binns@imgtec.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17360>

broadcom/compiler: don't over-estimate latency of TMU instructions

Over-estimating latency can cause us to delay the critical paths of
the shader unnecessarily, producing larger QPU programs that take more
time to execute as a result (and it also adds register pressure) so
striking a balance is important. The thread switching model in V3D
is quite effective at hiding latency and usuallly we just need to
hint it to delay TMU instructions a little bit to find the best
compromise for performance.

The new latency numbers have been chosen empirically by testing
V3DV with Sponza and a few UE4 samples.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17451>

broadcom/compiler: don't try to hide TMU latency at QPU scheduling

Based on empirical testing with Sponza and a few UE4 samples this is
consistently slightly benefitial for performance.

The most likely reason why this helps is that thrsw is probably
already quite effective at hiding latency and we are already trying
to hide latency at NIR scheduling and also via TMU pipelining, so
piling up on this when scheduling QPU typically ends up providing no
benefit at all for latency and is instead possibly preventing us to
unblock critical paths in the shader that depend on the TMU result,
requiring us to execute more cycles to complete the program.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17451>

aco: fix packed 16bit fneg/fsat optimization

Make sure that the Operand is '1.0.xx'.

Fixes: b03be30e07546b813acc1156407b964e68892beb ('aco: optimize packed fneg')
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17395>

amd: Revert gfx10 addrlib changes

These broke a bunch of RADV VK CTS tests.
Revert the ADDR_SW_4KB_R_X changes from the commit to get CTS passing again.

Fixes: 3514b732445 ("amd: update addrlib - trivial changes")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6844
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17453>

radv/shader_info: fix load_frag_coord and load_sample_pos read masks

Fixes: a8c471f962e3a86f0cb6a91f14090b0c8cfb73ff ('radv: gather more information about PS in the shader info pass')
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17417>

radv: remove the radv_report_apu_as_dgpu workaround for Red Dead Redemption 2

This workaround looks actually broken. We added it in the past
because otherwise the game would just report 3GiB of video memory
(ie. size of GTT on SD). Though, with this workaround enabled, the
game explodes in memory easily.

One theory is that because we fake integrated GPUs as discrete GPUS,
and because we report 6GiB of VRAM (ie. driver redistributes memory
for small carveout), the game thinks there is 6GiB of VRAM only and
then keep allocating stuff.

People reported that the memory explosion is gone without this
workaround applied and I confirmed this myself.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17421>

r600/sfn: Add missing copyright information

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17450>

r300: Keep rc_rename_regs() from overflowing RC_REGISTER_MAX_INDEX

This pass tries to move register usage closer to SSA, and for large
shaders this means we can overflow the register index, which only has
RC_REGISTER_INDEX_BITS size. This creates invalid code and leads to
crash at a later stage. Limit the pool of available registers to
RC_REGISTER_MAX_INDEX, currently is was two times the number of
shader instructions.

This means we'll fail the compile right away if we wanted more than
RC_REGISTER_MAX_INDEX temps, but when we've got that many we're
already well past how many instructions we can support anyway.

CC: mesa-stable
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6017
Signed-off-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17393>

nir/opt_shrink_vectors: Round to supported vec size

The set of supported vector sizes in NIR has holes in it. For example, we
support vec5 and vec8, but not vec6 or vec7. However, this pass did not take
that into account, and would happily shrink a vec8 down to a vec7, causing NIR
validation to fail. Instead, the pass should round up to the next supported
vector size.

Fixes NIR validation fail in OpenCL's test_basic hiloeo subtest.

v2: Clamp -> round rename.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17194>

mesa: fix SignalSemaphoreEXT behavior

the EXT_external_object spec originally was underspecified with regards
to this function, leaving room for synchronization errors where:
* app calls SignalSemaphoreEXT to signal a semaphore
* mesa defers pipe_context::fence_server_signal with threaded context
* driver defers gpu submission
* SignalSemaphoreEXT has long since returned, app submits vk cmdbuf waiting on semaphore
* spec violation / device lost

to prevent this, the spec is being changed to:
1) require an implicit flush when calling SignalSemaphoreEXT
2) require that this implicit flush also forces GPU submission before SignalSemaphoreEXT returns

all affected drivers have been updated

fixes #6568

cc: mesa-stable

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17376>

Add Adreno 616 and 620 to use turnip on these GPU's .

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17441>

vc4: Add notes on the remaining dEQP failures.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17350>

vc4: Move previous existing 3D xfails up to the group of 3d xfails.

Clears up known issues from ones that should be investigated and
explained.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17350>

vc4: Disable OES_texture_3D being exposed.

The hardware doesn't support 3D textures.  We had been lying about 3D
texture level support in the past so that we got GL 2.1, but now reporting
levels==0 doesn't disable GL 2.1 (since we don't check for GL2 extensions
any more).  But, by not lying, we now fix the majority of the remaining
GLES2 deqp failures.

This regresses a few desktop GL piglits which get GL errors that they
notice instead of what would be silent rendering failures on 3D texturing
operations.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17350>

mesa/st: Disable OES_texture_3D for PIPE_CAP_MAX_TEXTURE_3D_LEVELS==0.

This will be used for vc4, where incorrectly exposing 3D textures accounts
for most of the GLES2 conformance failures it has.  This leaves
EXT_texture3d exposed in the (already non-conformant) GL2.1 support it
exposes, which has always been a best-effort thing.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17350>

util: remove now unused debug_assert

Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17403>

Change all debug_assert calls to assert

Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17403>

util: remove our redefinition of assert

I think we can all agree that our own assert is unnecessary.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6780

Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Acked-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17403>

ci: add script for manage Mesa CI

Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16891>

dri: Check buffer height and avoid overflow

The dri2_allocate_buffer() can be called with arbitrary height, however
the struct pipe_resource .height0 member is uint16_t. Check height for
maximum size to avoid overflow. Note that .width0 is unsigned int, so
it does not have the same issue.

The uint16 limit comes from commit:
e6428092f5e ("gallium: decrease the size of pipe_resource - 64 -> 48 bytes")

The overflow can be triggered e.g. by requesting large BO:
```
gbm_bo_create(dev, 1, 640*480*4, GBM_FORMAT_R8, GBM_BO_USE_LINEAR);
```

Signed-off-by: Marek Vasut <marex@denx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16513>

radeonsi: fix random PS wave size

Fixes: b3b2f97f2e25b2c "radeonsi: add Wave32 heuristics and shader profiles"

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17410>

ac/surface: expose all 64K_R_X and 256K_R_X modifiers on gfx11

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17410>

amd: update addrlib - trivial changes

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17410>

CI: Add dzn to debian vulkan/clang/release builds

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17423>

dzn: Use fabs on floating point input

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17423>

dzn: Re-order dzn_physical_device_desc so = { 0 } doesn't initialize a struct

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17423>

dzn: Fix comparison order for format enums

Fixes: 53a352e1 ("dzn: Provide a helper to check if 2 formats are compatible")
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17423>

dzn: Disable Wunused-value for C++ files

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17423>

dzn: Only add Werror flags that the compiler supports

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17423>

dzn: Re-order includes in dzn_nir to prevent TRUE redefinition

Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17423>

radv: Expose VK_KHR_performance_query.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Implement listing performance counters.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Implement support for querying performance counters.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Add a list of performance counters.

Plus helpers that will be shared for listing counters + doing the
actual queries.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Add perf counter lock/unlock commandbuffers.

These set the pass and make sure we don't have multiple submissions
at the same time touching the perf counters/pass at the same time.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Add helper BO for perf counters.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Make inhibiting clockgating not static.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Rename SPM functions.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Add pstate locking for perfcounters.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: move ac_perfcounters to physical_device.

Going to need it there for vkGetPhysicalDeviceQueueFamilyPerformanceQueryPassesKHR.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

radv: Add performance counter reg write.

Needed for reliably writing performance counter selectors.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16879>

docs: briefly document Gallium Nine

Credits: Axel Davy, lorn10, Pavel Ondračka

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4937

Acked-by: Filip Gawin <filip@gawin.net>
Acked-by: Pavel Ondračka <pavel.ondracka@gmail.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17154>

intel/compiler: Don't set SBID on EOT send messages

The send message with EOT is the last instruction so there's no need to
set any SWSB annotations for future instructions.

Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17202>

nir/serialize: restore ray query variables

The ray query status of a variable is tracked in the
nir_variable::data. We need to store it in the serialization otherwise
restoring NIR from a cache will drop the annotation.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 5a9cdab17050 ("nir: track variables representing ray queries")
Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16059>

ci: Remove GIT_STRATEGY from .test-manual-mr.

Since .test-manual-mr is just a modifier of an existing test job's deps,
it should already have this var set appropriately.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Fix non-freedreno performance jobs running during Marge merges.

I mistakenly applied .gl-rules to the non-freedreno perf jobs, which
caused them to be incorrectly run pre-merge when core GL files changed.
Pull the freedreno core GL performance job rules out, explain a bit more
what is going on, and use it from iris and virgl performance testing.

This also drops running freedreno performance when core vulkan files
change -- freedreno perf testing doesn't have any turnip usage, nor does
it watch for turnip file changes.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Remove .build-rules from core test job definitions.

If you accidentally re-included your test job core definition after your
driver-specific ruleset, you'd end up running the driver job on every
source code change.  This had happened with a630_gles_asan: it included
.baremetal-test-arm64-asan (and thus .baremetal-test) after including
.a630-test, to override .baremetal-test-arm64's depednencies to use asan
artifacts instead.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Drop .build-rules from container jobs.

The rules: in this job overrides the .build-rules.  This was a leftover
from retry: being the former definition of .build-rules.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Rename .ci-run-policy rules to .build-rules.

... and explain what they're doing, compared to the test rules in
test-source-dep.yml.

Unfortunately, we can't really pull them into test-source-dep.yml with
other source deps, because of various '&'-'*' references.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

ci: Make the retry policy default for all jobs.

We had to make sure to enable .ci-run-policy from every job to get the
retry, but we can just put it in the default section.

Acked-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17386>

anv: Implement VK_EXT_shader_module_identifier

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

intel,anv,iris,crocus: Drop subgroup size from the shader key

Use nir->info.subgroup_size instead.

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Use vk_pipeline_shader_stage_is_null()

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

vulkan: Add a vk_pipeline_shader_stage_is_null() helper

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Use vk_pipeline_shader_stage_to_nir

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Put a VkPipelineShaderStageCreateInfo* in anv_pipeline_stage

It's an entirely temporary struct used by the compile process and never
escapes vkCreate*Pipelines so it's safe to just stuff the pointer in
there.  This makes it easier to use some of our new helpers.

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

anv: Drop unnecessary parameters to anv_pipeline_compile_cs

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

mesa,glsl,ttn: Set subgroup_size to UNIFORM

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

spirv,vulkan: Set shader_info::subgroup_size

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

vulkan: Constify vk_spirv_version

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

vulkan: Re-order pipeline hashing

Match the order in vkPipelineShaderStageCreateInfo

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

shader_info: Move subgroup_size out of cs and make it an enum

Reviewed-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17337>

d3d12: add more formats to supported conversions

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4761
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17435>

vulkan/wsi: Fix structure chaining in wsi_create_buffer_image_mem

First, because we're using __vk_append_struct which attacks it on the
end, memory_wsi_info is modified even though it's const.  Make things
non-const so we aren't silently violating assumptions.  Also, we set a
pNext in memory_export_info which causes a loop in the pNext chain in
the handle_types != 0 case.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6826
Fixes: 124848bf9efa ("vulkan/wsi: Support tiled CPU images")
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17434>

vulkan/wsi/wayland: Only memcpy if the swapchain is actually software

Otherwise, we'll segfault. :-(

Fixes: aca545d616d8 ("vulkan/wsi/wayland: Use host pointer import when available")
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17434>

panfrost/ci: Disable T760 jobs

These keep timing out due to abusive jobs.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17433>

freedreno/a6xx: Do clip-plane lowering in backend

Our GS-lowered-to-quasi-VS confuses core nir passes, so handle clip-
plane lowering ourself.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: Handle driver-params in GS/DS

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno: rename ir3_emit_driver_params()

Driver-params are not VS specific, rename helper to reflect this.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Fix GS clip-plane lowering

And also handle tess.  In all cases, we want to use the VS lowering pass
on the last geometry stage.  We don't make a special exception for GS
like other drivers, because GS gets lowered into a quasi-VS.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: Fix VS const packet size

Need to account for the PKT7 header.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: Fix indentation

Another victim of automated re-indenting being unaware of the semantics.
Re-indent this to put each dword of the packet on it's own line to make
the size of the packet more clear.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Add more tess varying slots

Fixes some piglits that I stumbled across by mistake.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Copy vars if needed on EndPrimitive()

If we didn't EmitPrimitive() then the shadow (old) outputs would not
get copied to the emit temps (to eventually be copied back to the real
outputs.  This isn't so bad except that means the realy vertex_flags
output has an undefined value.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Add copy_vars() helper

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/ir3: Don't lower_gs multiple times

At least with gallium, this can be called multiple times via
pipe_screen::finalize_nir().  But it is not designed to be called
multiple times, and can result in vertex_flags getting 'optimized'
away.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6720
Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

turnip: assert valid vertex_flag reg

If this somehow gets optimized out, the GS will run forever.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

freedreno/a6xx: assert valid vertex_flags reg

If this somehow gets optimized out, the GS will run forever.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17341>

intel/fs: Remove non-_LOGICAL URB messages

The _LOGICAL versions are lowered direct to SEND, so nothing can ever
generate these messages.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/fs: Lower URB messages to SEND

Before rebasing on top of Ken's split-SEND optimization (see !17018),
this commit just caused some scheduling changes in various tessellation
and geometry shaders.  These changes were caused by the addition of real
latency information for the URB messages.

With the addition of the split-SEND optimization, the changes
are... staggering.  All of the shaders helped for spills and fills are
vertex shaders from Batman Arkham Origins.  What surprises me is that
these shaders account for such a high percentage of the spills and fills
in fossil-db.  85%?!?

v2: Use FIXED_GRF instead of BRW_GENERAL_REGISTER_FILE in an assertion.
Suggested by Ken.

Tiger Lake, Ice Lake, and Skylake had similar results. (Ice Lake shown)
total instructions in shared programs: 20013625 -> 19954020 (-0.30%)
instructions in affected programs: 4007157 -> 3947552 (-1.49%)
helped: 31161
HURT: 0
helped stats (abs) min: 1 max: 400 x̄: 1.91 x̃: 2
helped stats (rel) min: 0.08% max: 59.70% x̄: 2.20% x̃: 1.83%
95% mean confidence interval for instructions value: -1.97 -1.86
95% mean confidence interval for instructions %-change: -2.22% -2.18%
Instructions are helped.

total cycles in shared programs: 859337569 -> 858636788 (-0.08%)
cycles in affected programs: 74168298 -> 73467517 (-0.94%)
helped: 13812
HURT: 16846
helped stats (abs) min: 1 max: 291078 x̄: 82.83 x̃: 4
helped stats (rel) min: <.01% max: 37.09% x̄: 3.47% x̃: 2.02%
HURT stats (abs)   min: 1 max: 1543 x̄: 26.31 x̃: 14
HURT stats (rel)   min: <.01% max: 77.97% x̄: 4.11% x̃: 2.58%
95% mean confidence interval for cycles value: -55.10 9.39
95% mean confidence interval for cycles %-change: 0.62% 0.77%
Inconclusive result (value mean confidence interval includes 0).

Broadwell
total cycles in shared programs: 904844939 -> 904832320 (<.01%)
cycles in affected programs: 525360 -> 512741 (-2.40%)
helped: 215
HURT: 4
helped stats (abs) min: 4 max: 1018 x̄: 60.16 x̃: 39
helped stats (rel) min: 0.14% max: 15.85% x̄: 2.16% x̃: 2.04%
HURT stats (abs)   min: 79 max: 79 x̄: 79.00 x̃: 79
HURT stats (rel)   min: 1.31% max: 1.57% x̄: 1.43% x̃: 1.43%
95% mean confidence interval for cycles value: -75.02 -40.22
95% mean confidence interval for cycles %-change: -2.37% -1.81%
Cycles are helped.

No shader-db changes on any older Intel platforms.

Tiger Lake, Ice Lake, and Skylake had similar results. (Ice Lake shown)
Instructions in all programs: 142622800 -> 141461114 (-0.8%)
Instructions helped: 197186

Cycles in all programs: 9101223846 -> 9099440025 (-0.0%)
Cycles helped: 37963
Cycles hurt: 151233

Spills in all programs: 98829 -> 13695 (-86.1%)
Spills helped: 2159

Fills in all programs: 128142 -> 18400 (-85.6%)
Fills helped: 2159

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/fs: Add _LOGICAL versions of URB messages

The lowering is currently fake.  It just changes the opcode from the
_LOGICAL version to the non-_LOGICAL version.

v2: Remove some rebase cruft.  's/gfx8_//;s/simd8_/' in
brw_instruction_name.  Both suggested by Ken.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/compiler: Move logical-send lowering to a separate file

brw_fs.cpp was 10kloc.  Now it's only 7.5kloc.  Ugh.

v2: Rebase on 9680e0e4a2d.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/eu: Validate some aspects of URB messages

If these checks had been in place previously, some bugs
that... eh-hem... practically took down the Intel CI would have been
caught earlier. *blush*

v2: Update to account for split sends.

v3: Add some more Gfx version checks.  Remove the redundant "src0 is a
GRF" check.  Both suggested by Ken.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

intel/compiler: Rename vec4 state URB opcodes to have VEC4_ prefix

An argument could be made that all stage-specific opcodes for vec4
stages should be prefixed with VEC4_ like the stage-agnostic opcodes.
I'll leave those additional sed jobs for another day.

    egrep -lr '(VS|GS|TCS)_OPCODE_URB_WRITE' src |\
    while read f; do
        sed --in-place 's/\(VS\|GS\|TCS\)_OPCODE_URB_WRITE/VEC4_\1_OPCODE_URB_WRITE/g' $f
    done

    egrep -lr 'T.S_OPCODE[_A-Z]*URB_OFFSETS' src |\
    while read f; do
        sed --in-place 's/\(T.S_OPCODE[_A-Z]*URB_OFFSETS\)/VEC4_\1/g' $f
    done

Suggested-by: Kenneth Graunke <kenneth@whitecape.org>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17379>

dzn: Add for condition to break nested loop

Fixes: d132ec92 ("dzn: Support native image copies when formats are compatible")
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17377>

dzn: Fix incompatible pointer type error affecting MSYS2 MINGW32
Suggested-by: Yonggang Luo <luoyonggang@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/6807

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17414>

ci/traces: piglit, be more verbose

Print more information about traces testing progress.

Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17416>

radv/ci: enable fossils testing for GFX1100

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/16447>

aco: use scratch_* for VGPR spill/reload on GFX9+

fossil-db (navi21):
Totals from 12 (0.01% of 162293) affected shaders:
Instrs: 122808 -> 122782 (-0.02%); split: -0.11%, +0.09%
CodeSize: 711248 -> 710788 (-0.06%); split: -0.16%, +0.10%
SpillSGPRs: 928 -> 831 (-10.45%)
SpillVGPRs: 1626 -> 1624 (-0.12%)
Latency: 4960285 -> 4932547 (-0.56%)
InvThroughput: 2574083 -> 2559953 (-0.55%)
VClause: 3404 -> 3402 (-0.06%)
Copies: 36992 -> 37181 (+0.51%); split: -0.05%, +0.56%
Branches: 3582 -> 3585 (+0.08%)
PreVGPRs: 3055 -> 3057 (+0.07%)

fossil-db (vega10):
Totals from 12 (0.01% of 161355) affected shaders:
Instrs: 124817 -> 124383 (-0.35%); split: -0.46%, +0.12%
CodeSize: 705116 -> 703664 (-0.21%); split: -0.44%, +0.23%
SpillSGPRs: 1012 -> 898 (-11.26%)
SpillVGPRs: 1632 -> 1624 (-0.49%)
Scratch: 201728 -> 200704 (-0.51%)
Latency: 6160115 -> 6266025 (+1.72%); split: -0.34%, +2.06%
InvThroughput: 6440203 -> 6544595 (+1.62%); split: -0.35%, +1.97%
VClause: 3409 -> 3423 (+0.41%)
Copies: 37929 -> 37748 (-0.48%); split: -1.16%, +0.69%
Branches: 3851 -> 3855 (+0.10%); split: -0.13%, +0.23%
PreVGPRs: 3053 -> 3055 (+0.07%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: use scratch_* for scratch load/store on GFX9+

fossil-db (navi21):
Totals from 52 (0.03% of 162293) affected shaders:
Instrs: 83190 -> 82145 (-1.26%)
CodeSize: 454892 -> 447260 (-1.68%); split: -1.68%, +0.00%
VGPRs: 4768 -> 4672 (-2.01%)
Latency: 1490887 -> 1487170 (-0.25%); split: -0.68%, +0.43%
InvThroughput: 935500 -> 933060 (-0.26%); split: -0.72%, +0.46%
VClause: 2715 -> 2632 (-3.06%); split: -4.53%, +1.47%
SClause: 1902 -> 1883 (-1.00%)
Copies: 8839 -> 8496 (-3.88%)
PreSGPRs: 2012 -> 1807 (-10.19%)
PreVGPRs: 3282 -> 3192 (-2.74%)

fossil-db (vega10):
Totals from 41 (0.03% of 161355) affected shaders:
Instrs: 35772 -> 35699 (-0.20%)
CodeSize: 187040 -> 186584 (-0.24%)
VGPRs: 4044 -> 4072 (+0.69%)
Latency: 243088 -> 242379 (-0.29%)
InvThroughput: 180301 -> 179783 (-0.29%)
VClause: 1204 -> 1216 (+1.00%)
SClause: 653 -> 637 (-2.45%)
Copies: 3736 -> 3704 (-0.86%); split: -0.88%, +0.03%
PreSGPRs: 1331 -> 1207 (-9.32%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: initialize scratch base registers on GFX9-GFX10.3

fossil-db (navi21):
Totals from 1142 (0.70% of 162293) affected shaders:
Instrs: 271636 -> 271974 (+0.12%)
CodeSize: 1532020 -> 1533792 (+0.12%)
Latency: 7484066 -> 7485698 (+0.02%)
InvThroughput: 4048824 -> 4049579 (+0.02%)
SClause: 4171 -> 4212 (+0.98%)
PreSGPRs: 11203 -> 12276 (+9.58%)

fossil-db (vega10):
Totals from 3327 (2.06% of 161355) affected shaders:
Instrs: 257413 -> 257601 (+0.07%)
CodeSize: 1424244 -> 1425372 (+0.08%)
Latency: 8598402 -> 8600466 (+0.02%)
InvThroughput: 7906335 -> 7908234 (+0.02%)
SClause: 4932 -> 4973 (+0.83%)
PreSGPRs: 22010 -> 25405 (+15.42%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: treat flat-like as vmem in some scheduling heuristics

fossil-db (navi21):
Totals from 12 (0.01% of 162293) affected shaders:
Instrs: 48754 -> 48762 (+0.02%)
CodeSize: 267092 -> 267124 (+0.01%)
Latency: 1293798 -> 1292303 (-0.12%); split: -0.12%, +0.00%
InvThroughput: 854599 -> 853578 (-0.12%)
VClause: 1623 -> 1619 (-0.25%)
SClause: 1187 -> 1188 (+0.08%); split: -0.08%, +0.17%

fossil-db (vega10):
Totals from 1 (0.00% of 161355) affected shaders:
Latency: 18720 -> 18848 (+0.68%)
InvThroughput: 5775 -> 5776 (+0.02%)
SClause: 12 -> 11 (-8.33%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: include scratch/global in VMEM WAW optimization

fossil-db (navi21):
Totals from 2 (0.00% of 162293) affected shaders:
Instrs: 4788 -> 4785 (-0.06%)
CodeSize: 25884 -> 25872 (-0.05%)
Latency: 255008 -> 252950 (-0.81%)
InvThroughput: 170005 -> 168633 (-0.81%)
VClause: 206 -> 205 (-0.49%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: avoid WAW hazard with BVH MIMG and other VMEM

According to LLVM, image_bvh64_intersect_ray does not write results in
order with other VMEM instructions.

fossil-db (navi21):
Totals from 7 (0.00% of 162293) affected shaders:
Instrs: 39978 -> 39985 (+0.02%)
CodeSize: 219356 -> 219384 (+0.01%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: refactor VGPR spill/reload lowering

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: handle subtractions in parse_base_offset

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>

aco: combine additions and constants into scratch load/store

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/17079>