venus: fallback to passthrough pci bus info when not natively supported

This avoids an assert with VN_DEBUG=all on virtgpu backend.

Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20383>

r600/sfn: Fix possible dest channels for variable size dot

Fixes:  357e5fac9953b26eedc8819ab528b981be6e1b69
    r600/sfn: Use variable length DOT on Evergreen and Cayman

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20398>

radv: Fix uninitalized struct for trap handler

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20354>

r600/sfn: Don't try to re-use the iterator when uses is updated

It seems on libc++ the iterator is invalidated when an element is removed
from the set, so make sure that we don't implicitely use the old,
invalidated iterator in the range based - open code the loop using while
instead.

Fixes: f3415c (r600/sfn: copy propagate register load chains)

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7931

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20394>

radv: Enable EXT_mesh_shader by default on GFX10.3.

Only enable mesh+task shaders when IBs and gang submit are enabled.
We won't support gang submit with noibs.

Also remove the RADV_PERFTEST=ext_ms option.
Side note, GFX11 task/mesh support is still a TODO.

Don't skip the CTS tests which require GFX->ACE synchronization.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv/amdgpu: Remove cs_add_buffers because it's no longer necessary.

This function was added because previously ACE and GFX work was
submitted separately and we needed to make sure they both use the
same BOs. Now they are part of the same submission so this
function is not necessary anymore.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv/amdgpu: Remove command submission with scheduled dependencies.

These are no longer used by any part of RADV, so we
can just safely delete it.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Add gang wait preambles and postambles to ensure correct behaviour.

Add new preambles and postambles for synchronizing gang members in a
gang submission using semaphores.
These semaphores are both located in a small BO.

Gang wait preambles:

- gang leader writes 1 to a semaphore
- gang member waits for it to be written

When task shaders are used, make sure ACE waits until GFX starts to execute.
Userspace is required to emit this wait to make sure it behaves correctly
in a multi-process environment, because task shader dispatches are not
meant to be executed on multiple compute engines at the same time.

Gang wait postambles:

- gang member writes 1 to a semaphore
- gang leader waits for it to be written

This ensures that the gang leader waits for the whole gang,
which is necessary because the kernel signals the userspace fence
as soon as the gang leader is done, which may lead to bugs because the
same command buffers could be submitted again while still being executed.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Remove ACE->GFX wait at the end of each command buffer.

This is now handled in the queue submission code so is not necessary.
However, keep the semaphore for future use.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Rename radv_update_ace_preambles to radv_update_gang_preambles.

The new name reflects more accurately what this is.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Implement gang submit in radv_queue_submit_normal.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Update ACE preambles in radv_queue_submit_normal.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Refactor command buffer handling in radv_queue_submit_normal.

Move processing of the command buffer array inside the loop that
splits a submission.

We now also add the perf counter lock/unlock to each submission
instead of just the first and last.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Remove radv_queue_submit_with_ace and disable task/mesh shaders.

It would be too difficult to keep the radv_queue_submit_with_ace
function while also refactoring the radv_queue_submit_with_ace function,
so let's delete it first.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Enable gang submit on the null winsys.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

ac: Add has_gang_submit field to ac_gpu_info.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv/amdgpu: Allow using more than 1 preamble with cs_submit.

Add the ability to have multiple preambles,
except for the sysmem (NOIBS/GFX6) code path which still only allows 1.

This is necessary because with gang submit we will need a way to submit
a preamble to different queues at the same time.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv/amdgpu: Add IP type to IB info and use it when submitting.

This is necessary for supporting gang submit.

With gang submit, the kernel now allows us to submit multiple IBs
with different IP types. Therefore, RADV will also need to group
various CSs with different IP types together and remember the IP
type of each CS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv/amdgpu: Introduce a custom struct for submitted BO info.

We are going to need additional data which is not present in
the currently used struct.

This commit just adds the new struct but does not yet add
new fields to it.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv/amdgpu: Set fallback submit request ip_type from the last CS.

When using gang submit, the last IB is considered the "gang leader"
and its IP type will determine which fence to signal when the
submission is done. Therefore, use the last CS to set the IP type.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv/amdgpu: Change extra_cs to array in get_bo_list.

This is necessary because we are going to want to allow using
more than just 1 preamble.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Return correct error code when failed to allocate CS array.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Determine use_ace in radv_update_preambles.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Rename uses_perf_counters to use_perf_counters.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Stylistic changes in radv_queue_submit_normal.

Move some variables, add const, and fix a comment.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Destroy mesh shader scratch ring in radv_queue_state_finish.

Fixes: 0280b526d58e85d65b53d3f9c8b0f7364d853751
Signed-off--by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

radv: Remove trailing whitespace again.

Also ran clang-format on the affected line.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20010>

lvp: fix load_helper_invocation not being lowered

nir_lower_is_helper_invocation should be after nir_lower_system_values
to handle possible nir_intrinsic_is_helper_invocation which may be
produced by nir_lower_system_values.

Happens with SPIR-V 1.6 for which gl_HelperInvocation is translated into
"BuiltIn HelperInvocation" + "Volatile", which nir_lower_system_values
translates into is_helper_invocation.

Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19677>

tu: fix load_helper_invocation not being lowered

nir_lower_is_helper_invocation should be after nir_lower_system_values
to handle possible nir_intrinsic_is_helper_invocation which may be
produced by nir_lower_system_values.

Happens with SPIR-V 1.6 for which gl_HelperInvocation is translated into
"BuiltIn HelperInvocation" + "Volatile", which nir_lower_system_values
translates into is_helper_invocation.

Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19677>

nir/nir_lower_is_helper_invocation: Lower helper invocation if required

nir_lower_is_helper_invocation lowers intrinsic_is_helper_invocation
and uses load_helper_invocation (which is lowered by nir_lower_system_values).
While nir_lower_system_values may lower SYSTEM_VALUE_HELPER_INVOCATION
into intrinsic_is_helper_invocation.

So they depend on each other. Break the dependency by making
nir_lower_is_helper_invocation aware of lower_helper_invocation option
and emitting lowered load_helper_invocation when required.

Happens with SPIR-V 1.6 for which gl_HelperInvocation is translated into
"BuiltIn HelperInvocation" + "Volatile", which nir_lower_system_values
translates into is_helper_invocation.

Signed-off-by: Danylo Piliaiev <dpiliaiev@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19677>

anv: Ensure we clear ANV_PIPE_PSS_STALL_SYNC_BIT on flush

Add the PSS stall bit to ANV_PIPE_STALL_BITS so that it get's cleared on
flush.

Fixes: f3c62973 ("anv,iris: PSS Stall Sync around color fast clears")

Signed-off-by: Rohan Garg <rohan.garg@intel.com>
Reviewed-by: Nanley Chery <nanley.g.chery@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20317>

util: Fixes os_get_option_cached with bool options_tbl_exited

Fixes: cdad035cfd9 ("util: Add function debug_get_option_cached os_get_option_cached")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7922

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20386>

d3d12: Don't flush indiscriminately for query result accumulation

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20217>

d3d12: Fix race condition when getting query results

Before, when an application called into d3d12_query_result, and
the results were not ready to be read, a flush-and-wait would
be attempted via synchronized mapping of the query result resource.

This can end up calling close/execute on the command list while it is
already being executed by the driver thread.

With the current fence value attached to the query, we now wait
for completion if necessary and then map the resource unsynchronized, or
return false if the result is not ready and wait == false.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20217>

d3d12: Move query structs to header

Also adds `fence_value` to `d3d12_query` to track which batch of work
this query exists in and must be waited on.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20217>

vulkan: update headers/registry to 1.3.238 for video decode

This moves the video decoding to KHR and out of beta.

Acked-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Acked-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20384>

vulkan/util: handle LUNARG in the list of special prefixes

The latest headers update has a lunarg extension that needs this.

Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Acked-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20384>

vulkan/cmd_queue: update generator to handle guarded types.

The latest includes/registry have some pNext structs that need
guarding spearately to their initial structs, so we have to track
the define guards better and use them in the generated code

For example:
VkVideoEncodeRateControlInfoKHR extends VkVideoCodingControlInfoKHR

However the first struct is protected by BETA, but the second isn't
the generator was always generating unguarded code for all structs

Reviewed-by: Jason Ekstrand <jason.ekstrand@collabora.com>
Acked-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20384>

turnip: Lazily call tu6_emit_descriptor_sets() at draw time.

This lets us batch up the state changes from multiple
vkCmdBindDescriptorSets, which ANGLE and zink will both do in a single
draw.

Improves ANGLE (sysmem) driver_overhead perf by 5.18806% +/- 1.03444% (n=5).
Improves ANGLE aztec_ruins_high perf by ~.3%. (clear result in the graph,
but the screen went to sleep mid way through and so it was high variance)

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20084>

lavapipe: Enable VK_KHR_shader_non_semantic_info.

Already implemented in common code.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7907

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20353>

ci: move the CI handling scripts into bin/ci

Any change in .gitlab-ci/ directory triggers an entire CI run,
which is unnecessary for just changing the developers' scripts.

Added symlink to `.gitlab-ci/bin` to keep original invocation compatibility.

Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20371>

ci: ci_run_n_monitor.py: fix args.rev leftover

Fixes: a8b6b2367efc ("ci: allow omitting of --rev for ci_run_n_monitor.py")

Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20371>

iris: Add BO_ALLOC_PLAIN flag

We had iris_bo_alloc() call with the wrong flags value, it was
mistaken by aligment, see commit 68652dca0c28.

To avoid such mistakes in future here adding BO_ALLOC_PLAIN so
iris_bo_alloc() calls have a more descriptive flag parameter when
there is no special allocation request.

While at it, also standardizing unsigned as the type for this flag.

No behavior changes here.

Reviewed-by: Paulo Zanoni <paulo.r.zanoni@intel.com>
Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19857>

radv/bvh: Handle inactive triangles and AABBs

Reviewed-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20325>

radv/bvh: Move leaf node initialization into separate functions

Reviewed-by: Tatsuyuki Ishi <ishitatsuyuki@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20325>

ac/llvm: remove deref image/texture code

They have been lowered to bindless ones in nir.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

ac,radv,radeonsi: remove unused param of load_sampler_desc abi

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

ac/llvm: remove samples_identical nir to llvm code

It has been lowered in nir.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

ac/llvm: remove implicite ms texture fmask handling

It has been lowered to fragment_mask_load_amd in nir.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

ac/llvm: remove image/texture descriptor_amd nir to llvm code

They have been lowered in nir.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

radeonsi: replace llvm resource code with nir lower

Port from ac_nir_to_llvm.c and si_shader_llvm_resource.c.

Due to need waterfall of llvm backend, we can't get bind-texture
descriptor directly in nir. So we keep load_sampler_desc abi only
for bind-texture index to desc.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

nir/divergence_analysis: add missing intrinsics

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Singed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

ac/llvm: add disable_aniso_single_level abi

RADV use dri option to enabled this for some apps, but it's
done in nir lower currently. I'm afraid it still needs this
option to handle the non-uniform case as desc is loaded in
llvm.

radeonsi always enable this for bind-textures.

radeonsi will lower all bind-textures to bindless-textures,
and only bind-textures use desc index, so add this abi for
bindless desc index path.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

nir: lower image add lower_to_fragment_mask_load_amd option

Like lower_to_fragment_fetch_amd option in lower tex,
this is for radeonsi to lower MS image ops.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

nir: add image fragment mask load intrinsic

Like nir_texop_fragment_mask_fetch_amd, this is used to load multi
sample image fmask data for AMD GPU.

We will lower multi sample image load and samples_identical intrinsics
to use it latter for radeonsi. RADV does not need this because it
always expand fmask images before dispatch compute shader.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

radeonsi: replace llvm load_ssbo abi with nir lower

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

ac/llvm: check load_ssbo present before call it

radeonsi will remove it later.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

radeonsi: add si_nir_lower_resource pass

Replace the load_ubo abi.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18666>

radv/rt: Refactor exiting PLOC

The previous approach was susceptible to sync hazards, causing hangs in Doom Eternal.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7897
Fixes: 271865373 ("radv: Add PLOC shader")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20377>

radv/rt: Execute memory barrier before updating the phase end count

We want to be sure that the values were updated before letting other invocations continue.

Fixes: 271865373 ("radv: Add PLOC shader")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20377>

radv/rt: PLOC fixes for active_leaf_count

Fixes: 271865373 ("radv: Add PLOC shader")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20377>

util/tools: add the tool that generates perfect hash functions for GLenums

The big comment explains it.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

mesa: optimize _mesa_bytes_per_vertex_attrib using a hash-based translation

This reduces time spent in gl*Pointer for viewperf.

Generated by find_hash_func.c.

Reviewed-by: Rob Clark <robclark@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

mesa: compute _PipeFormat and _ElementSize after checking if the format changed

Split _mesa_set_vertex_format into 2 functions and set the derived fields
after we know that the format is different.

This skips _PipeFormat and _ElementSize recomputation every time we get
gl*Pointer with the same format.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

mesa: move user-specified vertex format fields into gl_vertex_format_user

to be used in the next commit

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

mesa: change gl_vertex_format::Format to bool Bgra to free bits

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

st/mesa: simplify prepare_indexed_draw

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

st/mesa: optimize index buffer reference counting in st_indirect_draw_vbo

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

mesa: set pipe_draw_info::index::resource directly and remove gl_bo

The motivation is to remove branching from prepare_indexed_draw
and remove the unsafe hack in vbo_save_playback_vertex_list.

There is some duplication that is not identical in all 3 cases.

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

st/mesa: fold GLThread.enabled into pin_thread_counter to skip that checking

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

st/mesa: use u_bit_scan64 only for 64-bit CPUs in st_validate_state

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20287>

docs: update to latest venus driver support

Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20310>

venus: fix VK_EXT_pci_bus_info support

Natively support it for virtgpu while passthrough renderer support for
vtest. This is a must support to avoid prime blit in common wsi.

Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20310>

venus: sync protocol fixes and support for VK_EXT_pci_bus_info

Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20310>

etnaviv: Do expose 2D texture support on pre-halti GPUs

Commit d08bd9a8d8b ("etnaviv: don't expose array and 3D texture support on pre-halti GPUs")
started returning 0 from PIPE_CAP_MAX_TEXTURE_2D_SIZE as well due to switch
case fallthrough. Reinstate the behavior of PIPE_CAP_MAX_TEXTURE_2D_SIZE,
while at the same time, retain the new behavior introduced by commit
d08bd9a8d8b , Otherwise not even kmscube displays the spinning cube,
weston does not display desktop and so on.

Triggered on "Vivante GC600 rev 4653" , i.MX8M Mini .

Fixes: d08bd9a8d8b ("etnaviv: don't expose array and 3D texture support on pre-halti GPUs")
Closes: #7898
Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Signed-off-by: Marek Vasut <marex@denx.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20373>

panfrost: Add architecture description XML for v10

Add the GenXML hardware description for Mali architecture v10, as implemented in
Mali-G610. This is not 100% complete but it should be good enough for parity
with v9.

The XML itself is forked off of v9, with all Job Managerisms replaced with
CSFisms. This notably includes a large number of new structures defining the
instructions that run on the Command Execution Unit (CEU).

This is the first step towards supporting Mali-G610 (i.e. RK3588) upstream. Next
up will be pandecode support.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20360>

freedreno/drm: Short-circuit kernel for timeout=0 waits

We already know from the userspace fence whether the fence has signaled
or not.  No need to do a syscall if we aren't going to wait.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm/virtio: Guest side fence waits

Now that fd_fence and fd_submit_fence are unified, we can wait on the
fence fd rather than needing to poll the host.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm/virtio: More tracepoints

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno: Limit cached staging BOs to PIPE_MAP_READ

WC is fine for staging upload buffers, and will let us hit the default
heap.  Probably introducing a separate cached-coherent heap is overkill,
and given that staging uploads are common (but downloads uncommon, out-
side of dEQP/piglit), it is better to just ask for WC.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm/virtio: Remove prealloc_list

Now that we are allocating cmdstream from the heap, we shouldn't need
this.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Add sub-allocator

Add a heap that we can use for allocations of small mappable buffers.
This avoids the churn of mmap/unmap, which is especially expensive in
a VM.  It also allows packing more smaller allocations together in a
page, which is useful for PIPE_BUFFERs (which are also mappable).

This avoid jank caused by the overhead of setting up or tearing down
guest mappings when running in a VM.  And also significantly reduces
the # of BOs referenced on a submit.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

util/vma: Add "nospan"

Add a way to request that allocations do not span a specified PoT
boundary.  This can be used, for ex, to ensure that allocations do
not span 4GB boundaries to work around hw/fw bugs.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Merge in_fence_fd's

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Move no_implicit_sync accounting

No need to duplicate this.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Drop fd_bo_fence

Remove the last fence-like-struct that isn't fd_fence.  Now that
fences are refcnt'd we can just attach them directly to BOs.  This
will simplify busy-ness tracking for suballoc BOs.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Re-enable submit fence merging

Now that we (a) create the fence ourselves, returning a reference from
fd_submit_flush(), and (b) refcnt the fences, we can now merge submits
with on out-fence by just attaching the same fence to all the submits.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Simplify deferred submit flushing

Once we are sharing an out-fence for multiple deferred/merged submits
the complicated logic in fd_pipe_sp_flush() no longer works (or makes
sense), so just simplify all the paths into a single helper.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Return fence from submit flush

This moves away from embedding the submit fence inside the pipe fence,
which lets us start refcnt'ing the fence.  This will enable several
cleanups and improvements:

1. Get rid of fd_bo_fence, and just have fd_bo hold pending fd_fence
   refs instead, which will be needed for cpu_prep implementation of
   sub-allocated buffers.
2. For merged submits, we can just return a new reference to an
   existing fence.

Note that this temporarily defeats submit-merging, which will be
fixed (and improved) in a following commit.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno: Rename fd_fence -> fd_pipe_fence

So we don't have namespace clashes in the next commit.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Combine fd_fence and fd_submit_fence

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Replace shared/nosync fields

These are redundent with the alloc_flags.  A following commit will map
alloc_flags directly to bo heap that should be used, and adding the
FD_BO_SHARED bit when a BO is exported neatly avoids returning it to
the pool to be potentially erroneously reused.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Reorg BO alloc flags

And drop the unneeded _FD_BO_RING flag.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Invert BO destruction

Sub-alloc'd heap BOs will need to do something a bit more special, ie.
not immediately cleanup fences (because those will be needed to know
when the BO is actually idle and vma node can be released), not tear
down the mmap (because the BO doesn't own it), and not close the handle
(also because the BO doesn't own it).

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Shuffle around BO init

Suballocated heap BOs will bypass the rest of bo_new() (as they skip the
BO cache, don't get an entry in the handle table, etc).  So move more of
the initialization into fd_bo_init_common().

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Drop pipe ref to device

The context and screen already holds refs to the device.  But since
fd_bo_fence holds a ref to the pipe, anyone anywhere leaking a single
bo with unwaited fences could cause the device to leak indirectly.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Get rid of fd_bo_del_locked()

This moves the table_lock into a small critical section in the BO delete
path when we are actually removing the entries from the handle/name
tables, so finally table_lock isn't the big-bo-lock!

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Rework batch BO delete

Get rid of the per-device table of deferred handles, and instead keep
this on the stack so that no additional locking is needed.  This will
simplify getting rid of table_lock in the bo delete path.  For the BO
cache, add fd_bo_del_list_nocache() which works like fd_bo_del_array()
except that it bypasses returning BOs to the cache.  This gets rid of
the BO cache calls to private fd_bo fxns (which is useful now that the
bo_del/close_handles() dance has become a bit more complicated).

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Start prying apart table_lock

We want to get back to the point where table_lock is only doing what
it's named for (ie. protecting handle/name tables).  This is the first
step.  Add new locks to protect fencing and the bo caches.  A single
global lock for fences, to avoid atomic instructions in the loop per
submit bo attaching fences.

This couldn't be broken appart into different commits, otherwise there
would have been an ABBA deadlock in BO cache's call to fd_bo_state()
to check if a BO is idle.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>

freedreno/drm: Simplify cleanup_fences()

To prepare for moving fence state out from under table_lock, simplify
cleanup_fences() and stop trying to re-use it in the bo destruction
path (which otherwise wouldn't need the fence lock, since there is
no other owner of the bo ref at that point, and simply needs to drop
pipe refs).

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20263>