zink: use info.has_transform_feedback_varyings to determine xfb enablement

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: rename variable in update_so_info()

be more consistent

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: evaluate existing slot map during program init and force new map as needed

if the number of explicit xfb outputs or new varyings added to the existing size
of the slot map would cause an overflow, we have to force a new slot map to
ensure that everything fits

this means iterating all the stages which can produce new varyings and calculating
all the slots required in order to compare against the max size available

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: handle direct xfb output from output variables

if an entire variable is being dumped into an xfb buffer, there's no need
to create an explicit xfb variable to copy the value into, and instead
the xfb attributes can just be set normally on the variable

this doesn't work for geometry shaders because outputs are per-vertex

fixes all KHR-GL46.enhanced_layouts xfb tests

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: stop allocating xfb slot map

this can just be inlined since it's a small static size

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: run more nir passes for tess shaders

running nir_lower_io_arrays_to_elements_no_indirects for only some stages
breaks location-setting for the stages which don't run it when
e.g., dmat2x3 variables are sometimes split across locations and
sometimes jammed into a single location (TCS I'm looking at you)

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: fix location usage for explicit xfb outputs

ensure that this accurately handles multi-slot emission

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: fix slot mapping for fat io variables

big types like dmat2x3 need multiple slots, and trying to jam them into
single slots breaks everything

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: fix streamout emission for super-enhanced layouts

if we get some crazy matrix types in here then we need to ensure that
we accurately unwrap them and copy the components

fixes KHR-GL46.enhanced_layouts.xfb_stride

Fixes: 1b130c42b8d ("zink: implement streamout and xfb handling in ntv")

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

zink: remove ntv streamout assert

this was added during review, but it was never correct and just crashes
valid cases like streamout from a mat3x4 type

Fixes: b6f8f3a3ba4 ("zink: fix streamout for clipdistance")

Reviewed-by: Dave Airlie <airlied@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9271>

microsoft/compiler: Don't separate phis while inserting upcasts

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4414
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9464>

nir: Add a nir_after_instr_and_phis helper

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9464>

intel/mi_builder: Fix a couple of #ifs

All this does is remove a field on Gen7 and stop asserting on it.  No
actual functional change.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9467>

intel/mi_builder: Fix some indentation

This got lost in the rebase on top of the s/gen_mi_/mi_/ change

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9467>

anv: Add mem heap/type support for local-mem

This will take effect in future patches when we are able to query the
kernel to set device->vram.size to a non-zero size.

Builds on Sagar's ("anv: Query memory region info") patch, and
re-organizes things as recommended by Lionel (and Jason).

Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9324>

anv: Put cache memory type first on non-llc platforms

Suggested-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9324>

anv: Restructure mem heap/type init code

Just treat the llc and non-llc paths as separate cases. This will also
help when adding the local memory setup.

Signed-off-by: Jordan Justen <jordan.l.justen@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9324>

anv: Add anv_memregion structure

Signed-off-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9324>

spirv: Update a couple of comments in variable handling

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9440>

spirv: Explicitly break when finished handling SpvDecorationBuiltIn

When tyding up this section in 1e5b09f42f6 ("spirv: Tidy some repeated
if checks by using a switch statement.") the break got lost.  It is
not a real problem because the next case just break, but better to
have it explicitly here instead of a FALLTHROUGH.

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9440>

spirv: Reuse nir_is_per_vertex_io()

Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9440>

nir-to-tgsi: Fix handling of partial writemasks on SSA/REG decls.

In nouveau's PBO path with GS support and no VS layer export, we got:

        intrinsic store_output (ssa_1, ssa_0) (0, 15, 0, 160, 128) /* base=0 */ /* wrmask=xyzw */ /* component=0 */ /* src_type=float32 */ /* location=0 slots=1 */     /* out_pos */
        [...]
        vec3 32 ssa_4 = mov ssa_3.xxx
        intrinsic store_output (ssa_4, ssa_0) (0, 4, 0, 160, 128) /* base=0 */ /* wrmask=z */ /* component=0 */ /* src_type=float32 */ /* location=0 slots=1 *//* out_pos */

The mov's SSA value we would decide we could store directly to the output,
since nothing else used it.  However, the store has a writemask, and the
ALU op was stomping over it instead of ANDing with the output decl's
existing writemask.

Fixes: f79f382c81f8 ("nir_to_tgsi: Store directly to TGSI outputs when possible.")
Closes: #4380
Tested-by: Ilia Mirkin <imirkin@alum.mit.edu>
Acked-by: Ilia Mirkin <imirkin@alum.mit.edu>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9376>

nir: Make nir_ssa_def_rewrite_uses_after take an SSA value

This replaces the new_src parameter of nir_ssa_def_rewrite_uses_after()
with an SSA def, and rewrites all the users as needed.

Acked-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9383>

nir: Make nir_ssa_def_rewrite_uses take an SSA value

This commit replaces the new_src parameter of nir_ssa_def_rewrite_uses()
with an SSA def, removes nir_ssa_def_rewrite_uses_ssa(), and rewrites
all the users as needed.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Acked-by: Alyssa Rosenzweig <alyssa@collabora.com>
Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9383>

nir: Add and use a new nir_ssa_def_rewrite_uses_src helper

This is currently an alias for nir_ssa_def_rewrite_uses but we move all
the instances which used it to write a non-SSA source to the newly named
helper.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Acked-by: Alyssa Rosenzweig <alyssa@collabora.com>
Acked-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9383>

intel/mi_builder: Add control-flow support

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/mi_builder: Return an address from __gen_get_batch_address

While we're here, add __gen_get_batch_address declarations to more files
because we're about to start requiring it on all GFX 12.5+.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/mi_builder: Use softpin for tests on gen8+

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/batch_decoder: Don't follow predicated MI_BATCH_BUFFER_START

The stuff after these may be executed so we want to decode it too.

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

genxml: Clean up MI_SET_PREDICATE

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/mi_builder: Add load/store_offest on GFX 12.5+

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/mi_builder: Support inverted values in mi_store

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/mi_builder: Added support for command streamer shift operations

Add logical shift left and right operations support to mi_builder.

v1:
- Add GEN_GEN > 12 check (Jordan Justen)
- Add gen_mi_has_shift function (Jordan Justen)
- Fix commit title (Jordan Justen)

v2 (Jason Ekstrand):
- Add _imm versions of all of them
- Better handle corner-cases in _imm helpers
- Handle the power-of-two limitation for _imm versions
- Add tests

Signed-off-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/mi_builder: Add ieq/ine helpers

Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

intel/mi_builder: Use AddCSMMIOStartOffset for LRI

In 06cf838cbdcb03184 we started using the AddCSMMIOStartOffset feature
on Gen11+ but we missed one place.

Fixes: 06cf838cbdcb "intel/mi_builder: Support gen11 command-streamer..."
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9445>

freedreno/cffdec: Use rb trees for tracking buffers

Gets rid of the arbitrary size limitation, and should make decoding
faster with many buffers.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8838>

ac/surface: select best swizzle mode for 3D sampler performance

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9448>

driconf: add performance tweaks for viewperf

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9449>

aco: Fix vector::reserve() being called with the wrong size

The container is moved from before and hence returns size 0. To get the
correct value, the new instruction container must be used instead.

This was flagged by clang-tidy. The fixed call still triggers the
corresponding diagnostic, hence this change silences it by adding a
redundant clear() after move.

Fixes: 7f1b537304d ("aco: add new NOP insertion pass for GFX6-9")
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9432>

nir/lower_viewport_transform: Allow geom/tess

This pass needs to run on the last shader in a pipeline writing
gl_Position. In GLES2, that's always the vertex shader, but in ES3.2, it
can be a geometry or tessellation shader. The shared code works the same
in this case, just make the assert more generous.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Erico Nunes <nunes.erico@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9444>

pan/bi: Treat +DISCARD.f32 as message-passing

Likely errata, matches blob's handling. Closes #4387

total nops in shared programs: 86266 -> 86272 (<.01%)
nops in affected programs: 347 -> 353 (1.73%)
helped: 1
HURT: 2

total clauses in shared programs: 20813 -> 20833 (0.10%)
clauses in affected programs: 343 -> 363 (5.83%)
helped: 0
HURT: 20
Clauses are HURT.

total quadwords in shared programs: 91572 -> 91588 (0.02%)
quadwords in affected programs: 1322 -> 1338 (1.21%)
helped: 1
HURT: 14
Quadwords are HURT.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Tested-by: Icecream95 <ixn@disroot.org>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9446>

pan/bi: Set clause_state.message conservatively

Accidentally prevented scheduling message-passing instructions to
anywhere but the last ADD of a clause.

total nops in shared programs: 86280 -> 86266 (-0.02%)
nops in affected programs: 1609 -> 1595 (-0.87%)
helped: 9
HURT: 4
Inconclusive result (value mean confidence interval includes 0).

total clauses in shared programs: 20993 -> 20813 (-0.86%)
clauses in affected programs: 3488 -> 3308 (-5.16%)
helped: 116
HURT: 0
Clauses are helped.

total quadwords in shared programs: 91697 -> 91572 (-0.14%)
quadwords in affected programs: 12257 -> 12132 (-1.02%)
helped: 53
HURT: 2
Quadwords are helped.

Fixes: f0c0082ab01 ("pan/bi: Schedule blocks")
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Tested-by: Icecream95 <ixn@disroot.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9446>

pan/bi: Mark message-passing sources/dests live

More general, same data race.

Fixes: 44726101d1e ("pan/bi: Don't fill garbage")
Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Tested-by: Icecream95 <ixn@disroot.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9446>

st/nine: Set default dynamic_texture_workaround to true

Now the texture virtual memory usage is less of a problem,
we can use this workaround permanently.

In the spirit of the API it's certainly not the proper way
of implementing DYNAMIC textures (it seems they are ok
to have hidden copies in driver managed memory, but not have
virtual addressing space reduced), but it makes sense for us,
both performance wise, and to avoid bugs.

Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377>

st/nine: Add driconf option to limit texture memory

Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377>

st/nine: Control the memfd virtual limit

Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377>

st/nine: Use the texture memory helper

Switch to the new texture RAM memory API.

Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377>

st/nine: Add RAM memory manager for textures

On 32 bits, virtual memory is sometimes too short for apps.
Textures can hold virtual memory 3 ways:
1) MANAGED textures have a RAM copy of any texture
2) SYSTEMMEM is used to have RAM copy of DEFAULT textures
   (to upload them for example)
3) Textures being mapped.

Nine cannot do much for 3). It's up to driver to really unmap textures
when possible on 32 bits to reduce virtual memory usage.

It's not clear whether on Windows anything special is done for
1) and 2). However there is clear indication some efforts have
been done on 3) to really unmap when it makes sense.

My understanding is that other implementations reduce the usage
of 1) by deleting the RAM copy once the texture is uploaded
(Dxvk's behaviour is controlled by evictManagedOnUnlock).

The obvious issue with that approach is whether the texture is
read by the application after some time. In that case,
we have to recreate the RAM backing from the GPU buffer.

And apps DO that. Indeed I found that for example Mass Effect 2
with High Texture mods (one of the crash case fixed by this patch serie),
When the character gets close to an object, a high res texture and replaces
the low res one. The high res one simply has more levels, and the game seems
to optimize reading the high res texture by retrieving the small-resolution
levels from the original low res texture.
In other words during gameplay, the game will randomly read MANAGED textures.
This is expected to be fast as the data is supposed to be in RAM...

Instead of taking that RAM copy eviction approach, this patchset
proposes a different approach: storing in memfd and release the
virtual memory until needed.

Basically instead of using malloc(), we create a memfd file
and map it. When the data doesn't seem to be accessed anymore,
we can unmap the memfd file.
If the data is needed, the memfd file is mapped again.
This trick enables to allocate more than 4GB on 32 bits apps.

The advantage of this approach over the RAM eviction one,
is that the load is much faster and doesn't block the GPU.

Of course we have problems if there's not enough memory to map the
memfd file. But the problem is the same for the RAM eviction approach.

Naturally on 64 bits, we do not use memfd.

Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377>

st/nine: Add new function to know if we are the worker

This will be useful in a later patch

Signed-off-by: Axel Davy <davyaxel0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9377>

mesa: fix fbo attachment size check for RBs, make it trigger in ES2

Makes dEQP-GLES2.functional.fbo.completeness.size.distinct pass.

Signed-off-by: Ilia Mirkin <imirkin@alum.mit.edu>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9441>

mesa: fix conditions for fp16 render format eligibility

GLES3 adds all of these, but they're also available in GLES2 with an
ext.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4400
Signed-off-by: Ilia Mirkin <imirkin@alum.mit.edu>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9441>

tegra/context: unwrap indirect_draw_count as well

Fixes: 22f6624ed318 "gallium: separate indirect stuff from pipe_draw_info - 80 -> 56 bytes"
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9425>

tegra/context: fix regression in tegra_draw_vbo

We should only pass in a new indirect_info object if we actually set valid
values in it.

Fixes: abe8ef862fe5 "gallium: make pipe_draw_indirect_info * a draw_vbo parameter"
Signed-off-by: Karol Herbst <kherbst@redhat.com>
Reviewed-by: Ilia Mirkin <imirkin@alum.mit.edu>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9425>

st/mesa: Update constants on alpha test change if it's lowered

nir_lower_alpha_test creates a uniform for the alpha reference value;
this needs to be updated when changing alpha test state.

Fixes: b1c4c4c7f53 ("mesa/gallium: automatically lower alpha-testing")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4390
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9439>

zink/ci: update results after layer extensions enabled in lavapipe

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9401>

lavapipe: enable EXT_shader_viewport_index_layer

This is already implemented afaik

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9401>

llvmpipe: add support for shader viewport layer

This should already be implemented just never enabled the CAP

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9401>

draw/prim_assembler: write correct decomposed primitive lengths

In order for shader viewport index to be calculated correctly,
the cliptest code needs proper primitive lengths to work out
the provoking vertex. I half fixed this before for GL4 but looks
like I didn't make it all the way.

This fixes:
dEQP-VK.draw.shader_viewport*

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9401>

draw: fix uses viewport index for tess eval shader

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9401>

vbo: Fix vbo_sw_primitive_restart for start > 0

Commit e99e7aa4 began passing start > 0 to indexed draw calls rather
than keeping start at 0 and manually advancing ib->ptr.  This should
work fine, however, there have been instances of software fallbacks
not handling things right.

vbo_sw_primitive_restart had a bug where it was ignoring "start" and
always calling find_sub_primitives with start = 0 and end = ib->count.
This meant that when start > 0, it was analyzing the wrong part of the
index buffer when finding subprimitives.

In theory, each _mesa_prim can have a different "start" value.  But
the code only calls find_sub_primitives once, because it wants to
map, analyze, and unmap the index buffer before calling ctx->Draw,
as some drivers don't support drawing with the index buffer mapped.

To handle this, we break vbo_sw_primitive_restart calls into sections
where "start" matches across all the primitives, similar to how I
handled the issue in tnl in commit bd6120f562d57e150aa2071f9108.

In the common case, start matches and we handle it in one pass anyway.

Fixes Piglit's primitive-restart VBO_COMBINED_VERTEX_AND_INDEX test
and KHR-GL33.pipeline_statistics_query_tests_ARB.functional_primitives_vertices_submitted_and_clipping_input_output_primitives
on Intel Ivybridge and older (which don't do arbitrary cut indices).

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4052
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9417>

zink: more and better debug printfs

Use debug_printf more consistently, normalize formatting a bit, and
trace a few more places you're likely to care about.

Reviewed-By: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9436>

r600/sfn: eliminate loading unused component loads from shared memory

LDS loads are quite expensive, so try to eliminate as many as possible

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9416>

radv: cache pipeline statistics

Applications rarely require them, but this improves fossil-db replay time.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9411>

radv,aco: remove aco_compiler_statistics

This removes a pointer from radv_shader_binary_legacy::data.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9411>

.mailmap: resolve duplicates for Yogesh Mohan Marimuthu

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Satyeshwar Singh

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Mun Gwan-gyeong

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Maya Rashish

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Matthias Lorenz

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Matthias Hopf

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Mark Menzynski

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Lin Johnson

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Jan Zielinski

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for James Xiong

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Indrajit Das

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Emmanuel Vadot

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Christopher Li

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: resolve duplicates for Icecream95

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

.mailmap: colapse duplicates for Timothy Arceri

Signed-off-by: Andres Gomez <agomez@igalia.com>
Acked-by: Erik Faye-Lund <erik.faye-lund@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/8837>

anv: fix MI_PREDICATE_RESULT write

This register is only 32bits.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 1952fd8d2ce905 ("anv: Implement VK_EXT_conditional_rendering for gen 7.5+")
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9428>

pan/bi: Implement fsin/fcos

Instead of lowering it in NIR, use the lookup tables as inputs to a
second-order Taylor expansion. shader-db results aren't amazing but keep
in mind this is without backend CSE yet.

total instructions in shared programs: 115913 -> 115707 (-0.18%)
instructions in affected programs: 3151 -> 2945 (-6.54%)
helped: 12
HURT: 0
Instructions are helped.

total nops in shared programs: 84045 -> 84041 (<.01%)
nops in affected programs: 1571 -> 1567 (-0.25%)
helped: 1
HURT: 7
Inconclusive result (value mean confidence interval includes 0).

total clauses in shared programs: 20498 -> 20489 (-0.04%)
clauses in affected programs: 188 -> 179 (-4.79%)
helped: 6
HURT: 0
Clauses are helped.

total quadwords in shared programs: 90395 -> 90291 (-0.12%)
quadwords in affected programs: 2287 -> 2183 (-4.55%)
helped: 12
HURT: 0
Quadwords are helped.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9420>

pan/bi: Allow negating constants

Useful for representing -0 in transcendental sequences matching the
blob.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9420>

pan/bi: Use replace_index in more places

Needed to respect abs/neg.

Signed-off-by: Alyssa Rosenzweig <alyssa@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9420>

radeonsi/sqtt: export shader code to RGP

With these changes the shader code is visible in RGP.

Vk pipeline feature is emulated using si_update_shaders: when shaders are
updated we compute a sha1 of their code and use it as a pipeline hash.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

radeonsi/sqtt: don't always use WGP 0

Because it may be disabled. Instead use the cu mask to
pick the first active WGP.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

radeonsi/sqtt: remove duplicate token

V_008D18_REG_INCLUDE_CONTEXT was set twice.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

radeonsi/sqtt: keep a copy of the uploaded shader code

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

ac/rgp: move radv/sqtt functions to ac

pso_correlation and code_object_loader don't depend on drivers
specific logic so move them to the shared code.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

ac/rtld: make ac_rtld_upload returns the code size

This will be useful to keep a copy of the uploaded code.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

ac/rgp: make the max gap between shader code a warning

For radeonsi the shaders don't live in the same BOs, so they're
unlikely to be less that 0x1000 bytes apart.

So this commit bumps the threshold to 0x10000 and warns once
when hitting it.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

radeonsi: properly set SPI_SHADER_PGM_HI_ES

When not using S_00B324_MEM_BASE the value isn't properly truncated.

Cc: mesa-stable
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9277>

broadcom/compiler: fix flags check for ldvary merge

We were checking that the previous instruction doesn't write flags,
but we also need to check it doesn't read them.

Fixes: 1784dd22a32 ('broadcom/compiler: pipeline smooth ldvary sequences')
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9431>

broadcom/compiler: ldvary doesn't implicitly write to r3 since V3D 4.1

total instructions in shared programs: 13805979 -> 13786037 (-0.14%)
instructions in affected programs: 2263244 -> 2243302 (-0.88%)
helped: 10646
HURT: 1508
Instructions are helped.

total threads in shared programs: 412220 -> 412242 (<.01%)
threads in affected programs: 58 -> 80 (37.93%)
helped: 17
HURT: 6
Threads are helped.

total uniforms in shared programs: 3793200 -> 3790401 (-0.07%)
uniforms in affected programs: 131281 -> 128482 (-2.13%)
helped: 1547
HURT: 281
Uniforms are helped.

total max-temps in shared programs: 2326309 -> 2324834 (-0.06%)
max-temps in affected programs: 31836 -> 30361 (-4.63%)
helped: 1139
HURT: 153
Max-temps are helped.

total spills in shared programs: 5932 -> 5940 (0.13%)
spills in affected programs: 80 -> 88 (10.00%)
helped: 2
HURT: 3

total fills in shared programs: 13370 -> 13372 (0.01%)
fills in affected programs: 480 -> 482 (0.42%)
helped: 2
HURT: 3

total sfu-stalls in shared programs: 30829 -> 30685 (-0.47%)
sfu-stalls in affected programs: 2190 -> 2046 (-6.58%)
helped: 570
HURT: 533
Sfu-stalls are helped.

total inst-and-stalls in shared programs: 13836808 -> 13816722 (-0.15%)
inst-and-stalls in affected programs: 2276152 -> 2256066 (-0.88%)
helped: 10643
HURT: 1525
Inst-and-stalls are helped.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9430>

radv: don't set sx_blend_opt_epsilon for V_028C70_COLOR_10_11_11

Matches radeonsi and PAL. From PAL:
// 1 is recommended, but doesn't provide sufficient precision

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4394
Fixes: ed946381564 ("radv: Enable RB+ where possible.")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9427>

broadcom/compiler: always restart ldvary pipelining when scheduling ldvary

When we were only able to pipeline smooth varyings, if we had to disable
ldvary pipelining in the middle of a sequence it would stay disabled for
the rest of the program, to prevent us from prioritizing scheduling of
ldvary instructions that we would not be able to pipeline effectively.
Now that we can pipeline all ldvary sequences we can change this.

This change re-enables ldvary pipelining upon finding the next
ldvary in the program in the hopes that we can continue pipelining
succesfully. To do this, we track the number of ldvary instructions we
emitted so far and compare that to the number of inputs in the fragment
shader we are scheduling. This also allows us to simplify our ldvary
tracking at nir to vir time, since that is all now handled in the QPU
scheduler.

total instructions in shared programs: 13817048 -> 13810783 (-0.05%)
instructions in affected programs: 810114 -> 803849 (-0.77%)
helped: 4843
HURT: 591
Instructions are helped.

total max-temps in shared programs: 2326612 -> 2326300 (-0.01%)
max-temps in affected programs: 4689 -> 4377 (-6.65%)
helped: 285
HURT: 7
Max-temps are helped.

total sfu-stalls in shared programs: 30942 -> 30865 (-0.25%)
sfu-stalls in affected programs: 207 -> 130 (-37.20%)
helped: 120
HURT: 42
Sfu-stalls are helped.

total inst-and-stalls in shared programs: 13847990 -> 13841648 (-0.05%)
inst-and-stalls in affected programs: 825378 -> 819036 (-0.77%)
helped: 4899
HURT: 590
Inst-and-stalls are helped.

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9404>

radv: re-enable TC-compat HTILE for MSAA D32S8 images on GFX9+

Should help MSAA games. Note that it's broken on GFX8 because
the tiling doesn't match.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3868
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9284>

virgl: use atomic operations when increase sub_ctx_id

Use atomic operations to avoid competition. In addition,
since sub_ctx_id 0 has been used by default, sub_ctx_id
should start from 1.

Signed-off-by: Xin He <hexin.op@bytedance.com>
Reviewed-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9406>

radv: skip useless FCE when fast-clearing MSAA images with DCC enabled

The clear code is 0xCC which means CMASK isn't fast-cleared.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9392>

radv: remove useless check about mips+layers for TC-compat HTILE images

radv_use_htile_for_image() prevents it.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9405>

radv: cleanup enabling TC-compat HTILE for depth surfaces

It makes more sense to try to enable TC-compat if the image has HTILE.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9405>