pan/bi: Use canonical texture op names in IR

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Fix simple txl test

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Fixes: 731dfc6066d ("pan/bi: Allow vertex txl with lod=0 as compact")
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Expose GL 2.1 on Bifrost

Needed for glamor. These features should work fine.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Handle vector moves

And fix the bad assertion that let this slip.

Like combines, nir_op_vec can be vector, and we need to lower this
ourselves. Thankfully, the lowering is simple.

Fixes
dEQP-GLES2.functional.shaders.loops.for_uniform_iterations.nested_tricky_dataflow_1_*

Fixes: b2c6cf2b6db ("pan/bi: Eliminate writemasks in the IR")
Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Pass flow_control through directly

More than just a single bool!

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Use canonical flow control enum

Merges multiple bits and adds some new combinations. The semantics are
the compiler are evidently wrong, we'll fix that next.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Use canonical term dependency

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Use canonical term "message type"

These identify the type of message produced by a message-passing
instruction, rather than information about the clause per se.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Print message types as strings

Even if we're not in verbose mode to match the canonical syntax.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Add missing message types

Names are not canonical but that's ok.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Expand clause type to 5-bit

The upper bit is reserved.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Use canonical name for staging registers

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Use canonical next_clause_prefetch

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Canonicalize terminate_discarded_threads

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Use canonical floating-point modes

First few pre-clause modifiers.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Cull unnecessary edges on the CF graph

If a block ends in an unconditional jump, we don't need to record a
fallthrough successor as well, since it's unreachable.

Likely harmless but makes the IR harder to parse with messy CF graphs.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Drop if 0'd combine lowering

This is supposed to be optimized but actually just broken. When we look
at optimizing this later we'll probably want a more straightforward copy
prop pass instead.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Fix memory corruption in scheduler

If empty the last will be bogus, I think. Missing Rust hard right around
now.

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/decode: Ensure mappings are zeroed

Fixes valgrind error when running with =sync

==30966== Conditional jump or move depends on uninitialised value(s)
==30966==    at 0x5B424E8: pandecode_find_mapped_gpu_mem_containing (decode_common.c:56)
==30966==    by 0x5B4CFB7: pandecode_jc (decode.c:2075)
==30966==    by 0x5ABBFA7: panfrost_batch_submit_ioctl (pan_job.c:1020)
==30966==    by 0x5ABD397: panfrost_batch_submit_jobs (pan_job.c:1042)
==30966==    by 0x5ABD397: panfrost_batch_submit (pan_job.c:1109)

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

pan/bi: Add copy for register COMBINEs

Fixes:

dEQP-GLES2.functional.shaders.loops.for_constant_iterations.infinite_with_conditional_break_fragment

Signed-off-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7081>

ci: piglit: conditionally build OpenCL tests

For baremetal piglit this is not yet needed.

Signed-off-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7075>

vc4: Enable lower_umax and lower_umin

VC4 doesn't have support for UMAX and UMIN integer operations. So
we should avoid algebraic optimizations that generate umax/umin ops.

Fixes: 8e1b75b330954a ("nir/algebraic: optimize iand/ior of (n)eq zero")
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7083>

nir/algebraic: optimize iand/ior of (n)eq zero when umax/umin not available

Before 8e1b75b330954a ("nir/algebraic: optimize iand/ior of (n)eq zero") this
optimization didn't need the use of umax/umin. VC4 HW supports only signed
integer max/min operations.

lower_umin and lower_umax are added to allow enabling previous optimizations
behaviour for this cases.

Fixes: 8e1b75b330954a ("nir/algebraic: optimize iand/ior of (n)eq zero")
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Reviewed-by: Daniel Schürmann <daniel@schuermann.dev>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7083>

panfrost: Add a debug flag to disable AFBC

Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7076>

radeonsi: disable SDMA on gfx6-7 and gfx10.3 to decrease CPU overhead

same as gfx8-10

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7055>

radeonsi: update the DMA perf test

- don't test 2 waves/SA
- create the compute shader only once per subtest
- use only 1 TIME_ELAPSED query per subtest
- don't invalidate sL0 (it's not used)
- don't invalidate L2 for L2_LRU to test L2 throughput
- don't flush the CS after every run
- remove unused min/max computation

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7055>

winsys/amdgpu: apply the VM alignment optimization to the physical alignment too

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7055>

winsys/amdgpu: rework the VM alignment optimizations

- don't increase the alignment past 2 MB
- apply the second tweak to older chips too

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7055>

ac/surface: fix valgrind warnings in DCC retile tile lookups

==12920== Conditional jump or move depends on uninitialised value(s)
==12920==    at 0x8F39391: util_fast_urem32 (fast_urem_by_const.h:71)
==12920==    by 0x8F39391: hash_table_search (hash_table.c:285)
==12920==    by 0x8B06D5D: ac_compute_dcc_retile_tile_indices (ac_surface.c:136)

Fixes: a37aeb128d5f7c "amd/common: Cache intra-tile addresses for retile map."

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7055>

CI: build our own spirv tools

This causes a lot of hiccups on the CL tests, but I've got most of
them fixed in another MR in pieces.

This should at least give a much more realistic baseline.

v2: use script in both places

Reviewed-by: Michel Dänzer <mdaenzer@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7073>

ci: fix deqp clone + fetch

This was taking > 10 minutes and I got bored, don't do a depth 1 fetch
in the first place just to do a proper fetch later.

Acked-by: Eric Anholt <eric@anholt.net>
Acked-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7073>

disk_cache: build option for disabled-by-default

On some systems it is problematic to have the shader cache enabled
by default. This adds a build option to support the disk cache but
keep it disabled unless the environment variable
MESA_GLSL_CACHE_DISABLE=false.

For example, on Chrome OS, Chrome already has it's own shader
disk cache implementation so it disables the mesa feature. Tests
do not want the shader disk cache enabled because it can cause
inconsistent performance results and the default 1GB for the
disk cache could lead to problems that require more effort to
work around. The Mesa shader disk cache is useful for VMs though,
where it is easy to configure the feature with environment
variables. With the current version of Mesa, Chrome OS would need
to have a system-wide environment variable to disable the disk
cache everywhere except where needed. More elegant to just build
Mesa with the cache feature disabled by default.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6967>

radv: use radv_optimize_nir() less in radv_link_shaders()

fossil-db (Navi):
Totals from 11 (0.01% of 137413) affected shaders:
CodeSize: 99372 -> 99480 (+0.11%)
Instrs: 19119 -> 19110 (-0.05%)
Cycles: 222144 -> 222000 (-0.06%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6891>

radv: move optimizations in shader_compile_to_nir() to after io_to_scalar

This results in at least one less radv_optimize_nir() iteration.

No fossil-db changes.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6891>

nir: return progress from nir_lower_io_to_scalar_early

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6891>

panfrost: Move the blend shader cache at the context level

Blend shaders can be shared among blend states, so let's move the blend
shader one level up so we don't have to re-create/re-compile shaders
when another blend state already asked for it.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

panfrost: Get rid of the constant patching done on blend shader binaries

When constants are used in the blend equation we simply recompile the
shader.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

panfrost: Let compile_blend_shader() allocate the blend shader object

This way we avoid an extra copy in panfrost_get_blend_shader().
Note that the allocation is attached to the blend state object
which simplifies the delete_blend_state() path.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

panfrost: Don't leak NIR blend shaders

Right now we create shaders that are not attached to any memory
context, leading to memory leaks. Ideally, we should free the NIR
shader as soon as we've turned it into a binary, but there's no
function explicitly destroy a shader. Let's attach those to the blend
state so they get destroyed when this state is freed.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

panfrost: Allocate blit_blend with ralloc()

This way we can use blend states as memory context which will help
simplify the blend shader creation/destruction logic.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

panfrost: Pass compile arguments through a struct

So we can extend it more easily without having to patch all callers.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

panfrost: Move the blend constant mask extraction out of make_fixed_blend_mode()

This way we can get a constant mask for the blend shader case too.

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

panfrost: Constify the rt_fmts arg passed to pan_lower_framebuffer()

Signed-off-by: Boris Brezillon <boris.brezillon@collabora.com>
Reviewed-by: Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7066>

radv: Set fce metadata correctly on DCC initialization.

The fce metadata can always be set to false as we don't care about
the compressed clear color.

Avoiding useless fast clear eliminates improves basemark performance by
1%-1.5%.

Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7005>

aco/ngg: Calculate workgroup size of NGG shaders.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Allocate NGG GS space early for const vertex/primitive counts.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Use more efficient LDS layout to help reduce bank conflicts.

The LLVM backend has a trick which helps reduce LDS bank conflicts
by swizzling the LDS address where each vertex is emitted.
This commit implements the same thing for ACO.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

radv/aco: Enable NGG GS by default.

ACO NGG GS now supports everything we need except streamout
(aka. transform feedback), but we don't use NGG anyway when
streamout is needed.

Also add a note to the new features txt.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Add shader query support to NGG GS.

In each GS thread, we calculate the number of "real" primitives that
were emitted (points, lines, triangles, not strips). Then we
accumulate the number of "real" primitives emitted by the
entire threadgroup in GDS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Place workgroup barrier outside control flow for NGG GS.

Merged shaders have a workgroup barrier which makes sure that
the first half is completed in every wave before the 2nd half
is started.

This barrier is located in divergent control flow, so that waves
that don't have any invocations in the 2nd half can finish as early
as possible. This is problematic for NGG GS because it has more
workgroup barriers after the 2nd half.

So, for NGG GS we need to put the barrier outside
control flow because otherwise the waves that have 0 GS threads
won't be able to wait for the waves which have non-zero GS threads.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Implement NGG GS output.

We store emitted GS vertices in LDS.
Then, at the end of the shader, the emitted vertices are compacted
and each thread loads a single vertex from LDS in order to export
a primitive as needed, and the vertex attributes.

The reason this is done is because there is an impedance mismatch
between how API GS and the NGG HW works. API GS can emit an arbitrary
number of vertices and primites in each thread, but NGG HW can only
export one vertex per thread.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Implement workgroup reduce / exclusive scan for NGG GS.

This function calculates two things at once:

1. The total number of vertices emitted by the threadgroup.
2. Exclusive scan of emitted vertex count accross the threadgroup.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Create LDS layout for NGG GS.

For NGG GS, we need to store the following in LDS:

1. The ESGS ring, similarly to legacy ESGS.
2. Emitted vertices from the GS threads.
3. Temporary space used by the workgroup scan.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Setup NGG GS.

Make it possible for ACO to recognize when to use HW NGG GS.
Also add a few notes about the various GS stages in the comments.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Allow NGG GS to create VS exports.

NGG GS need to use the same instructions to export vertex
attributes at the end.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Allow NGG GS to load per-vertex GS inputs.

They work the same way as in legacy GS, so we can reuse that.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Allow NGG GS to store ES outputs.

We can reuse the existing ES output code.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Clean up and reorganize NGG VS/TES code.

Make the NGG VS/TES code easier to follow, give better names to
some functions and make ngg_nogs_early_prim_export a variable.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Make primitive export packing less prone to error.

Use lshl_or instead of lshl_add, which makes it more robust in
handling -1 and -2 indices which will now just become null
exports, which is what we want.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Refactor ngg_emit_prim_export in preparation for NGG GS.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco/ngg: Refactor gs_alloc_req in preparation for NGG GS.

Previously, this function inferred the vertex and primitive counts
from the gs_tg_info shader argument, but in case of NGG GS, it will
need to be calculated in runtime.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco: Add wave-specific opcode for s_lshl and s_flbit.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco: Optimize thread_id_in_threadgroup when there is just one wave.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco: Use thread_id_in_threadgroup helper for ES outputs.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco: Extract thread_id_in_threadgroup to a separate function.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco: Extract lanecount_to_mask to a separate function.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco: Clarify missing export error message in assembler.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

aco: Add loop creation helpers.

Will be useful for NGG GS and probably testing. The helpers take care of
divergence but not creating correct phis.

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

radv/aco: Use new GS lowering options for ACO with NGG GS.

This makes it easier for ACO to implement NGG GS:

1. No need to keep track of vertex and primitive counts.
2. No need to discard incomplete primitives.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

nir: Add ability to count primitives per stream.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

nir: Count vertices per stream.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

nir: Add ability to overwrite incomplete GS primitives.

After each end_primitive and at the end of the shader before emitting
set_vertex_and_primitive_count, we check if the primitive that is being
emitted has enough vertices or not, and we adjust the vertex and
primitive counters accordingly.

As a result, if the backend uses this option, the backend compiler
will not have to worry about discarding the unneeded vertices
and primitives.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

nir: Add ability to count emitted GS vertices per primitive.

Add an option to nir_lower_gs_intrinsics so that it can also track
the number of emitted vertices per primitive, not just the total
vertex count.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

nir: Add ability to count emitted GS primitives.

Add an option to nir_lower_gs_intrinsics which tells it to track
the number of emitted primitives, not just vertices. Additionally,
also make it per-stream.

Also rename the set_vertex_count intrinsic to
set_vertex_and_primitive_count.

Signed-off-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6964>

radv,anv: use CLOCK_MONOTONIC_FAST when CLOCK_MONOTONIC_RAW is undefined

CLOCK_MONOTONIC_FAST is a similar clock from FreeBSD.

Acked-by: Eric Engestrom <eric.engestrom@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6995>

util/os_misc: os_get_available_system_memory() for FreeBSD

Based on the OpenBSD variant.
The only difference between those two system is the sysctl mib.

Signed-off-by: Emmanuel Vadot <manu@FreeBSD.org>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6995>

radv: use the same NIR compiler options for both compiler backends

No changes, they are already similar.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6938>

radv/llvm: do not lower sub

To match ACO.

Totals from 268 (0.20% of 136420) affected shaders:
CodeSize: 1214060 -> 1214096 (+0.00%); split: -0.05%, +0.06%

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6938>

lima/parser: Fix varyings decoding in RSW

Reviewed-by: Vasily Khoruzhick <anarsoul@gmail.com>
Signed-off-by: Andreas Baierl <ichgeh@imkreisrum.de>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7044>

aco: dump the program if the disassembler failed

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6979>

clover: don't call validate spirv at all if not needed

Suggested on irc seems like a good plan.

Reviewed-by: Pierre Moreau <dev@pmoreau.org>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7051>

util/disk_cache: remove unused function param

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7059>

swr/rasterizer: Remove BuilderGfxMem member mpTrackMemAccessFuncTy.

mpTrackMemAccessFuncTy is not used anywhere.

Fix defect reported by Coverity Scan.

Uninitialized pointer field (UNINIT_CTOR)
uninit_member: Non-static class member mpTrackMemAccessFuncTy is not
initialized in this constructor nor in any functions that it calls.

Suggested-by: Jan Zielinski <jan.zielinski@intel.com>
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Jan Zielinski <jan.zielinski@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6930>

meson: Use more portable compiler option -std.

GCC and Clang support --std and -std options but Intel C++
Compiler only supports -std.

icpc: command line warning #10159: invalid argument for option '--std'

Fixes: 8a05d6ffc65d ("driconf: Make the driver's declarations be structs instead of XML.")
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7020>

turnip: Add support for GetSwapchainGrallocUsage2ANDROID().

This is lifted straight from anv, which seems like a reasonable way to go.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7015>

turnip: Detect Qualcomm gralloc and its UBWC flag on gralloc surfaces.

And document where to find information on qcom gralloc's private handle
layout.  I chose not to #include the gralloc_priv because it seems that
there's not much we need yet, and I'm hoping we can avoid the build-time
dependency on the specific platform.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7015>

turnip/kgsl: Add support for importing dma-bufs.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7015>

turnip/kgsl: Fix last minute breakage of the build.

Need to land KGSL in CI!

Fixes: 8163c818e32e ("turnip: implement timestamp fences/semaphores for kgsl backend")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7015>

Revert "anv: Add driconf option to disable compression for 16bpp format"

This reverts commit bcfec61d1ee2675b8aaaf3adec8109ce6fa6c003.

The previous patch fixed the underlying issue that the above commit was
actually working around. It turns out that the previously observed
performance regression was due to invalid aux-map entries for
multi-layer HiZ+CCS buffers.

Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7046>

anv: Enable multi-layer aux-map init for HIZ+CCS

Fixes rendering corruption in the shadowmappingcascade Sascha Willems
Vulkan demo. To see the corruption, I adjusted the demo options as
follows:

 1. Enable "Display depth map"
 2. Set "Split lambda" to 0.100
 3. Make "Cascade" non-zero.

Fixes: 80ffbe915fe ("anv: Add support for HiZ+CCS")
Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7046>

freedreno/ir3: Don't leave holes the UBO upload plan.

Shaders may not use a particular region of a UBO in a given shader (think
UBOs shared between stages, or between shaders), and by just always
extending the existing range for a given UBO, we'd waste bandwidth
uploading it, and also waste our precious const space in storing the
unused data.

Instead, only upload exactly the ranges we can use, and merge ranges when
they're neighbors.  We may end up with more upload packets, but the
bandwidth savings is surely going to be worth it (and if find we want a
distance threshold for merging with nearby uploads, that would be easy to
add).

total instructions in shared programs: 9266114 -> 9255092 (-0.12%)
total full in shared programs: 343162 -> 341709 (-0.42%)
total constlen in shared programs: 1454368 -> 1275236 (-12.32%)
total cat6 in shared programs: 93073 -> 82589 (-11.26%)
total (ss) in shared programs: 212402 -> 206404 (-2.82%)
total (sy) in shared programs: 122905 -> 114007 (-7.24%)

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7036>

freedreno/ir3: Clean up the UBO upload plan setup.

No more start > end for signaling that the slot isn't used, no more funny
setup of num_enabled.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7036>

ci/softpipe: Add another flaky GS test to the skips list.

We've seen the new GS flake once or twice a week for a long time.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7070>

util/hash_table: add function for reserving size in a hash table

rehashing a populated hash table is very expensive, so for the case where
the maximum/likely table size is already known, this function allows for
pre-sizing the table to avoid ever needing a rehash

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7037>

nir/opt_deref: Fix the vector bitcast optimization

It assumes the parent is a vector or scalar so we need to fail if it
isn't.

Fixes: 9190f82d573c "nir/opt_deref: Add an optimization for bitcasts"
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7064>

clover: Stop leaking NIR shaders

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7068>

util/xmlconfig: eliminate memory leak

It fixes coverity issue: CID 1467703: (RESOURCE_LEAK):
`Variable "cp" going out of scope leaks the storage it points to.`

Fixes: 23c3eb1fe16 ("driconf: Delete disjoint range support")
Reviewed-by: Eric Anholt <eric@anholt.net>
Signed-off-by: Andrii Simiklit <andrii.simiklit@globallogic.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7021>

radv/llvm: remove dead code for 64-bit GS inputs

64-bit IO are lowered with NIR.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/7008>

radv/llvm: do not lower nir_op_fsat

To match ACO.

fossilds-db (Navi10):
Totals from 20869 (15.30% of 136420) affected shaders:
SGPRs: 1851128 -> 1851920 (+0.04%); split: -0.41%, +0.46%
VGPRs: 1607360 -> 1608212 (+0.05%); split: -0.20%, +0.25%
SpillSGPRs: 267331 -> 261350 (-2.24%); split: -3.67%, +1.43%
CodeSize: 155460104 -> 155303508 (-0.10%); split: -0.21%, +0.11%
MaxWaves: 179156 -> 178928 (-0.13%); split: +0.48%, -0.60%

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6932>

ac/nir: implement nir_op_fsat

With fmed3 if available, otherwise fallback to fmin/fmax.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6932>