radeonsi: adjust ps args for aco

aco need explicite args including PS arg compaction and
scratch_offset.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: resolve aco scratch addr symbols

Used for scratch buffer operation and reg spill when aco.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add symbols to si_shader_binary

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add initial aco compile code

Only for monolithic PS.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: lower non uniform texture access when aco

aco need all resource have been lowered to descriptor.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add has_non_uniform_tex_access shader info

Can be used to skip nir_lower_non_uniform_access pass.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: lower vector const to scalar at last for aco

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: lower some 64bit ops aco does not support

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ac/llvm,radeonsi: lower nir_fpow for aco and llvm

aco does not implement fpow, need nir to lower it
first. llvm will do by itself in the same way, so
we always lower fpow in nir now.

Remove the llvm fpow implementation that has special
handling for the muliplication. It's not used any
more and does not match GLSL spec as fpow(0,0)=NaN
but here we get 0.

There's some pixel changes for gl-radeonsi-stoney:
  ror-default 2 (no tolerance), 0 (1% tol.)

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ac/llvm,radeonsi: lower some pack/unpack ops not supported by aco

aco only support the split vertion of these instructions.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ac/llvm,radeonsi: lower ineg in nir

aco does not implement it.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ac/llvm,radeonsi: lower txf offset in nir

aco will complain if txf has offset. Not if other
texture ops.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ac/llvm,radeonsi: lower fsin/fcos in nir

ACO only support nir_fsin/cos_amd.

There's some pixel changes for gl-radeonsi-stoney trace.
Different pixels:
  furmark  61 (no tolerance), 0 (1% tol.)
  gimark   93867 (no tolerance), 888 (1% tol.)
  tessmark 39 (no tolerance), 0 (1% tol.)

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ac/llvm,radeonsi: lower idiv in nir

aco does not implement these idiv ops.

nir_lower_idiv is for idiv ops <= 32bit and ported from
llvm amdgpu, so llvm do the same.

nir_lower_divmod64 is for 64bit idiv ops.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

tgsi_to_nir: call nir_lower_int64 when required

Use case: radeonsi will generate internal tgsi shader
with 64bit udiv instruction, and we want all 64bit udiv
to be lowered in nir by lower_int64_options.

For GLSL shaders, this is done in glsl to nir, so we do
the same for tgsi here.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: remove ps vgpr index save when args init

They will be set by ac_get_fs_input_vgpr_cnt() later anyway.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: support print raw shader binary

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: support raw shader binary upload

Only monolithic shader.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ac/binary: pack prefech align code to a function

To be used by radeonsi raw shader binary.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add a raw shader binary type

It's the output of ACO compiler. To share the si_shader_binary
struct with ELF type:
  * add a type field to indicate RAW or ELF
  * rename elf_buffer/size to code_buffer/size

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: init spi ps input shader config when aco

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: pack spi ps input fixup to a function

To be shared with ACO spi ps input construction.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add shader info uses_sampleid

Used by ACO to set spi_ps_intput.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add shader info for frag coord and sample pos read

To construct spi_ps_input when ACO compilation.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add use_aco field for struct si_shader

We are going to use aco for monolithic ps first.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

radeonsi: add aco debug option

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

meson: build radeonsi with aco

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22573>

ci/skqp: handle all warnings printed with clang >= 14

Useful for the https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/21977

Reviewed-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22808>

radv: Stop running constant folding during ray query lowering

Now that committed is an intrinsic index, there is no need for constant
folding.

Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22963>

nir: Make rq_load committed src an index

committed has to be a constant so there is no need to have a src and
depend on constant folding to remove the i2b.

Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Acked-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22963>

gtest: Update to 1.13.0

Fix msan issue found with recent GCC on Debian 12.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8915

Acked-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22814>

util/tests: adjust for new gtest

GTest deprecated the GTEST_ARRAY_SIZE_ macro.

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22814>

nv50: Fix return type of nv50_blit_is_array

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22997>

nvc0: Free blitter->vp

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22997>

nv50,nvc0: Free nir from blitter fp shader

Fixes: d11145e837 ("nv50,nvc0: Use nir in nv50_blitter_make_fp")
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22997>

dxil: Use unified atomics

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22993>

microsoft/compiler: Back-propagate type requirement information

For ALU ops where input types are known, we can store that info on
the input sources. This can be used to produce the correct overloads
of load instructions that don't immediately need to be followed by
bitcasts, or similarly to produce a constant value which can be directly
consumed by the relevant instruction without needing a bitcast.

Similarly for values that will be stored in an output, we know type
information. And using that info, we can use more-correct information
for phis instead of forcing all phi sources to be bitcast to int just
to be bitcast back to float on the other side for an alu or an output
store.

One missing piece is SSBO stores, where we can support int or float.
If the input is coming from a phi, we don't influence the phi's type,
so it'll be int, even though the incoming sources might've been float.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22972>

microsoft/compiler: Duplicate some SSA values to simplify SSA typing

For each phi src, ensure that it's only used as a phi src.
This lets us give each phi their own unique types without worrying
about them stomping on each other. Also scalarize phis.

For each constant, ensure that it's only used once. The DXIL backend
will already dedupe these consts within the module, but this lets a
single load_const have multiple types depending on how it's used.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22972>

microsoft/compiler: Remove alu type info from store_dest()

We pass in a *typed* value, we don't need to pass in additional
type info. That's just more opportunities to get it wrong.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22972>

ac/llvm: rewrite and unify how GLC, DLC, SLC are set

Use ACCESS_* flags in call sites instead of GLC/DLC/SLC.

ACCESS_* flags are extended to describe other aspects of memory instructions
like load/store/atomic/smem.

Then add a function that converts the access flags to GLC, DLC, SLC.

The new functions are also usable by ACO.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22770>

ac/llvm: don't treat ACCESS_NON_READABLE as ACCESS_COHERENT

... and expect it to behave like ACCESS_NON_TEMPORAL.
Handling ACCESS_NON_TEMPORAL is sufficient.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22770>

aco: don't treat ACCESS_NON_READABLE as ACCESS_COHERENT

... and expect it to behave like ACCESS_NON_TEMPORAL.
Handling ACCESS_NON_TEMPORAL is sufficient.

This was copied from ac_nir_to_llvm.c.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22770>

intel: Secondary CB print primary CB's renderpass

Reviewed-by: Mark Janes <markjanes@swizzler.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22723>

intel: batch consecutive dispatches into implicit renderpasses

Reviewed-by: Mark Janes <markjanes@swizzler.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22723>

intel: refactor INTEL_MEASURE pointer dumping

Refactor framebuffer to renderpass to mirror previous INTEL_MEASURE
changes.

We dump hashes/pointers for shaders and framebuffer/renderpass.
Reduce from 64bit to 32bit pointers. We don't benefit from the
extra precision and reduced output size is convenient.

Reviewed-by: Mark Janes <markjanes@swizzler.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22723>

anv: re-enable RT data in INTEL_MEASURE

Per-RenderTarget analysis was removed from anv's INTEL_MEASURE
previously, probably after switching to dynamic rendering model.
Restore capability by tracking count of beginRenderPass calls.

Reviewed-by: Mark Janes <markjanes@swizzler.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22723>

anv: fix INTEL_MEASURE on MTL

Ensure counter buffer is coherent. Required for MTL which changes
coherence policy.

Reviewed-by: Mark Janes <markjanes@swizzler.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22723>

nir/validate: Handle unified atomics

nir_validate checks that the format of an atomic (if specified) is compatible
with the atomic operation. For example, we can't fadd R64_UINT texels. The logic
can't be extended as-is to unified atomics because it's split across different
switch cases for different atomic-op intrinsics. So we add our own validation
case, porting over the logic from the separate existing cases below.

(The redundant logic will be deleted once we delete legacy atomics.)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir/opt_uniform_atomics: Handle unified atomics

This is the one place where using nir_atomic_op instead of nir_op directly is a
little annoying, since we need to translate between the two enums, but it's not
a big deal.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir/lower_ssbo: Handle unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir/lower_io: Handle unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir/lower_task_shader: Handle unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir: Load/store atomic op indices when lowering image intrinsics

They might not be stored in the same const index after lowering

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir: Handle unified atomics in simple cases

Lots of passes can be made unified-atomics-aware simply by adding extra cases in
their switch statements. This commit fixes a bunch of passes.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

ir3: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Rob Clark <robclark@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nv50/ir: Use unified atomics

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: M Henning <drawoc@darkrefraction.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

zink: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

aco,radv: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

ac/llvm: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

ac/llvm: Don't handle atomic derefs

Should not be seen, already would be stubbed out.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Suggested-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

ntt: Use unified atomics

Nice deduplication of the NIR->TGSI enum translation.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

gallivm: Use unified atomics

This is a huge win because gallivm duplicated the translations in a zillion
places.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

pan/mdg: Use unified atomics

This doesn't actually save anything, since Italo already introduced magic macros
for this, but it ticks off one more driver on the list to convert. It's also
more legible, so that's nice :-)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

pan/mdg: Fix icky formatting

clang-format butchered this initializer pretty badly.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

pan/bi: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

agx: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir: Add pass to lower atomics to unified

In the future, we'd like to have all drivers only ingest unified atomics, and
all frontends only produce unified atomics, and garbage collect the existing
non-unified atomics. To get to that future, it's a lot nicer to convert drivers
one-by-one. Add a pass to translate old-style atomics to new-style atomics so
drivers can opt-in to the new form one-by-one. Once all drivers are converted,
we can convert producers one-by-one. Finally, we can just drop the calls to the
pass and garbage collect this pass and the old atomics. That's probably a while
out, though, so this will be out bridge to get there.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rob Clark <robclark@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir: Add unified atomics

Currently, we have an atomic intrinsic for each combination of memory type
(global, shared, image, etc) and atomic operation (add, sub, etc). So for m
types of memory supported by the driver and n atomic opcodes, the driver has to
handle O(mn) intrinsics. This makes a total mess in every single backend I've
looked at, without fail.

It would be a lot nicer to unify the intrinsics. There are two obvious ways:

1. Make the memory type a constant index, keep different intrinsics for
   different operations. The problem with this is that different memory types
   imply different intrinsic signatures (number of sources, etc).  As an
   example, it doesn't make sense to unify global_atomic_amd with
   global_atomic_2x32, as an example. The first takes 3 scalar sources, the
   second takes 1 vector and 1 scalar. Also, in any single backend, there are a
   lot more operations than there are memory types.

2. Make the opcode a constant index, keep different intrinsics for different
   operations. This works well, with one exception: compswap and fcompswap
   take an extra argument that other atomics don't, so there's an extra axis of
   variation for the intrinsic signatures.

So, the solution is to have 2 intrinsics for each memory type -- for atomics
taking 1 argument and atomics taking 2 respectively. Both of these intrinsics
take an nir_atomic_op enum to describe its operation. We don't use a nir_op for
this purpose, as there are some atomics (cmpxchg, inc_wrap, etc) that don't
cleanly map to any ALU op and it would be weird to force it.

The plan is to transition to these new opcodes gradually. This series adds a
lowering pass producing these opcodes from the existing opcodes, so that
backends can opt-in to the new forms one-by-one. Then we can convert backends
separately without any cross-tree flag day. Once everything is converted, we can
convert the producers and core NIR as a flag day, but we have far fewer
producers than backends so this should be fine. Finally we can drop the old
stuff.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rob Clark <robclark@freedesktop.org>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

aco/gfx11: fix VMEM/DS->VALU WaW/RaW hazard

Previously, we could safely read/write unused lanes of VMEM/DS
destination VGPRs without waiting for the load to finish. That doesn't
seem to be the case on GFX11.

fossil-db (gfx1100):
Totals from 6698 (4.94% of 135636) affected shaders:
Instrs: 11184274 -> 11199420 (+0.14%); split: -0.00%, +0.14%
CodeSize: 57578344 -> 57638928 (+0.11%); split: -0.00%, +0.11%
Latency: 198348808 -> 198382472 (+0.02%); split: -0.00%, +0.02%
InvThroughput: 24376324 -> 24378439 (+0.01%); split: -0.00%, +0.01%
VClause: 192420 -> 192559 (+0.07%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8722
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8239
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22965>

meson/vaon12: fix driver file name for mingw build

This fixes vaon12 driver file name to be consistent with libva
expectation - vaon12_drv_video.dll - without lib prefix.

Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22995>

lvp: avoid accessing member of NULL ptr for global entries

Cc: mesa-stable
Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22979>

pipe-loader: avoid undefined memcpy behavior

If either dest or src is an invalid or null pointer, the behavior is
undefined, even if count is zero.

Cc: mesa-stable
Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22979>

zink: disable dynamic state exts if the previous ones aren't present

this would be weird if a driver did it

cc: mesa-stable

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22996>

zink: disable have_EXT_vertex_input_dynamic_state without EDS2

this is disabled already in the draw paths but not the pipeline paths

cc: mesa-stable

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22996>

nir_lower_system_values: Add ASSERTED to assert-only variable

Fixes: 1e0e4657 ("spirv/nir: wire ray interection triangle position fetch")
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22994>

ci: replace write + cat with tee

Signed-off-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22945>

ci: stop marking environment variable list as executable

We're only going to read it, not execute it.

Signed-off-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22945>

ci: drop GENERATE_ENV_SCRIPT

Added in 16b417b8d617df418ad1 ("ci, valve: Add the dEQP runners for
Valve CI") but never used.

Signed-off-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22945>

nir: Use nir_block_last_phi_instr more

We have a helper, don't open code it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

ac/llvm: Use nir_foreach_phi

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

dxil: Use nir_foreach_phi_safe

This should be faster, since we're not iterating pointlessly over all the
non-phis after the phi.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

nir: Use nir_foreach_phi(_safe)

The pattern shows up all the time open-coded. Use the macro instead.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

nir: Add nir_foreach_phi(_safe) macro

Serious preprocessor voodoo here. There are two tricks here.

1. Iterating only phis. We know that phis come only at the beginning of a block,
   so all over the tree, we open-code iteration like:

      nir_foreach_instr(instr, block) {
         if (instr->type != phi)
            break;

         /* do stuff */
      }

   We can express this equivalently as

      nir_foreach_instr(instr, block)
         if (instr->type != phi)
            break;
         else {
            /* do stuff */
         }

   So, we can define a macro

      #define nir_foreach_phi(instr, block)
         if (instr->type != phi)
            break;
         else

   and then

      nir_foreach_phi(..)
         statement;

   and

      nir_foreach_phi(..) {
         ...
      }

   will expand to the right thing.

2. Automatically getting the phi as a phi. We want the instruction to go to some
   hidden variable, and then automatically insert nir_phi_instr *phi =
   nir_instr_as_phi(instr_internal); We can't do that directly, since we need to
   express the assignment implicitly in the control flow for the above trick to
   work. But we can do it indirectly with a loop initializer.

      for (nir_phi_instr *phi = nir_instr_as_phi(instr_internal); ...)

   That loop needs to break after exactly one iteration. We know that phi
   will always be non-null on its first iteration, since the original
   instruction is non-null, so we can use phi==NULL as a sentinel and express a
   one-iteration loop as for (phi = nonnull; phi != NULL; phi = NULL).

Putting these together gives the macros implemented used.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

aco: don't apply dpp if the alu instr uses the operand twice

CP77 has a ton of fma(dpp(a), dpp(a), b).

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco: use VOP3+DPP

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/ra: convert VOPC_DPP instructions without vcc to VOP3

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco: add assembler tests for VOP3(P) with DPP

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/builder: support VOP3(P) with dpp

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/assembler: support VOP3P with DPP

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/optimizer: copy pass flags for newly created valu instructions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/optimizer: don't use pass_flags for mad idx

fma can use DPP on GFX11+, so we want to keep the exec id in pass_flags

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/vn: compare all valu modifers

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Fixes: 9b4ea9ff901 ("aco/vn: hash opsel for VOP12C")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

zink: add back some anv qbo flakes

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22987>

docs: Include ALU opcode descriptions

If we have a description for an ALU opcode in NIR, include it with our generated
HTML documentation so people don't need to go to nir_opcodes.py anyway because
the documentation is missing the documentation ;-)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Jesse Natalie <jenatali@microsoft.com>
Tested-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22929>

nir: Make ALU descriptions machine-readable

We already document a lot of ALU opcodes, let's make this machine-readable so we
can put the descriptions in our generated HTML documentation.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22929>

nir: Allow adding descriptions to ALU opcodes

This will let us generate nicer documentation.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22929>

vc4/ci: re-enable VC4 testing

But keeps piglit's quick_gl disabled, as it contains the test causing
the GPU resets.

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22982>

amd/drm-shim: Update docs for more devices.

We don't have to update the docs for every new entry.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22975>

amd/drm-shim: Add raphael&mendocino, polaris12 and gfx1100.

Decided to follow the chip names pretty much.This set happens to be
what is in my workstation currently.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22975>

amd/drm-shim: Add vangogh entry.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22975>