zink: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

aco,radv: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

ac/llvm: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

ac/llvm: Don't handle atomic derefs

Should not be seen, already would be stubbed out.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Suggested-by: Marek Olšák <marek.olsak@amd.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

ntt: Use unified atomics

Nice deduplication of the NIR->TGSI enum translation.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

gallivm: Use unified atomics

This is a huge win because gallivm duplicated the translations in a zillion
places.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

pan/mdg: Use unified atomics

This doesn't actually save anything, since Italo already introduced magic macros
for this, but it ticks off one more driver on the list to convert. It's also
more legible, so that's nice :-)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

pan/mdg: Fix icky formatting

clang-format butchered this initializer pretty badly.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

pan/bi: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

agx: Use unified atomics

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir: Add pass to lower atomics to unified

In the future, we'd like to have all drivers only ingest unified atomics, and
all frontends only produce unified atomics, and garbage collect the existing
non-unified atomics. To get to that future, it's a lot nicer to convert drivers
one-by-one. Add a pass to translate old-style atomics to new-style atomics so
drivers can opt-in to the new form one-by-one. Once all drivers are converted,
we can convert producers one-by-one. Finally, we can just drop the calls to the
pass and garbage collect this pass and the old atomics. That's probably a while
out, though, so this will be out bridge to get there.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rob Clark <robclark@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

nir: Add unified atomics

Currently, we have an atomic intrinsic for each combination of memory type
(global, shared, image, etc) and atomic operation (add, sub, etc). So for m
types of memory supported by the driver and n atomic opcodes, the driver has to
handle O(mn) intrinsics. This makes a total mess in every single backend I've
looked at, without fail.

It would be a lot nicer to unify the intrinsics. There are two obvious ways:

1. Make the memory type a constant index, keep different intrinsics for
   different operations. The problem with this is that different memory types
   imply different intrinsic signatures (number of sources, etc).  As an
   example, it doesn't make sense to unify global_atomic_amd with
   global_atomic_2x32, as an example. The first takes 3 scalar sources, the
   second takes 1 vector and 1 scalar. Also, in any single backend, there are a
   lot more operations than there are memory types.

2. Make the opcode a constant index, keep different intrinsics for different
   operations. This works well, with one exception: compswap and fcompswap
   take an extra argument that other atomics don't, so there's an extra axis of
   variation for the intrinsic signatures.

So, the solution is to have 2 intrinsics for each memory type -- for atomics
taking 1 argument and atomics taking 2 respectively. Both of these intrinsics
take an nir_atomic_op enum to describe its operation. We don't use a nir_op for
this purpose, as there are some atomics (cmpxchg, inc_wrap, etc) that don't
cleanly map to any ALU op and it would be weird to force it.

The plan is to transition to these new opcodes gradually. This series adds a
lowering pass producing these opcodes from the existing opcodes, so that
backends can opt-in to the new forms one-by-one. Then we can convert backends
separately without any cross-tree flag day. Once everything is converted, we can
convert the producers and core NIR as a flag day, but we have far fewer
producers than backends so this should be fine. Finally we can drop the old
stuff.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Reviewed-by: Rob Clark <robclark@freedesktop.org>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22914>

aco/gfx11: fix VMEM/DS->VALU WaW/RaW hazard

Previously, we could safely read/write unused lanes of VMEM/DS
destination VGPRs without waiting for the load to finish. That doesn't
seem to be the case on GFX11.

fossil-db (gfx1100):
Totals from 6698 (4.94% of 135636) affected shaders:
Instrs: 11184274 -> 11199420 (+0.14%); split: -0.00%, +0.14%
CodeSize: 57578344 -> 57638928 (+0.11%); split: -0.00%, +0.11%
Latency: 198348808 -> 198382472 (+0.02%); split: -0.00%, +0.02%
InvThroughput: 24376324 -> 24378439 (+0.01%); split: -0.00%, +0.01%
VClause: 192420 -> 192559 (+0.07%)

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Georg Lehmann <dadschoorse@gmail.com>
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8722
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/8239
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22965>

meson/vaon12: fix driver file name for mingw build

This fixes vaon12 driver file name to be consistent with libva
expectation - vaon12_drv_video.dll - without lib prefix.

Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22995>

lvp: avoid accessing member of NULL ptr for global entries

Cc: mesa-stable
Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22979>

pipe-loader: avoid undefined memcpy behavior

If either dest or src is an invalid or null pointer, the behavior is
undefined, even if count is zero.

Cc: mesa-stable
Signed-off-by: Yiwei Zhang <zzyiwei@chromium.org>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22979>

zink: disable dynamic state exts if the previous ones aren't present

this would be weird if a driver did it

cc: mesa-stable

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22996>

zink: disable have_EXT_vertex_input_dynamic_state without EDS2

this is disabled already in the draw paths but not the pipeline paths

cc: mesa-stable

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22996>

nir_lower_system_values: Add ASSERTED to assert-only variable

Fixes: 1e0e4657 ("spirv/nir: wire ray interection triangle position fetch")
Reviewed-by: Matt Turner <mattst88@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22994>

ci: replace write + cat with tee

Signed-off-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22945>

ci: stop marking environment variable list as executable

We're only going to read it, not execute it.

Signed-off-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22945>

ci: drop GENERATE_ENV_SCRIPT

Added in 16b417b8d617df418ad1 ("ci, valve: Add the dEQP runners for
Valve CI") but never used.

Signed-off-by: Eric Engestrom <eric@igalia.com>
Reviewed-by: Emma Anholt <emma@anholt.net>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22945>

nir: Use nir_block_last_phi_instr more

We have a helper, don't open code it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

ac/llvm: Use nir_foreach_phi

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

dxil: Use nir_foreach_phi_safe

This should be faster, since we're not iterating pointlessly over all the
non-phis after the phi.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

nir: Use nir_foreach_phi(_safe)

The pattern shows up all the time open-coded. Use the macro instead.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

nir: Add nir_foreach_phi(_safe) macro

Serious preprocessor voodoo here. There are two tricks here.

1. Iterating only phis. We know that phis come only at the beginning of a block,
   so all over the tree, we open-code iteration like:

      nir_foreach_instr(instr, block) {
         if (instr->type != phi)
            break;

         /* do stuff */
      }

   We can express this equivalently as

      nir_foreach_instr(instr, block)
         if (instr->type != phi)
            break;
         else {
            /* do stuff */
         }

   So, we can define a macro

      #define nir_foreach_phi(instr, block)
         if (instr->type != phi)
            break;
         else

   and then

      nir_foreach_phi(..)
         statement;

   and

      nir_foreach_phi(..) {
         ...
      }

   will expand to the right thing.

2. Automatically getting the phi as a phi. We want the instruction to go to some
   hidden variable, and then automatically insert nir_phi_instr *phi =
   nir_instr_as_phi(instr_internal); We can't do that directly, since we need to
   express the assignment implicitly in the control flow for the above trick to
   work. But we can do it indirectly with a loop initializer.

      for (nir_phi_instr *phi = nir_instr_as_phi(instr_internal); ...)

   That loop needs to break after exactly one iteration. We know that phi
   will always be non-null on its first iteration, since the original
   instruction is non-null, so we can use phi==NULL as a sentinel and express a
   one-iteration loop as for (phi = nonnull; phi != NULL; phi = NULL).

Putting these together gives the macros implemented used.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22967>

aco: don't apply dpp if the alu instr uses the operand twice

CP77 has a ton of fma(dpp(a), dpp(a), b).

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco: use VOP3+DPP

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/ra: convert VOPC_DPP instructions without vcc to VOP3

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco: add assembler tests for VOP3(P) with DPP

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/builder: support VOP3(P) with dpp

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/assembler: support VOP3P with DPP

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/optimizer: copy pass flags for newly created valu instructions

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/optimizer: don't use pass_flags for mad idx

fma can use DPP on GFX11+, so we want to keep the exec id in pass_flags

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

aco/vn: compare all valu modifers

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Fixes: 9b4ea9ff901 ("aco/vn: hash opsel for VOP12C")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22698>

zink: add back some anv qbo flakes

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22987>

docs: Include ALU opcode descriptions

If we have a description for an ALU opcode in NIR, include it with our generated
HTML documentation so people don't need to go to nir_opcodes.py anyway because
the documentation is missing the documentation ;-)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Jesse Natalie <jenatali@microsoft.com>
Tested-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22929>

nir: Make ALU descriptions machine-readable

We already document a lot of ALU opcodes, let's make this machine-readable so we
can put the descriptions in our generated HTML documentation.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22929>

nir: Allow adding descriptions to ALU opcodes

This will let us generate nicer documentation.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22929>

vc4/ci: re-enable VC4 testing

But keeps piglit's quick_gl disabled, as it contains the test causing
the GPU resets.

Signed-off-by: Juan A. Suarez Romero <jasuarez@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22982>

amd/drm-shim: Update docs for more devices.

We don't have to update the docs for every new entry.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22975>

amd/drm-shim: Add raphael&mendocino, polaris12 and gfx1100.

Decided to follow the chip names pretty much.This set happens to be
what is in my workstation currently.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22975>

amd/drm-shim: Add vangogh entry.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22975>

anv: handle missing astc for gfx125 in CreateImageView

Cc: mesa-stable
Signed-off-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22818>

radv: configure PA_CL_VRS_CNTL entirely from the cmd buffer

We already have all the information needed to configure it.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22942>

radv: re-emit fragment shading rate state when PA_CL_VRS_CNTL changes

Found by inspection.

Cc: mesa-stable
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22942>

radv: dirty the dynamic vertex input state only when needed

This shouldn't be necessary when the VS doesn't have a prolog.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22944>

radv: reset the emitted VS prolog when a new vertex shader is bound

When a new vertex shader is bound, the VS prolog needs to be
re-emitted, and this allows us to avoid tracking if the pipeline is
dirty.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22944>

hasvk: avoid assert due to unsupported format

Fixes: 0a4c92b646f ("hasvk: Use the common vk_ycbcr_conversion object")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9011
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22978>

anv: Enable MCS init with ISL_AUX_OP_AMBIGUATE

Up until now, we have been initializing MCS with fast clears. This is
mostly safe, but there's a corner case that can be an issue.

The issue is with a workaround for MCS that requires the sampler not see
any fast-cleared blocks for certain surfaces (14013111325). Even though
we have been initializing MCS with fast clears, we expect most
applications to be safe because we expect that they would only sample
the samples they've rendered to previously (and the render would've
removed the fast-cleared blocks). In other words we don't expect that
apps would transition from VK_IMAGE_LAYOUT_UNDEFINED to
VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL and start sampling immediately.

If an application took the unexpected path of sampling undefined
samples, it's possible they'd hit the issue described in the workaround.

Fix this corner case by using an ambiguate to initialize MCS.

Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22545>

anv: Drop the MCS initialization performance warning

The comment above the warning explains that not all bit patterns are
necessarily valid. While we're at it, fix a typo in that comment.

Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22545>

iris: Enable MCS init with ISL_AUX_OP_AMBIGUATE

Add support for using BLORP's ambiguate pass to initialize MCS instead
of mapping and memsetting it on the CPU. Note that this won't be used if
the first operation on the MSAA layer is a fast clear.

Since we're no longer mapping, this removes a blocker towards getting
MCS_CCS enabled in small-BAR mode.

This functionality is difficult to test because of the way iris is set
up. It always tries to compress writes. So, a test would only read the
ambiguated MCS element if it tries to read from undefined samples.

To test this, I locally disabled fast clears and rendering with MCS (via
iris_resource_render_aux_usage). I continued to allow sampling with MCS
in iris_resource_texture_aux_usage. So, writes go directly to the main
surface and reads go through the ambiguated MCS surface.

When I then ran the test group, dEQP-GLES3.functional.multisample.*, all
48/64 supported tests passed on my Ice Lake. If I slightly changed
BLORP's ambiguate pass, I observed several tests failing.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22545>

intel: Implement ISL_AUX_OP_AMBIGUATE for MCS

Implement the ambiguate operation for MCS. This clears MCS layers with a
sample-dependent "uncompressed" value that tells the sampler to go look
at the main surface.

Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22545>

iris: Add a barrier to iris_mcs_partial_resolve

Partial resolves read from the MCS and write to the MSAA surface.
Add a texture barrier to prepare for the reads.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/4179
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22545>

intel/isl: Bump the MCS halign value for BDW+

Select a horizontal alignment value that matches the main MSAA surface.

We need a valid horizontal alignment to perform MCS ambiguates. The
halign value doesn't actually affect test behavior, but it is validated
by isl_surf_fill_state. We currently have an invalid halign for gfx125.
This patch fixes that.

Reviewed-by: Ivan Briano <ivan.briano@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22545>

ail: Add MSAA tests

This tests the following matrix:
- Format: RGBA8Unorm, RGBA16Unorm, RGBA32Float
- Samples: 2 or 4
- Layers: 1 or 2
- Width: Interesting values 1..4097
- Height: Interesting values 1..4097

Compression is based on the dimensions (that is, everything that can be
compressed is). This test compares both the total texture size and the
compression metadata offset.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

ail: Handle larger block sizes

We need to support up to 16 bytes/sample * 4 samples/pixel = 64 bytes/pixel for
multisampling to work with formats like RGBA32F.

Fixes dEQP-GLES3.functional.fbo.msaa.4_samples.rgba32f

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Use ail_can_compress() in agx_compression_allowed()

This moves the compression size threshold logic into ail, where
it belongs.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

ail: Implement multisampling for compression meta calculation

For multisampled textures, the decision about whether to compress or not
is based on the effective width and height in samples, not pixels.

Introduce ail_can_compress() to encode this logic in ail, so the driver
can use it to decide whether to compress or not before the full layout
is determined.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Make bo->writer_syncobj atomic

BOs can be written from several contexts, so writing to this member is
racy. We only care about this for the purposes of exporting BOs after a
submission (and if the app is racing writers/submissions at that point
all bets are off), so just keeping track of the last written value is
sufficient.

Switch to atomic operations to eliminate the race, and drop the assert
in the batch cleanup path that no longer holds when the BO might have
been written to from another context.

Fixes: asahi/mesa#20

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Lazily initialize batch state on first draw

We track buffers written by batches, but this gets messy when we end up
with an empty batch that is never submitted, since then it might have
taken over writer state from a prior already submitted batch (for its
framebuffers).

Instead of trying to track two tiers of resource writers, let's just
defer initializing batch state until we know we have a draw (or compute
launch, or clear). This means that if a batch is marked as a writer for
a buffer, we know it will not be an empty batch.

This should be a small performance win for empty batches (no need to
emit initial VDM state or run the writer code), but more impontantly it
eliminates the empty batch writer state revert corner case.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Partially identify some missing index list stuff

Still unclear what the extra 2 blocks do, but at least we know the
size/order now.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Add some more system registers

Core and opfifo stuff from the compute helper blob, vm_slot because it
was the only one changing when I poked around yesterday and it hit me
what it was ^^

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Fix check for sprite coord mode in agx_bind_rasterizer_state

We need to set ctx->rast = so after comparing them.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Add missing stdbool include to lib/hexdump.h

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Disable tilebuffer write masking optimization

This seems to flake some dEQPs due to some kind of race/UB (which
doesn't even always cause the dEQPs to fail due to leeway in the image
comparison, since the problem is usually just a few pixels, but it's
there).

I spent a bunch of time trying other flags/things, and almost everything
changed the bad pixel pattern randomly but nothing fixed it. Let's
revisit this one later, since it looks like a pretty deep rabbit hole.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Make framebuffer texture barriers a no-op

Framebuffer fetch is coherent, so there is no need for barriers here.
This avoids pointless flushing if an app calls glBlendBarrier().

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Implement create_fence_fd and fence_server_sync

Apparently we were still missing some fence stuff, and it started
crashing Firefox in apitrace? I'm not sure why we never noticed this
before, but it's trivial enough. Cargo culted from Panfrost.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

asahi: Implement memory_barrier

Cargo culted from panfrost.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22971>

intel: Disable shader cache when executing intel_clc during the build

With the shader cache enabled, intel_clc attempts to write to ~/.cache.
Many distributions' build systems limit file-system access, and will
kill the process thus causing the build to fail.

Fixes: 639665053fa ("anv/grl: Build OpenCL kernels")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22968>

radv: improve externalMemoryFeatures for android ahb

VK_EXTERNAL_MEMORY_FEATURE_DEDICATED_ONLY_BIT should always be set, as
required by the spec.

VK_EXTERNAL_MEMORY_FEATURE_EXPORTABLE_BIT should be set when
radv_ahb_format_for_vk_format knowns the format.  That is,
radv_create_ahb_memory should at least know how to call
AHardwareBuffer_allocate.

VK_EXTERNAL_MEMORY_FEATURE_IMPORTABLE_BIT is always set.  We can't know
if gralloc can allocate the format/flags/usage combo or not (gralloc
might use a private format for the combo).

Fixed
dEQP-VK.api.external.memory.android_hardware_buffer.image_formats.*.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

anv,hasvk: android ahb is not always exportable

anv_ahb_format_for_vk_format needs to know the format at least.  There
is no guarantee that AHardwareBuffer_allocate will succeed, but we are
reluctant to check with AHardwareBuffer_isSupported which may
test-allocate internally and is expensive.

v2: add anv_ahb_format_for_vk_format to anv_android_stubs.c

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

vulkan: add vk_image_format_to_ahb_format

There should be no functional change.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

anv,hasvk,radv: do not fall back to AHARDWAREBUFFER_FORMAT_BLOB

When allocating a VkDeviceMemory exportable as AHB, it seems incorrect
to fall back to AHARDWAREBUFFER_FORMAT_BLOB when the image has no known
AHB format.  We should fail the allocation instead.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

vulkan: add vk_ahb_format_to_image_format

There should be no functional change.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

vulkan: define inline stubs when android api level < 26

This allows us to reduce ANDROID #ifdef's.

v2: always include vk_android.h in radv_formats.c

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

vulkan: rename vk_image::ahardware_buffer_format

Rename it to ahb_format.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

vulkan: make sure vk_image_view::format is never UNDEFINED

Remove redundant override in anv and hasvk as well.

Fixed
android.graphics.cts.BasicVulkanGpuTest#testBasicBufferImportAndRenderingExternalFormat
for radv.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

vulkan: make sure vk_image::format is never UNDEFINED

vk_image::android_external_format is only used for sanity check and is
removed.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

hasvk: Use the common vk_ycbcr_conversion object

Based on commit 30a91d333d1 ("anv: Use the common vk_ycbcr_conversion
object").

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

hasvk/android: Use VkFormat for externalFormat

Same as commit 18feb32df0a ("anv/android: Use VkFormat for
externalFormat"), but for hasvk.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

hasvk: Refactor Android externalFormat handling in CreateYcbcrConversion

Same as commit 9fc046a87dc ("anv: Refactor Android externalFormat
handling in CreateYcbcrConversion"), but for hasvk.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22619>

dzn: Enable KHR_shader_integer_dot_product

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22952>

microsoft/compiler: Enable packed dot product intrinsics for SM6.4+

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22952>

microsoft/compiler: Take inputs from callers before providing nir options

The base nir options were assuming all bit sizes were supported at
shader model 6.2. Multiple callers were then changing properties
based on actual support.

Standardize behavior by providing the majority of things that can
impact nir options when getting them. Some callers (e.g. meta blit
shaders or libclc) don't bother, because they are known to have
contents that are unaffected by these options. Other callers might
munge more properties afterwards, but this minimizes that.

Note that lower_helper_invocation was incorrectly being turned off
for SM6.6+ by some callers, despite load_helper_invocation being
unimplemented by the backend.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22952>

dzn: Enable 64-bit ints and floats

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22952>

d3d12: Convert from D3D shader model to Mesa shader model earlier

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22952>

spirv2dxil: Support int64 and doubles

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22952>

zink: Always set a blend state for shader-db

If we're compiling shaders in shader-db, with shader-db's ./run and
ZINK_DEBUG=shaderdb, we won't get much state set on the graphics pipeline, since
shader-db doesn't actually do any rendering. For a driver like RADV, that is
*almost* ok... Since we use dynamic vertex input, we don't need to make up any
state for vertex inputs; since we use dynamic rendering, we don't need to make
up any render attachments. All of that being said, we *do* need to make up a
blend state to ensure that the Vulkan driver doesn't optimize away all of
store_derefs in the fragment shader (and in turn, optimize the entire fragment
shader away, if there are no image/SSBO writes.) So set the obvious blend state,
fixing fragment shaders in shader-db with zink + radv.

I don't know why other people would want to use Zink with shader-db, but for me
it's an easy way to test ACO, at least until radeonsi gains aco support.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22948>

spirv: Use NIR_PASS for spirv2nir --optimize

This allows us to use NIR_DEBUG=print to see each step.
Also use an OPT macro to make code slightly more readable.

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22764>

spirv: Do more on spirv2nir --optimize

Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22764>

intel/mi_builder: fixup tests for newer kernel uAPI

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22966>

anv: Set memory types supported by Xe KMD

Due the lack of APIs to set mmap modes, Xe KMD can't support the same
memory types as i915.
So here adding a i915 and Xe function to set memory types supported
by each KMD.

Iris function iris_xe_bo_flags_to_mmap_mode() has a table with all the
mmaps modes of each type of placement.

Signed-off-by: José Roberto de Souza <jose.souza@intel.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22906>

radeonsi: Use vcn version instead of CHIP family for VCNs

Decouple it from CHIP family, based on HW query infomation.

Signed-off-by: Leo Liu <leo.liu@amd.com>
Reviewed-by: Boyuan Zhang <Boyuan.Zhang@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22904>

amd: Add vcn ip version info

And make it support for kernel w/wo ip_discovery.

Signed-off-by: Leo Liu <leo.liu@amd.com>
Reviewed-by: Boyuan Zhang <Boyuan.Zhang@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22904>

radeonsi: Remove redundant vcn_decode from info

Use the number of queue instead.

Signed-off-by: Leo Liu <leo.liu@amd.com>
Reviewed-by: Boyuan Zhang <Boyuan.Zhang@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22904>

dzn: Fix segmentation fault when Direct3D 12 user mode
driver from at least one of GPUs is not available.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22961>

agx: Optimize multiplies

We have an imad instruction and our iadd has a small immediate shift on the
second source. Together, these allow expressing lots of integer multiplies more
efficiently. Add some rules to optimize these now that the backend compiler can
ingest the optimized forms.

Half-register changes are from load_const scheduling changing in some vertex
shaders.

   total instructions in shared programs: 1539092 -> 1537949 (-0.07%)
   instructions in affected programs: 167896 -> 166753 (-0.68%)

   total bytes in shared programs: 10543012 -> 10533866 (-0.09%)
   bytes in affected programs: 1218068 -> 1208922 (-0.75%)

   total halfregs in shared programs: 483180 -> 483448 (0.06%)
   halfregs in affected programs: 1942 -> 2210 (13.80%)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22695>

agx: Fix packing of imsub instructions

The negate for imad is on the third source (a * b - c), not the second source.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/22695>