nvk: Don't use nir_ssa_for_src()

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24655>

nir: Drop "SSA" from NIR language

Everything is SSA now.

   sed -e 's/nir_ssa_def/nir_def/g' \
       -e 's/nir_ssa_undef/nir_undef/g' \
       -e 's/nir_ssa_scalar/nir_scalar/g' \
       -e 's/nir_src_rewrite_ssa/nir_src_rewrite/g' \
       -e 's/nir_gather_ssa_types/nir_gather_types/g' \
       -i $(git grep -l nir | grep -v relnotes)

   git mv src/compiler/nir/nir_gather_ssa_types.c \
          src/compiler/nir/nir_gather_types.c

   ninja -C build/ clang-format
   cd src/compiler/nir && find *.c *.h -type f -exec clang-format -i \{} \;

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Acked-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Acked-by: Emma Anholt <emma@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24585>

nir: clang-format src/compiler/nir/*.[ch]

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Add a .clang-format file

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Add a do to the do/while in nir_const_value_t_array()

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Wrap pass macros in braces

This makes clang-format not mess them up so bad.  It's also probably a
good idea to make sure anything we declare in the macro is properly
scoped.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Pretty format type mapping helpers

One of them was even breaking after every return statement. Classy...

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: More manual formatting

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Don't clang-format debug print setup

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Don't clang-format a couple typedefs

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Don't clang-format const_value helpers

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

nir: Re-align a couple enums and add clang-format comments

I actually kinda care about the human-readable formatting of these
enums.  Keep clang-format from messing them up.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

clang-format: Set the default ColumnLimit to 0

This is a better default than any actual limit because it causes
clang-format to mostly leave line wrapping alone.

Suggested-by: Marek Olšák <maraeo@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

clang-format: nir_foreach_src is not a foreach macro

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

clang-format: Add nir_foreach_reg_*

Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24382>

android: Introduce the Android buffer info abstraction

Both EGL and Vulkan implementations require obtaining buffer metadata,
e.g., format, modifier, offsets, strides, etc.

Currently, mesa3d doesn't have a generic solution, and every Vulkan
implementation uses its getters. Most of the getters rely on
kernel metadata storage that is available for x86-based GPU drivers.

ARM-based Vulkan drivers rely on userspace metadata sharing, making it
important to use advanced metadata API. Otherwise, the driver will work
with limited functionality (no YUV, lack of support for modifiers, etc.)

Current EGL buffer getter implementation is advanced enough and used as
a base for a common Android buffer-getter logic.

Use example:

    void
    android_buffer_test(android_handle_type *a_handle)
    {
       // First, get the gralloc object. It will be created if it doesn't
       // exist. Use U_GRALLOC_TYPE_AUTO to let the implementation choose
       // the best gralloc
       struct u_gralloc *gralloc = u_gralloc_create(U_GRALLOC_TYPE_AUTO);

       // Prepare the internal handle structure (hal_format and
       // pixel_stride are required for the fallback implementation).
       // Both Vulkan and EGL clients expose HAL format / pixel stride
       // in their structures.
       u_gralloc_buffer_handle hnd = {
          .handle = a_handle->native_handle,
          .hal_format = a_handle->hal_format,
          .pixel_stride = a_handle->pixel_stride,
       };

       // Get the basic buffer info
       u_gralloc_buffer_basic_info basic_info;
       int ret = u_gralloc_get_buffer_basic_info(gralloc, &hnd, &basic_info);
       if (ret) {
          // Handle the error
       }

       // Get the color info
       u_gralloc_buffer_color_info color_info;
       ret = u_gralloc_get_buffer_color_info(gralloc, &hnd, &color_info);
       if (ret) {
          // Handle the error
       }

       // unref the gralloc object
       u_gralloc_destroy(&gralloc);
    }

Signed-off-by: Roman Stratiienko <r.stratiienko@gmail.com>
Reviewed-by: Chia-I Wu <olvaffe@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18215>

util: Add NONNULL macro

Macro leverages __attribute__((__nonnull__)) to help users mark
the function parameter that isn't allowed to be NULL.

Suggested-by: Eric Engestrom <eric@igalia.com>
Signed-off-by: Roman Stratiienko <r.stratiienko@gmail.com>
Reviewed-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18215>

ci/deqp: really remove the uncompressed results.csv file

Arguments are parsed in order, so the `-c` here causes that first `--rm`
won't get applied and then it gets overriden by `-o` which writes the file.

Sadly zstd won't report this as a issue, but that's being reported in [1],
so maybe in future versions zstd will warn about it.

[1] https://github.com/facebook/zstd/issues/3719

Fixes: d110299d77c9 ("ci/deqp-runner: compress results.csv before uploading it to GitLab")

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24651>

anv: move genX(rasterization_mode) to gfx8_cmd_buffer.c

Only used there.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24632>

anv: get rid of genX(emit_multisample)

The initialization can be simplified and the real programming moved
over to genX_pipeline.c

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24632>

intel/decoder: add options to decode surfaces/samplers

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24632>

anv: emit 3DSTATE_GS only once per pipeline

Following 71ebd9b9d7, 3DSTATE_GS can be emitted as part of the
pipeline batch and as a dynamic state. Just do the latter.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 71ebd9b9d7 ("anv,hasvk: respect provoking vertex setting on geometry shaders")
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24632>

blorp: drop programming of 3DSTATE_(MESH|TASK)_SHADER

We can disable the MESH/TASK stages by just using the CONTROL
instruction and keep the rest of the HW programmed as before.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24632>

intel/decoder: constify some input parameters

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24632>

ci/farms: enabled Microsoft job only when conditions are met

Otherwise it gets run by default on newly pushed fork branch.

In general, this is copy-paste of `.container-rules`.

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24604>

zink/nir: add a zink specific intrinsic for push constants

Push costants in Zink are not flat indexed like in vulkan drivers which
makes the `nir_intrinsic_load_push_constant` intrinsic inappropiate.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24401>

zink: remove unused indices from `nir_load_push_constant` calls

Zink passes `base` and `range` indices to this intrinsic despite those
being ignored by nir_to_spirv. This change removes them completely.

Reviewed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Reviewed-by: Mike Blumenkrantz <michael.blumenkrantz@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24401>

ci/freedreno: document vs-nested-return-sibling-loop2 flake on Adreno 530

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24607>

ci/amd: add glx@glx-visuals-depth flake to raven

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24607>

ci/build: limit debian-build-testing to 30 minutes

Previous 45 minutes is too much.

 - 5 - 15 minutes mesa LTO build
 - 5 - 15 minutes shader-db run

Should be safe, in case something fails, we still can make another run.

Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24607>

nir/opt_if: also rewrite uniform uses for read_invocation

No Foz-DB changes, but maybe it matters in the future because dxil-spirv
will use read_invocation for WaveReadLaneFirst in fragment shaders.

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24618>

vulkan: ignore VkPipelineColorWriteCreateInfoEXT if the state is dynamic

Cc: mesa-stable
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24601>

egl/drm: fix EGL_EXT_buffer_age with gbm contexts

Fix remaining dri2 check from 3d59f4cfcbc, which caused gbm contexts to
not expose EGL_EXT_buffer_age anymore.

Fixes: 3d59f4cfcbc ("egl/drm: Use IMAGE_DRIVER instead of DRI2_LOADER")

Reviewed-by: Daniel Stone <daniels@collabora.com>
Signed-off-by: Erico Nunes <nunes.erico@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24630>

asahi: Legalize compression before blitting

Fixes invalid recursive blitting.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Do not allow creating vec8

mem_access_bit_size needs to split up 64x4 into 2 loads. Fixes:

dEQP-VK.spirv_assembly.instruction.compute.64bit_compare.int64.comp_opiequal_vector

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Handle <32-bit local memory access

I don't know if this is possible to hit with GL, but it is with Vulkan. Fixes:

dEQP-VK.spirv_assembly.instruction.compute.workgroup_memory.*

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Handle f2f16_rtne like f2f16

TBD whether we can control round modes later on.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Lower f2f16_rtz

Based on what VIR does.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Fix 64-bit immediate moves

Don't truncate them in the builder. Fixes:

dEQP-VK.spirv_assembly.instruction.compute.convertutof.uint64_to_float32_4294967296

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Fix uadd_sat packing

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Handle conversions to 8-bit

These can't be lowered by nir_lower_bit_sizes but it doesn't actually matter.
Fixes SPIR-V conversions tests.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Lower f2u8/f2i8

The less 8-bit we need to emulate in the backend the better.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Handle blocks with no predecessors

This can come up with loops with SPIR-V. Fixes:

dEQP-VK.spirv_assembly.instruction.compute.loop_control.none

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Fix incorrect BO bitmap reallocations

If the BO handle is greater than 2x what fits inside the current bitmap
size, then we end up overflowing. Make sure to always reallocate to a
large enough bitmap, not just 2x the previous size.

Found while replaying firefox apitraces with looping (which apparently
leaks a ton of objects, but that might just be apitrace).

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Handle non-written RTs correctly

When a bound RT is not written to, we need to force the pass type to
translucent to ensure that this draw does not cull draws that do write
to that RT.

Fixes Inochi2D regression after c24b753378.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Move nir_lower_fragcolor out of agx_preprocess_nir

Do not apply "nir_lower_fragcolor" in the common code.

This fix a crash on agxv side when a frag shader have SSBO writes.

This is caused by "nir_lower_frag_color" assuming that every
"store_deref" will have a variable backing the
output.

Signed-off-by: Mary <mary@mary.zone>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Stop setting forwarding bit

We need actual analysis to set it properly, and improperly setting it can cause
random data dependency hazards it turns out. Stop setting it. Fixes some flaky
tests with shuffle code inserted.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Remove unused allocation

Relic from early RA attempts.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Don't reuse ssa_to_reg across blocks

This avoids an entire class of bugs with live range splitting. Fixes with
AGX_MESA_DEBUG=demand:

   dEQP-GLES31.functional.separate_shader.random.8

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Don't use ssa_to_reg across blocks

This is a footgun with live range spltiting.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Assert invariant stated in the comment

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Use agx_replace_src

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Stop setting registers after the shader

Leftover from before live range splitting. Should be a no-op.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Set phi sources in predecessors

This ensures correctness with live range splits. Now agx_set_sources is only for
non-phis where it makes sense.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Fix accounting for phis

All affected shaders are in pubg. Presumably, with the new demand calculation,
RA is hitting a higher target thread count at the expense of a little more live
range splitting.

   total instructions in shared programs: 1773295 -> 1773310 (<.01%)
   instructions in affected programs: 6058 -> 6073 (0.25%)
   helped: 0
   HURT: 15
   Instructions are HURT.

   total bytes in shared programs: 11695360 -> 11695450 (<.01%)
   bytes in affected programs: 40496 -> 40586 (0.22%)
   helped: 0
   HURT: 15
   Bytes are HURT.

   total halfregs in shared programs: 530844 -> 530724 (-0.02%)
   halfregs in affected programs: 1785 -> 1665 (-6.72%)
   helped: 15
   HURT: 0
   Halfregs are helped.

   total threads in shared programs: 18909440 -> 18910400 (<.01%)
   threads in affected programs: 12480 -> 13440 (7.69%)
   helped: 15
   HURT: 0
   Threads are helped.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Allow 64-bit memory regs

The mask is based on the format, which can be at most 32-bits per channel. So if
we have 64-bit loads/stores we're still using a 32-bit format with double the
bits set in the mask. This will fix validation fails with spilling.

No shader-db changes.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Maintain ctx->max_reg while assigning regs

We can't calculate after since ssa_to_reg[] gets overwritten during live range
splits. Theoretical issue only, but let's fix it while squashing live range
splitting bugs.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Fix AGX_MESA_DEBUG=demand

No wonder it seemed like it wasn't doing anything!

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Optimize logical_end removal

We know logical_end instructions are only at the end of the block (validated),
so by changing how we iterate the pass goes from O(instructions) to O(blocks)
which is strictly better.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Optimize swaps of 2x16 channels

We can use extr to swap the low and high halves of a 32-bit register in one
instruction.

No shader-db changes, but it reduces xor's on a deqp I'm looking at. Yes, I'm
procrastinating on debugging deqps, how'd you guess?

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Use compressed fadd/fmul encodings

See applegpu commit b9b3582 ("FBinaryInstructions have compressed encodings")

   total bytes in shared programs: 11717310 -> 11716524 (<.01%)
   bytes in affected programs: 317504 -> 316718 (-0.25%)
   helped: 196
   HURT: 0
   Bytes are helped.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Require an immediate for `nest`

There's no good reason to allow non-immediate nesting values, and this lets us
use the (smaller) mov_imm instruction without special casing. This matches what
Metal produces, so it seems like a good preference.

   total bytes in shared programs: 11720338 -> 11717310 (-0.03%)
   bytes in affected programs: 2341580 -> 2338552 (-0.13%)
   helped: 1385
   HURT: 0
   Bytes are helped.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Fix length bit confusion

Man, this expression was wrong. First of all, raw is 64-bit so our mask needs to
be too. Second, length is in bytes -- not bits -- so we need to multiply by 8 to
get something sensible. In effect, the old wrong expression would always use the
long encoding for ALU instructions... whoops. This particular bug probably goes
back to the very first version of agx_pack...

Massive improvement in code density. Noticed while comparing assembly with the
blob. It's my Saturday, I can pointless optimize if I want to.

   total bytes in shared programs: 12175112 -> 11720338 (-3.74%)
   bytes in affected programs: 11963800 -> 11509026 (-3.80%)
   helped: 16624
   HURT: 0
   Bytes are helped.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Optimize out pointless else instructions

Now that they're in the right blocks, this is easy. Includes an informal proof
and the implementation itself is built around a finite state machine, which
together meant this code worked on its first try :~)

And hey, it's a pointless little instruction saving optimization I've wanted to
do for a while~

Major note is that this HAS to be done after register allocation, since it
doesn't update the control flow graph and would introduce critical edges
if it tried to actually deleted the else block. The intuitive reason for this is
simple: sometimes RA needs to insert instructions into the else block, even if
it was empty in the original NIR, so we always need an else block even if we can
delete it with this pass after RA.

   total instructions in shared programs: 1778390 -> 1776725 (-0.09%)
   instructions in affected programs: 268459 -> 266794 (-0.62%)
   helped: 1013
   HURT: 0
   Instructions are helped.

   total bytes in shared programs: 12185102 -> 12175112 (-0.08%)
   bytes in affected programs: 1927524 -> 1917534 (-0.52%)
   helped: 1013
   HURT: 0
   Bytes are helped.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Use unconditional else instruction

Rather than duplicating the condition. This matches the blob, so is presumably
the most energy-efficient way of expressing the logic.

No shader-db changes.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Put else instructions in the right block

According to Dougall's pseudocode, else_icmp operates as:

  if r0l == 0:
    r0l = n
  elif r0l == 1:
    if cc.compare(A[thread], B[thread]):
      r0l = 0
    else:
      r0l = 1

  exec_mask[thread] = (r0l == 0)

Notice that the comparison only happens when r0l == 1, that is, for threads that
are about to enter the else block. Threads that just executed the if body are
still active (r0l = 0) and skip the comparison. As such, the sources of
else_icmp are only read in the else block, and hence the whole instruction
should be placed in the else block for correctness with respect to live range
splitting.

shader-db is a wash, but shows some improvements due to correctly modelling the
liveness of the condition variable.

   total instructions in shared programs: 1778376 -> 1778390 (<.01%)
   instructions in affected programs: 14753 -> 14767 (0.09%)
   helped: 35
   HURT: 39
   Inconclusive result (value mean confidence interval includes 0).

   total bytes in shared programs: 12185018 -> 12185102 (<.01%)
   bytes in affected programs: 101522 -> 101606 (0.08%)
   helped: 35
   HURT: 39
   Inconclusive result (value mean confidence interval includes 0).

   total halfregs in shared programs: 531174 -> 531032 (-0.03%)
   halfregs in affected programs: 2320 -> 2178 (-6.12%)
   helped: 40
   HURT: 1
   Halfregs are helped.

   total threads in shared programs: 18909184 -> 18909440 (<.01%)
   threads in affected programs: 1792 -> 2048 (14.29%)
   helped: 2
   HURT: 0

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Do not move bindless handles

NIR->AGX translation relies on bindless handles being vec2 instructions with a
constant first index. Moving the entire vec2 into the preamble would mess this
up, so tell nir_opt_preamble to never do this. It's still allowed to move the
offset calculation into the preamble, if it thinks that's beneficial.

Fixes dEQP-GLES31.functional.shaders.opaque_type_indexing.sampler.*

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Fix shader info with sample mask writes

Otherwise the discard_agx isn't lowered.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Enable explicit coherency for G14D (multi-die)

It turns out we do need this explicit coherency dance for G14,
but only on G14D.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi,agx: Upload constant buffers immediately

The lifetime of the constant buffer's user_buffer is not guaranteed
to last until agx_upload_uniforms.
Fixes the same ASAN issue mesa/mesa!21685 is trying to address.

Fixes: 080b05e29e1 ("asahi: Add Gallium driver")
Signed-off-by: Janne Grunau <j@jannau.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Fix atomics with no destination

We need to:

* properly null out the dest in DCE.
* not assert out when packing with null dest

Fixes potential reg pressure blow up with atomics that don't use their
destinations, though I don't see shader-db changes.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx/dce: Use the helper

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Don't lower load_local_invocation_index

We have an SR for it, which can save a bit of math. This came up while working
on the spiller.

   total instructions in shared programs: 1778396 -> 1778376 (<.01%)
   instructions in affected programs: 3036 -> 3016 (-0.66%)
   helped: 10
   HURT: 3
   Instructions are helped.

   total bytes in shared programs: 12185182 -> 12185018 (<.01%)
   bytes in affected programs: 38640 -> 38476 (-0.42%)
   helped: 18
   HURT: 2
   Bytes are helped.

   total halfregs in shared programs: 531218 -> 531174 (<.01%)
   halfregs in affected programs: 471 -> 427 (-9.34%)
   helped: 6
   HURT: 0
   Halfregs are helped.

   total threads in shared programs: 18909056 -> 18909184 (<.01%)
   threads in affected programs: 1280 -> 1408 (10.00%)
   helped: 2
   HURT: 0

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi,agx: Fix stack buffer overflow in agx_link_varyings_vs_fs

Discovered while running dEQP-EGL under address sanitizer.

Fixes: f3877f56ba7 ("asahi,agx: Rewrite varying linking")
Signed-off-by: Janne Grunau <j@jannau.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Force linear for SHARED buffers with no/implicit modifier

Consumers might not pass through the modifier information in this case.

Fixes XWayland/mutter using dma-buf v4 feedback (though the fact they
try to use implicit modifiers is likely a bug on their end, and will
decrease performance).

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi,agx: Respect no16 even for I/O

Don't call lower_mediump_io for no16. This is helpful for debugging and soon
driconf-shaming apps with broken precision qualifiers.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Impose limits on resource shadowing

Apps can have pathological use cases where huge resources are shadowed
repeatedly. An app that alternately writes to a resource and then uses
it to draw can create an unbounded amount of shadow BOs.

To fix this, introduce both a maximum resource size for shadowing, and a
maximum cumulative size that resource may be shadowed before we start
flushing readers. The flush path then clears the counter, as does the
happy path where there are no readers left after flushing writers.

Fixes massive memory bloating in Firefox and probably others.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Print info about shadowed resources

If resource and perf debugging are both enabled, this prints resource
info for shadowed resources.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Always use resource size, not BO size

BOs can be oversized, as they can come from the BO cache. Make sure to
always use the resource layout size, not the BO size, when we need this
for some reason.

This fixes BO shadowing creating overlarge BOs, and also the attachment
size for submissions (probably doesn't matter, but it's more correct now).

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Fix race in BO stats accounting

These counters are accessed without locking, so they need to be atomic.
Should be cosmetic only.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Do not overallocate BOs by more than 2x

This is not likely to be useful, and might take over a correctly-sized
BO that is going to be reused later.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Add a noshadow debug flag

This lets us trivially test whether resource shadowing helps or hurts
any given workload.

Signed-off-by: Asahi Lina <lina@asahilina.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Handle 8-bit vecs

These should "just" work, promoting the 8-bit channels to 16-bit registers
internally, allowing us to use our 8-bit stores with 8-bit data vectors packed
in 16-bit registers. All other non-conversion ALU gets lowered by the previous
patch, this is just needed for simple things like nir_op_vec of lowered math
passed to a vectorized store.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

agx: Lower 8-bit ALU

No hardware support for it.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Move a bunch of helpers to common

These have no real Vulkan or Gallium dependence and are (as such) useful for
both VK and GL without any real change in level of abstraction. Do the code
motion.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

asahi: Stub num_dies

We'll use it in the upstreamable driver portion soon.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24635>

nvk: Support dynamic state for enabling sample locations

When switching dynamically we should also push the corresponding sample
locations, the default when disabled or the custom ones when enabled.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24592>

nvk: Fix support for VK_EXT_sample_locations

Fixes some crashes on sample locations pipeline tests.
The implementation was already there but the device properties were
missing.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24592>

gallium/st: lower NV21 to R8_B8G8 instead of G8_B8R8

When NV21 lowering with hardware sampling and shader CSC was added, the
incorrect PIPE_FORMAT_G8_B8R8_UNORM was used. That format is supposed to
represent vulkan NV12 instead.

This commit introduces PIPE_FORMAT_R8_B8G8_UNORM, which correctly describes the
gallium mapping for YUV CSC, with R as Y, instead of G as Y.

Fixes: 26e3be513dc ("gallium/st: add support for PIPE_FORMAT_NV21 and PIPE_FORMAT_G8_B8R8_420")
Signed-off-by: Italo Nicola <italonicola@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24266>

pan/bi: add support for I420 and YV12 sampling

These formats can be directly sampled, and they have a lower stride
alignment requirement.

Signed-off-by: Italo Nicola <italonicola@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24266>

gallium/st: add non-CSC lowering of YV12 as PIPE_FORMAT_R8_B8_G8_420

YV12 is the same as DRM_FORMAT_YVU420.
We lower it to PIPE_FORMAT_R8_B8_G8_420, which is equivalent to
PIPE_FORMAT_R8_G8_B8_420 with U/V planes swapped.

This is used for hardware that can sample from YUV but need CSC in shader.

Signed-off-by: Italo Nicola <italonicola@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24266>

gallium/st: add non-CSC lowering of I420 as PIPE_FORMAT_R8_G8_B8_420

This new format is similar to PIPE_FORMAT_G8_B8_R8_420, but with R as Y, G as U
and B as V. The need for two diferent formats here is because gallium maps the
YUV channels differently from vulkan.

Some hardware, e.g. Mali GPUs, can sample from I420 but need CSC in shader,
this patch implements that.

Signed-off-by: Italo Nicola <italonicola@collabora.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24266>

radeonsi/vcn: Update rate control when framerate changes with HEVC

Similar to H264/AV1, check for framerate changes and update
rate control also with HEVC.

Reviewed-by: Boyuan Zhang <boyuan.zhang@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24475>

aco: always use rtne for fquantize2f16

The SPIR-V spec says:

If Value is positive with a magnitude too large to represent as a
16-bit floating-point value, the result is positive infinity.
If Value is negative with a magnitude too large to represent as a
16-bit floating-point value, the result is negative infinity.

This is only the case for rtne v_cvt_f16_f32

Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24617>

agx: Lower flat shading in NIR

We get this as part of the lowering we added for interpolateAtOffset.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24498>

agx: Add interpolateAtOffset lowering pass

Add a lowering pass that lowers interpolation to math on the coefficient
registers. This handles interpolateAtOffset, as well as flat shading as an easy
special case.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24498>

agx: Forcibly vectorize pointcoord coeffs

This avoids regressions from scalarizing pointcoord loads.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24498>

agx: Set lower_fisnormal

We're going to generate this in our interpolation lower.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24498>

agx: Allow more varying slots

Don't overflow.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24498>

agx: Implement nir_intrinsic_load_coefficients_agx

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24498>