r600: Fix lower-to-scalar on TTN path

We must limit what alu ops are scalazized, and since we now
run finalize_nir we also have to make sure not to try to
scalarize backend specific instructions that are introduced
there.

While we touch the code, also remove the vectorization, it is
no longer helping.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7747

Fixes: a81c50a2 (r600/sfn: implement finalize_nir)

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19892>

r600: reference glsl type system during translation

VL doesn't initialize the type systen, but creates shaders in TGSI.
When these are translated to NIR and then compiled in the driver
we need to have a reference to the glsl type system.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7453

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19883>

intel_batch_decoder: disasm shaders when decoding states besides VS

Previously only 'decode_vs_state' would dump the referenced shader,
which meant we completely ignored every other shader when decoding
the '3DSTATE_PIPELINED_POINTERS' command.

Move the program disassembly logic from 'decode_vs_state' into a
common 'disasm_program_from_group' helper and call it from every
other decode_*_state function, too.

Signed-off-by: Daniil Tatianin <99danilt@gmail.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19880>

v3d: Minor fixes on sand8 blit based on sand30 modifications

* load_uniform for sand8_stride is uint32 instead of int32 and its range
  is 4 instead of 1 as it is counted in bytes.
* Now we save and restore constant buffer 1 properly for the ubo used
  in the blit. We need to take into account that in V3D the first UBO
  with index 0 is stored on constant buffer 1, because gallium uses
  internally contant buffer 0 (See for reference commit c8212731e7d67a7)
* Removed not needed return.
* Added shader information about uniforms, ubos, inputs and outputs.
* Fixed typos in the comments.

Fixes: 95c4f0f91098 "v3d: Enables DRM_FORMAT_MOD_BROADCOM_SAND128 support"
Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19639>

v3d: Also expose DRM_FORMAT_MOD_BROADCOM_SAND128 with PIPE_FORMAT_P030

Now we can display h265-10bit decoded frames formats on Raspberry Pi 4
devices on GL/ES applications.

v2: Don't expose LINEAR or UIF formats for P030 format.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19639>

v3d: Blit for P030 format with BROADCOM_SAND128 modifier to P010 UIF

Implements the support to blit SAND modifier with columns 128-bytes-wide
support for P030 format to P010 with UIF layout. This allows sampling
from H265 10-bit frames exported by the video decoder on the Raspberry
Pi 4 devices.

When a DRM_FORMAT_MOD_BROADCOM_SAND128 is enabled with an imported P030
texture. The sand30 blit converts the Luma and Chroma planes to
a tiled P010 format that can be sampled using gallium YUV lowerings
without the interleaved 128-bytes-wide-columns.

This patch follows a similar approach to SAND8 blit but extracting luma
and chroma components from the DRM_FORMAT_P030 format. P030 is a two
plane YCbCr420 format where 3 10 bit components with 2 padding bits are
packed in 4 bytes.

index 0 = Y plane, [31:0] x:Y2:Y1:Y0 2:10:10:10 little endian
index 1 = Cr:Cb plane, [63:0] x:Cr2:Cb2:Cr1:x:Cb1:Cr0:Cb0
                       [2:10:10:10:2:10:10:10] little endian

After the sand30_blit is done, the shadow texture is an UIF tiled texture
with an R16_UNORM format for luma and R16G16_UNORM for chroma.

To reduce the number of texture-fetch operations during the blit, we
read pairs of 32-bit dwords. They include 6 10-bit unorm components.
And then we write 4 UNORM16 components from an uvec4 because our render
targets do not support writing to UNORM16 formats.

As sampling will be done using 16bpp (luma) and 32bpp (chroma), the
sand30_blit writes consider the different microtile layouts of UIF
format between 64, 32 and 16 bpp.

v2: Fixes save and recovery of constant buffers (Iago)
    Typo corrections. (Iago)
    Removed not needed return. (Iago)
    Added shader information about uniforms, ubos, inputs and outputs.

Reviewed-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19639>

gallium/dri: Add P030 format

Acked-by: Iago Toral Quiroga <itoral@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19639>

zink: require extendedDynamicState3ColorBlendEquation for full ds3

Cc: 22.3 mesa-stable
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19851>

radv: Support accelerationStructureCaptureReplay.

The address itself was already stable assuming that the memory itself was
allocated with capture/replay. Enable the feature flag and add an equality
check to return VK_ERROR_INVALID_OPAQUE_CAPTURE_ADDRESS_KHR on mismatch.

Tested with:
- dEQP-VK.ray_tracing_pipeline.capture_replay.*
- q2rtx gfxrecon replays correctly without major errors.
  * There are debug logs about VkBuffers missing opaque address
  for unknown reason, however the AS part is confirmed to be correctly
  captured.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19841>

radv/ci: add one more pipeline barrier test as flake

With the new memory type, this test timeout most of the time but it
always pass.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

radv: advertise VK_EXT_descriptor_buffer

Pass dEQP-VK.*descriptor_buffer*.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

radv: implement VK_EXT_descriptor_buffer

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

radv: always restore NULL descriptor sets for meta operations

A valid but NULL descriptor set indicates a descriptor buffer.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

radv: rework writing image/buffer/accel_struct descriptors slightly

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

radv: avoid RMW operations in radv_make_texel_buffer_descriptor()

This would hurt performance depending on the memory type.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

radv: force 32-bit address-space for descriptor buffers usage

Descriptor pointers are passed through shaders with 32-bit.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

radv: add 32-bit memory types for visible VRAM and cached GTT

These would be only exposed for descriptor buffers which need to be
CPU accessible and because descriptor pointers are 32-bit.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19808>

asahi: Fix memory leak on error path.

Fix defect reported by Coverity Scan.

Resource leak (RESOURCE_LEAK)
leaked_storage: Variable nresource going out of scope leaks the storage it points to.

Fixes: 7522f4f7147 ("asahi: Make resource creation code modifier-aware")
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19889>

ac/llvm: fix 16bit varying llvm compile error

Found when 16bit vec3 varying with llvm14 (not found
when llvm15), one 32bit vec4 slot is filled like this:
  vec3[0] | undef
  vec3[1] | undef
  vec3[2] | undef
  undef   | undef

LLVM error is for the elements with undef:
  %287 = insertelement float %280, half %279, i64 0

After this change, we get:
  %287 = insertelement <2 x half> %280, half %279, i64 0

Fixes: 279eea5bda2 ("amd/llvm: Transition to LLVM "opaque pointers"")
Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Signed-off-by: Qiang Yu <yuq825@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19848>

asahi: Fix memory leak on error path.

Fix defect reported by Coverity Scan.

Resource leak (RESOURCE_LEAK)
leaked_storage: Variable rsc going out of scope leaks the storage it points to.

Fixes: 01964625eb2 ("asahi: Implement agx_resource_from_handle")
Signed-off-by: Vinson Lee <vlee@freedesktop.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19887>

mapi: update gles 1.1 extention packet link

Signed-off-by: Yusuf Khan <yusisamerican@gmail.com>
Reviewed-by: Yonggang Luo <luoyonggang@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19874>

radv/rt: use explicitly named sbt entries in radv_pipeline_group_handle

This improves readability, makes it less error-prone and
allows for easier changes in future.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19850>

v3d(v): account for debug flags when using the cache

Signed-off-by: Eric Engestrom <eric@igalia.com>
Cc: mesa-stable
Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19796>

asahi: Advertise all supported renderable formats

Now that we have the new tilebuffer infrastructure in place, all renderable
formats should be working.

This reverts commit 32ab63b4fcaaaf832d549ddcf7b3a225c5f73df4.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi,agx: Use new tilebuffer infrastructure

Flag day change to replace the previous hardcoded background/end-of-tile shaders
and the API-style load/store_output in fragment shaders with the generated
shaders and lowered *_agx intrinsics. This gets us working non-UNORM8 render
targets and working MRT. It's also a step in the direction of working MSAA but
that needs a lot more work, since the multisampling programming model on AGX is
quite different from any of the APIs (including Metal).

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add new clear/reload/store infrastructure

With multiple render targets, it's not practical to generate all
variants of the background and end-of-tile programs at start up. Rather
than trying, add a hash table of meta program keys to background
programs, and compile variants as they're needed.

With the new infrastructure, it's sensible to handle clears with the
same code path as reloads. In addition to getting us closer to multiple
render target support, this gets us support for non-RGBA8 render
targets, as the u8norm tilebuffer format was baked into the hardcoded
clear shader and store shaders used before.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Use correct tib settings for USC

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Extend texture descriptor packing for MSAA

Multisampling uses different values of the dimension enum in tandem with a new
samples field. Handle this in agx_pack_texture (split off here) so we can use
the new functionality for texture descriptors in reloads too.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Remove some bogus asserts

Hitting in dEQP-GLES31

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Implement texture_barrier trivially

For the advanced blending tests.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Calculate tilebuffer layout per batch

It won't be fixed soon.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add agx_usc_shared_none helper

Convenience for vertex USC programs.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add R16 SNORM formats

For completeness, since we do have hardware for this.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add more XML via PowerVR

These bits are the same as RGX.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add note to XML about 16-bit varyings

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Identify more depth-related fields in XML

Needed for gl_FragDepth writes.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add XML for layered rendering

We don't need to support this for a while but it's good to know the mechanism.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add NIR pass to lower tilebuffer access

The compiler can't handle load/store_output directly for nontrivial tilebuffer
layouts. Add a NIR pass to lower these intrinsics, applying a given layout.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add tilebuffer layout helpers

Laying out the tilebuffer is nontrivial and a task shared between GL and VK, so
add unit-tested helpers.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add some notes about unknowns to the XML

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Identify multisampling fields of shared layout

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Add _with_bo pool uploads

Will be useful for managing our meta shaders.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

agx: Add block_image_store instruction

This hw instruction writes out an entire block from the tilebuffer to an
attached render target (PBE descriptor). It is used (only?) in end-of-tile
shaders to implement write out. We need to handle it in the compiler as a
prerequisite to compiling end-of-tile shaders ourselves, instead of hardcoding.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

agx: Add helper to map pipe formats to agx_formats

Or a restricted subset thereof anyway.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

agx: Implement nir_texop_txf_ms

Mutlisampled texture fetch (txf_ms) is encoded like regular txf. However, we now
need to pack the multisample index in the right place, which we do by extending
our existing NIR texture source lowering pass. 2D MS arrays use a new value of
dim which requires tweaking the encoding slightly. Otherwise, everything is
bog standard.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

ail: Handle multisampling

It appears that multisampled textures on AGX have all samples of the same pixel
contiguous in memory, effectively using the layout of a single-sampled texture
with a larger block size. Handle in ail.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Make libasahi_lib depend on libasahi_decode

The track_alloc and track_free symbols are used, we need to link them in.
Depending on build flags / environment / etc, fixes the potential build error
hit by a CI job:

mold: error: undefined symbol: agxdecode_track_alloc
>>> referenced by agx_device.c
>>>               src/asahi/lib/libasahi_lib.a(src/asahi/lib/libasahi_lib.a.p/agx_device.c.o):(agx_shmem_alloc)>>> referenced by agx_device.c
>>>               src/asahi/lib/libasahi_lib.a(src/asahi/lib/libasahi_lib.a.p/agx_device.c.o):(agx_bo_create)
mold: error: undefined symbol: agxdecode_track_free
>>> referenced by agx_device.c
>>>               src/asahi/lib/libasahi_lib.a(src/asahi/lib/libasahi_lib.a.p/agx_device.c.o):(agx_bo_unreference)

...when trying to link with libasahi_lib without libasahi_decode for unit tests.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

asahi: Use PIPE_FORMATs for driver-compiler ABI

This avoids exposing the ISA-internal agx_format to the driver, instead hiding
it behind a real PIPE_FORMAT. This lets us use real pipe formats in formatted
intrinsics in NIR, which is convenient; it will allow us to simplify the
compiler/driver ABI; and it lets us use common format helpers (e.g.
util_format_get_blocksize) for the internal formats in driver lowering.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

nir: Define AGX intrinsics for local pixel access

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19871>

turnip: Apply the RB_DBG_ECO_CNTL_blit workaround.

On blob v512.490 on a615, using WRAP_GPU_ID to fake GPU versions, I see
0x41 used everywhere, except for BLIT_OP_SCALE on a630.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19794>

freedreno: Don't WFI and set RB_DBG_ECO_CNTL if it's not changing.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19794>

freedreno: Update RB_DBG_ECO_CNTL/RB_DBG_ECO_CNTL_blit.

On blob v512.490, using WRAP_GPU_ID to fake GPU versions, I see 0x41 used
everywhere, except for BLIT_OP_SCALE on a630.  Define the magic number in
dev info so it can be reused in the two places that set the
non-BLIT_OP_SCALE value.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19794>

r600/sfn: sort FS color outputs before all other outputs

The color outputs must be checked against the number of available
color buffers, therefore it is best to sort the color outputs to be
on the driver locations before the other FS outputs.

Fixes: 79ca456b4837b3bc21cf9ef3c03c505c4b4909f6
   r600/sfn: rewrite NIR backend

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7530

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19804>

r600: Print RAT instruction names in disassembly

Also print the swizzle of the address to indicate what
values may be used.

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19804>

r600: Update scratch buffer late

For some reason the setup that comes after the scratch buffer
setup calls clobber the PS output configuration. Emitting the
scratch buffer setup as last action before the actual draw commands
seems to fix this.

Signed-off-by: Gert Wollny <gert.wollny@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19804>

freedreno/drm/virtio: Defer flush on BO free

Freeing BOs tends to be bursty (ie. when a submit is retired, or
expiring entries from BO cache).  Sending lots of small SET_IOVA
messages to the host can quickly eat up the available virtqueue
slots, resulting in (eventually) starving the guest waiting for
free virtqueue space.  By batching, we can avoid this and handle
things more efficiently on the host (ie. in a single wakeup rather
than many).

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19832>

freedreno/drm: Support for batched frees

Batch up handles before closing them to give the drm backend a chance to
batch up any extra handling needed (ie. virtio batching up messages to
host to release IOVA).

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19832>

freedreno/drm: Add optimized path for freeing many BOs

Submits tend to hold references to a lot of BOs, which get unref'd when
the submit is destroyed/retired.  For now, all this does is reduce lock
aquire/release, but the next commit will build on it.

Signed-off-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19832>

asahi: Add batch tracking logic

We already have the notion of an agx_batch, which encapsulates a render
pass. Extend the logic to allow multiple in-flight batches per context, avoiding
a flush in set_framebuffer_state and improving performance for certain
applications designed for IMRs that ping-pong unnecessarily between FBOs. I
don't have such an application immediately in mind, but I wanted to get this
flag-day out of the way while the driver is still small and flexible.

The driver was written from day 1 with batch tracking in mind, so this is a
relatively small change to actually wire it up, but there are lots of little
details to get right.

The code itself is mostly a copy/paste of panfrost, which in turn draws
inspiration from freedreno and v3d.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19865>

asahi: Use the batch for submission

So we can submit background batches.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19865>

asahi: Use batch_reads for sysvals

Required for proper resource tracking.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19865>

asahi: Use a pipe_framebuffer_state batch key

More convenient for batch tracking.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19865>

asahi: Use batch instead of ctx for pipelines

So we can support multiple batches later.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19865>

asahi: Hide ctx->batch

This will make it easier to support multiple batches.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19865>

asahi: Factor out prepare_for_map

This will be expanded, let's expand in the direction of less spaghetti.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19865>

intel/fs: put scratch surface in the surface state heap

In 4ceaed7839af we made scratch surface state allocations part of the
internal heap (mapped to STATE_BASE_ADDRESS::SurfaceStateBaseAddress)
so that it doesn't uses slots in the application's expected 1M
descriptors (especially with vkd3d-proton).

But all our compiler code relies on BSS
(STATE_BASE_ADDRESS::BindlessSurfaceStateBaseAddress).

The additional issue is that there is only 26bits of surface offset
available in CS instruction (CFE_STATE, 3DSTATE_VS, etc...) for
scratch surfaces. So we need the drivers to put the scratch surfaces
in the first chunk of STATE_BASE_ADDRESS::SurfaceStateBaseAddress
(hence all the driver changes).

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Fixes: 4ceaed7839af ("anv: split internal surface states from descriptors")
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/7687
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19727>

iris: move bindless surface state heap inside the surface state heap

We're about to make scratch surface states part of the surface state
heap. Because those are required to be in the low 26bits parts surface
state heap (we're limited in bits handed in the CFE_STATE, 3DSTATE_VS,
etc... instructions), this change splits the 32bit surface state heap
as follow:

   - 8Mb of surface states for scratch
   - 1Gb - 8Mb of binding tables
   - 3Gb of surface states

That way all of the surfaces are located within a 4Gb region visible
from STATE_BASE_ADDRESS::SurfaceStateBaseAddress

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19727>

iris: prevent crash in decoder

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19727>

radv: Convert instance bvh address to node in bvh build.

So we don't have to do it in the traversal loop. Should 2 and
instructions and a 64-bit shift, so 4/8 cycles per instance node
visit.

Totals from 7 (0.01% of 134913) affected shaders:

CodeSize: 208460 -> 208292 (-0.08%)
Instrs: 38276 -> 38248 (-0.07%)
Latency: 803181 -> 803142 (-0.00%)
InvThroughput: 165384 -> 165376 (-0.00%)
Copies: 4912 -> 4905 (-0.14%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19706>

radv: Fiddle with opaque flag positions to reduce instructions.

Totals from 7 (0.01% of 134913) affected shaders:

CodeSize: 209076 -> 208460 (-0.29%)
Instrs: 38374 -> 38276 (-0.26%)
Latency: 803899 -> 803181 (-0.09%)
InvThroughput: 165530 -> 165384 (-0.09%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19706>

radv: Skip and for node_to_addr with bvh_base.

Cause the bvh base is always 64 byte aligned.

Totals from 7 (0.01% of 134913) affected shaders:

CodeSize: 209216 -> 209076 (-0.07%)
Instrs: 38402 -> 38374 (-0.07%)
Latency: 804537 -> 803899 (-0.08%)
InvThroughput: 165663 -> 165530 (-0.08%)
Copies: 4919 -> 4912 (-0.14%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19706>

radv: Move ray flag compares out of the loop.

To save on and+cmp combos with VALU instructions.

Totals from 7 (0.01% of 134913) affected shaders:

CodeSize: 208476 -> 209216 (+0.35%)
Instrs: 38384 -> 38402 (+0.05%)
Latency: 805725 -> 804537 (-0.15%)
InvThroughput: 165906 -> 165663 (-0.15%)
Copies: 4936 -> 4919 (-0.34%)
PreSGPRs: 393 -> 430 (+9.41%)
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19706>

Revert "nir/lower_shader_calls: put inserted instructions into a dummy block"

This reverts commit 35d82ecf1ee117bef575731100fa6ae532f225fc.

Cc: mesa-stable
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19820>

nir/lower_shader_calls: wrap only jumps rather than entire code blocks

Moving entire chunks of code into a dummy if block is causing issues
in some situations. To work around the issue that we tried to fix in
35d82ecf1e ("nir/lower_shader_calls: put inserted instructions into a
dummy block") which is that we cannot cut and past a block of
instruction that ends with a jump if there are more instruction behind
where we're going to past. We can instead just wraps the jumps into
dummy if blocks.

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: mesa-stable
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19820>

nir/lower_shader_calls: update metadata before validation

Signed-off-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Cc: mesa-stable
Reviewed-by: Konstantin Seurer <konstantin.seurer@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19820>

radv/bvh: Adjust sah cost based on depth

Adds a cost field to radv_ir_node and uses it to model the cost of tree
depth. This improves framerates by 2% if my benchmarking is correct.

Reviewed-by: Adam Jackson <ajax@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19756>

nir/range_analysis: Set higher default maximum for max_workgroup_count

Fixes: c2a81ebe19f ("nir: Add default unsigned upper bound configuration.")
Closes: #7676
Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19835>

iris: Pass devinfo directly in iris_setup_uniforms

Instead of reaching through brw_compiler.  This will make easy
future changes on brw_compiler side.

Reviewed-by: José Roberto de Souza <jose.souza@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19836>

intel/compiler: Require C++17

Fixes: 6c194ddd18f ("intel/compiler: Prepare SIMD selection helpers to handle different prog_datas")

Signed-off-by: Mykhailo Skorokhodov <mykhailo.skorokhodov@globallogic.com>
Acked-by: Caio Oliveira <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19833>

asahi: Don't support 16-bit vertex attributes

Currently broken, let vbuf deal with it. "Fixes" sysprof.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Fix logic ops

Need to set colour mask correctly. Fixes spec@!opengl 1.0@gl-1.0-logicop@GL_AND,
at least the non-MSAA portion.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Restrict rendering to what we support

Noticed with Kodi that tries to use rgb10a2.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Don't validate WSI (twiddled) strides

These are made up and won't necessarily be aligned.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Split out agx_usc.h into a common file

So the tilebuffer helpers can build the "shared" USC word. Also because Ella
will probably want to use these O:)

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Note some magic bits used with memoryless RTs

Obviously there can't *actually* be memoryless render targets, because
how would partial renders work? The control stream with memoryless looks
like everything would if it went to memory (e.g. full 2D MSAA
attachments for the partial loads/stores even if only a resolved
2D image for the final store). Except the memoryless attachments all
load from the same address 0xeeee0000. Clearly that's not actually what
happens, so what gives? Unclear... but I see the magic bits mentioned
here set, and I assume there are some firmware (or kernel) shenanigans
used to JIT allocate the backing storage for partial renders.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Identify "Sample mask after depth/stencil" bit

Corresponds to Metal [[sample_mask,post_depth_coverage]].

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Identify the pass type enum

Via PowerVR.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Identify PBE sample count

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Identify Dimension for Render Target

Metal uses when rendering to multisampled 2D.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Fix agx_set_framebuffer_state for MRT

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Set data_valid for the correct level

By inspection.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Implement stencil texturing

Stencil texturing is easy: S8_UINT is textured like R8_UINT (with a
little swizzle fixup), and stencil is always S8_UINT thanks to
u_transfer_helper. So we just need to do some fixups to make
u_transfer_helper's seperate_stencil work and everything will work out.

Passes dEQP-GLES31.functional.stencil_texturing.*

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Add internal formats for RGB10A2

We need to use I16 as the interchange format here. Fixes:

   dEQP-GLES3.functional.fragment_out.basic.uint.rgb10_a2ui*

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Implement perf_debug

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

asahi: Free the scanout resource

Fixes memory leaks with renderonly.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

agx: Handle collects in backwards isel

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

agx: Assert more invariants in RA

Was helpful for debugging.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

agx: Validate part of SSA form

To debug backend pass problems.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

agx: Split off NIR preprocessing from compiling

So we can specialize after lowering I/O.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>

agx: Handle scalar texture destinations

Fixes dEQP-GLES3.functional.shaders.texture_functions.texturelod.sampler2dshadow_fragment.

Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/19811>