aco: fix get_buffer_resource_flags()

Looks like a rebase error. After switching to derefs, we have to look
through a nir_op_mov.

fossil-db (Navi):
Totals from 846 (0.62% of 137413) affected shaders:
SGPRs: 36856 -> 44144 (+19.77%); split: -0.20%, +19.97%
VGPRs: 35968 -> 27852 (-22.56%); split: -22.64%, +0.08%
SpillSGPRs: 1366 -> 1662 (+21.67%); split: -0.95%, +22.62%
SpillVGPRs: 1909 -> 1893 (-0.84%)
CodeSize: 5209588 -> 5146536 (-1.21%); split: -1.89%, +0.68%
Scratch: 221184 -> 217088 (-1.85%)
MaxWaves: 11488 -> 14266 (+24.18%); split: +24.20%, -0.02%
Instrs: 994831 -> 974318 (-2.06%); split: -2.53%, +0.47%
Cycles: 45719692 -> 45843260 (+0.27%); split: -0.99%, +1.26%
VMEM: 147562 -> 94468 (-35.98%); split: +9.75%, -45.74%
SMEM: 32122 -> 66023 (+105.54%); split: +120.34%, -14.80%
VClause: 41051 -> 20565 (-49.90%); split: -50.00%, +0.09%
SClause: 18076 -> 40142 (+122.07%)
Copies: 100092 -> 103521 (+3.43%); split: -0.98%, +4.40%
Branches: 51244 -> 51533 (+0.56%); split: -0.02%, +0.58%
PreSGPRs: 32290 -> 34267 (+6.12%)
PreVGPRs: 27458 -> 25290 (-7.90%); split: -7.91%, +0.01%

Signed-off-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Fixes: 05b6612b4ec ('radv: do not lower UBO/SSBO access to offsets')
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6966>

nir: Add a pass to lower vec3s to vec4s

LLVM loves take advantage of the fact that vec3s in OpenCL are 16B
aligned and so it can just read/write them as vec4s.  This results in a
LOT of vec4->vec3 casts on loads and stores.  One solution to this
problem is to get rid of all vec3 variables.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

nir/opt_deref: Add an optimization for bitcasts

LLVM loves take advantage of the fact that vec3s in OpenCL are 16B
aligned so it can just read/write them as vec4s.  This is questionably
legal except that it uses a xyz write-mask when it does it.  The result
is a LOT of vec4->vec3 casts on loads and stores.  This optimization
detects this case as well as other bit-cast cases and rewrites them to
get rid of the cast.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

nir/opt_deref: Add an instruction type switch

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

nir: Add component mask re-interpret helpers

These are based on the ones which already existed in the load/store
vectorization pass but I made some improvements while moving them.  In
particular,

 1. They're both faster if the bit sizes are equal
 2. The check is faster if old_bit_size > new_bit_size
 3. The check now fails if it would use more than NIR_MAX_VEC_COMPONENTS

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

nir/opt_load_store_vectorize: Use bit sizes when checking mask compatibility

Without this, it was checking bit size compatibility with bit sizes such
as 96 which is clearly invalid.

No shader-db changes on Ice Lake

Fixes: ce9205c03bd20d "nir: add a load/store vectorization pass"
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

nir: Add a memcpy optimization pass

This pass attempts to optimize three broad categories of memcpy:

 1. Self-copies: These we can discard out-of-hand.

 2. Vector copies: It doesn't matter what the vector size is or if the
    source and destination have different vector types, it's still easy
    enough to emit a load/store pair.

 3. Tightly packed copies:  In the case where a type is tightly packed
    (no padding bits), we can replace the memcpy with a copy_deref
    instruction which the optimizer is far better at handling.

This has proven capable of getting rid of many of the memcpy instances
in some rather gnarly OpenCL C kernels I've been looking at, even after
coming out of LLVM's optimizer.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

nir: Handle memcpy in copy_prop_vars and combine_stores

Fixes: b2899f72657 "nir: Add a new memcpy intrinsic"
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

nir/find_array_copies: Properly discard copies for casts

In 9f3c595dfc4cd, we attempted to handle casts in opt_find_array_copies
but missed a critical case.  In particular, in the case where we begin
finding a copy but then encounter a cast, we need to discard everything
which might alias that cast.

Fixes: 9f3c595dfc4cd "nir/find_array_copies: Handle cast derefs"
Reviewed-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6871>

gallium/util: allow scissored blits for stencil-fallback

It's also useful to be able to use scissor-testing for fallback-blits,
as an CTS test-case does just that.

Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6960>

gallium/util: fix memory-leak

When I originally wrote this code, I forgot to release the views the
code creates, leaking a bit of memory that never gets cleaned up. That's
not great, so let's plug it.

Fixes: e8a40715a8b ("gallium/util: add blitter-support for stencil-fallback")
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6960>

meson: fix power8 option

Do not throw a deprecation warning if the power8 option is set to the
new 'disabled' value. Instead, warn if it is still set to the legacy
value 'false'.

Fixes: 138c003d2273 ("meson: deprecated 'true' and 'false' in combo options for 'enabled' and 'disabled'")
Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
Reviewed-by: Vinson Lee <vlee@freedesktop.org>
Reviewed-by: Eric Engestrom <eric@engestrom.ch>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6370>

turnip: don't initialize GRAS_LRZ_CNTL/RB_LRZ_CNTL tu6_init_hw()

They will be initialized when emitting the draw state.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: enable LRZ

v2:

* Use sub_cs when creating the IB in tu6_build_lrz(). (Jonathan Marek)
* Emit tu6_build_lrz() only when pipeline state changes or there is a
clear. (Jonathan Marek)

v3:

* Don't modify tu_pipeline object, track the changes in command buffer
state.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: disable LRZ on vkCmdClearattachments() 3D fallback path

Partial clears are not supported and we may end up having LRZ enabled
from past commands.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: disable LRZ on vkCmdClearAttachments()

We don't support partial clears on LRZ. Blob disables them too.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: emit correct LRZ fast clear setup

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: add support to clear LRZ

v2:

* Don't emit tu6_clear_lrz() using a IB but in the command stream
provided. (Jonathan Marek)
* Valid_clear_ib is always false if TU_DEBUG_NOLRZ is set. Remove the
useless condition. (Jonathan Marek)
* Added more comments.
* Use r2d function for blitting LRZ. (Jonathan Marek)

v3:
* Do LRZ tracking in the command buffer state (Connor).

v4:

* Simplify the emission of source setup (Jonathan Marek)

v5:

* Separate LRZ setup in a different function.
* Not hide LRZ setup inside GMEM path (Jonathan Marek)
* Fix iova address emission in tu6_clear_lrz() (Jonathan Marek)
* Add CCU sysmem flushes (Jonathan Marek)

v6:

* Fixed bug related to storing a VkClearValue pointer that could be
  out-of-scope when we access to it for emitting LRZ clear.

v7:

* Merge tu6_clear_lrz() and tu6_clear_lrz_setup() into the same
function and emit LRZ clear at the beginning of the renderpass.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: add LRZ valid tracking for secondary command buffers

After a secondary command buffer is executed, LRZ is not valid
until it is cleared again.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: add LRZ tracking to command buffer state

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: disable LRZ depending on fragment changes

Disable LRZ write if the fragment shader discard the fragments, modify
its position or if early-Z is disabled.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: disable LRZ writes when blend is enabled

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: disable LRZ on specific cases

There are depth compare op modes that are not supported by LRZ in the
HW. Also, it is not supported when blend or stencil are enabled.

v2:

* Set pipeline->lrz.write to the same value than depthWriteEnable.
* Improve comment on disabling LRZ write on blend.
* Remove pipeline's lrz invalidation when there is no clear mask in
render pass. It is confusing. (Jonathan Marek)
* Mark the pipeline state as changed.
* Add comment on not using GREATER flag.

v3:

* Replace {rb,gras}_lrz_cntl by flags in struct tu_pipeline.
* Added z_test_enable flag.

v4:

* Created struct tu_lrz_pipeline to avoid modifying immutable objects.

v5:

* Fixed crashes when pDepthStencilState pointer is NULL.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: create LRZ buffer

v2:
- Add missing vulkan subpass support. (Jonathan Marek)
- When creating the BO, mark it as not valid until it is cleared.
- Move LRZ struct to tu_image. (Jonathan Marek)
- Destroy BO when we destroy the image. (Jonathan Marek)

v3:
- Allocate the buffer as part of the image's BO (Connor)
- Moved image's LRZ values to its layout.

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

turnip: add environment variable to disable LRZ

Signed-off-by: Samuel Iglesias Gonsálvez <siglesias@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/5146>

ci/bare-metal: Move the "POWER_GOOD not seen in time" check to the right time.

The poweron failure happens before we get to the bootloader
("load_archive: loading locale_en.bin") not after we're trying to boot the
kernel and we're waiting for the deqp run to complete.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6970>

nir: Fix a misspelling

Fixes: cb95065dd122a "nir: Add lowering from regular ALU conversions..."
Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6975>

glsl: don't duplicate state vars as uniforms in the NIR linker

The linker was adding all state vars as uniforms, doubling the storage size
for shaders using only builtin uniforms, which increased CPU overhead for
constant buffer uploads.

When this code was originally ported from the GLSL IR linker we forgot
to exclude builtins because the check was not done in the
add_uniform_to_shader class but rather a check was done when passing
variables to this class for processing.

Fixes: 664e4a610dc8 ("glsl/nir: Fill in the Parameters in NIR linker")

Reviewed-by: Alejandro Piñeiro <apinheiro@igalia.com>
Tested-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6958>

freedreno/cffdec: fix decoding of bindless descriptors

Add ADDR suffix so that regbase() doesn't fail and return 0.

Signed-off-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6968>

intel/fs: NoMask initialize the address register for shuffles

Cc: mesa-stable@lists.freedesktop.org
Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/2979
Tested-by: Iván Briano <ivan.briano@intel.com>
Reviewed-by: Matt Turner <mattst88@gmail.com>
Reviewed-by: Francisco Jerez <currojerez@riseup.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6825>

virgl: fix stride + layer_stride inconsistency

With blob resources, stride doesn't necesarily have to
equal width * bpp.  The use case for this a minigbm blob
resource with blob mem BLOB_MEM_HOST3D_GUEST imported into
guest Mesa.  In addition, for BLOB_MEM_HOST we can repurpose
the transfer ioctls to also flush caches if need be, so this
seems a good time to fix this issue.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4821>

virgl: query blob mem

Resource blob also modifies resource info.  Let's use this
functionality.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4821>

virgl: support PIPE_CAP_BUFFER_MAP_PERSISTENT_COHERENT

We should have GL4.5 with this.  Piglit tests should now pass.
In terms of performance, we're between 70% to 80% of host
performance on Iris, based on a apitrace of a 2013 GL4.5
game:

11.204 FPS (guest)
15.947 FPS (host)

This is still better than the status quo, when said game was unplayable
with Virgl due to an inefficient GL4.3 fallback.

TEST=piglit -t arb_buffer_storage all results/ passes

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4821>

virgl/drm: add resource create blob function

A blob resource is a container for:
  - VIRTGPU_BLOB_MEM_GUEST: a guest memory allocation
    (referred to as a "guest-only blob resource")

  - VIRTGPU_BLOB_MEM_HOST3D: a host3d memory allocation
    (referred to as a "host-only blob resource")

  - VIRTGPU_BLOB_MEM_HOST3D_GUEST: a guest + host3d memory allocation
    (referred to as a "default blob resource").

Blob resources can be used to implement new features and fix shortcomings
with the current resource create path.  The subsequent patches how
blob resources may be leveraged to implement GL_ARB_buffer_storage
and get GL4.5.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4821>

virgl/drm: query for resource blob and host visible memory region

Check for these features.

v2: refactor querying params in general (@shadeslayer)

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4821>

drm-uapi: virtgpu_drm.h: resource create blob + host visible memory region

Matches current API at virgl/resource_blob. Of course, don't
submit until this lands in drm.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4821>

virgl: add flags to (*resource_create) callback

We never seemed to use these. But for ARB_buffer_storage we'll
need it.

Reviewed-by: Tomeu Vizoso <tomeu.vizoso@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4821>

Revert F16C series (MR 6774)

This reverts commit 4fb2eddfdf9adafde2e6f94de23202ee44123d59.
This reverts commit 7a1deb16f8af4e0ae4ed64511cbfcc606087f0ee.
This reverts commit 2b6a17234376817e75d1f81edf5bd1b28eefb374.
This reverts commit 5af81393e419eaf086e4de2a1d149af78cd1f54d.
This reverts commit 87900afe5bbe90c5f3ad0921b28ae1c889029ada.

A couple of problems were discovered after this series was merged that
cause breakage in different configurations:

   (1) It seems that using -mf16c also enables AVX, leading to SIGILL on
   platforms that do not support AVX.
   (2) Since clang only warns about unknown flags, and as I understand
   it Meson's handling in cc.has_argument() is broken, the F16C code is
   wrongly enabled when clang is used, even for example on ARM, leading
   to a compilation error.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3583
Reviewed-by: Eric Anholt <eric@anholt.net>
Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6969>

android: gallium/virgl: cleanup virgl_driinfo.h gen rules

Android.mk and Makefile.sources are still defining virgl_driinfo.h target
This patch removes the remaining gen rules

Fixes the following building error:

FAILED: out/target/product/x86_64/obj/STATIC_LIBRARIES/libmesa_pipe_virgl_intermediates/virgl/virgl_driinfo.h
...
cp: bad 'out/target/product/x86_64/gen/STATIC_LIBRARIES/libmesa_pipe_virgl_intermediates/virgl/virgl_driinfo.h': No such file or directory

Fixes: 974981c4e6b9 ("gallium/drm: Make the pipe loader handle the driconf merging.")
Signed-off-by: Mauro Rossi <issor.oruam@gmail.com>
Acked-by: Eric Anholt <eric@anholt.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6880>

android: gallium/radeonsi: cleanup si_driinfo.h gen rules

Android.mk and Makefile.sources are still defining si_driinfo.h target
This patch removes the remaining gen rules

Fixes the following building error:

FAILED: out/target/product/x86_64/obj/STATIC_LIBRARIES/libmesa_pipe_radeonsi_intermediates/radeonsi/si_driinfo.h
...
cp: bad 'out/target/product/x86_64/gen/STATIC_LIBRARIES/libmesa_pipe_radeonsi_intermediates/radeonsi/si_driinfo.h': No such file or directory

Fixes: 974981c4e6b9 ("gallium/drm: Make the pipe loader handle the driconf merging.")
Signed-off-by: Mauro Rossi <issor.oruam@gmail.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6880>

android: gallium/iris: cleanup iris_driinfo.h gen rules

Android.mk and Makefile.sources are still defining iris_driinfo.h target
This patch removes the remaining gen rules

Fixes the following building error:

FAILED: out/target/product/x86_64/obj/STATIC_LIBRARIES/libmesa_pipe_iris_intermediates/iris/iris_driinfo.h
...
cp: bad 'out/target/product/x86_64/gen/STATIC_LIBRARIES/libmesa_pipe_iris_intermediates/iris/iris_driinfo.h': No such file or directory

Fixes: 974981c4e6b9 ("gallium/drm: Make the pipe loader handle the driconf merging.")
Signed-off-by: Mauro Rossi <issor.oruam@gmail.com>
Reviewed-by: Lionel Landwerlin <lionel.g.landwerlin@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6880>

nir: Add lowering from regular ALU conversions to the intrinsic

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

clover/nir: Call nir_lower_convert_alu_types

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

spirv: Implement vload[a]_half[n] and vstore[a]_half[n][_r]

Note, the aligned versions aren't handled specially yet.

The float16buffer capability is now at least partially supported after
this patch, so move it to be supported when kernels are supported.

v2 (Jason Ekstrand):
 - A few cosmetic cleanups around type/base_type
 - Rebased on top of the big SPIR-V SSA value rework
 - Use the new version of the conversion helpers

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

spirv/opencl: Drop dest_type from handle_v_load_store

At that point in the function, we don't know if it's a load or a store
so calling it dest_type isn't really helpful.  Also, we don't really
want the glsl_type; we want the base_type.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

spirv: Handle all OpenCL conversion ops with full rounding

This is done for kernels via the new convert_alu_types intrinsic.  For
Vulkan and OpenGL, we maintain the old path so that drivers don't have
to add that lowering pass.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

spirv: Add some conversion handling helpers

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

nir: Add a passes for nir_intrinsic_convert_alu_types

This adds primarily two passes:  One is a lowering pass which turns
these conversion intrinsics into a series of ALU ops.  The other is an
optimization pass which attempt to simplify the conversion whenever
possible in the hopes that we can turn it into a "normal" conversion op
which doesn't need special treatment.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

nir: Add builder helpers for OpenCL type conversions

Most of these were originally written by Daniel Stone in the Microsoft
ClOn12 branch, reworked by Jesse Natalie, fixed by Boris Brezillon, and
possibly touched by others along the way.  Unfortunately, none of that
is in the commit history thanks to living in the CLOn12 branch.

I ported them to mesa master and further reworked things for better
cosmetics.  In particular,

 1. They now live in a builder helper rather than in vtn_alu.c.

 2. Instead of looping inside each builder helper, we just trust NIR
    vector instructions to handle vectors.

 3. Lots of re-arranging of the helpers for clarity, better asserting,
    and better re-use with the upcoming lowering pass.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

nir: Add a conversion and rounding intrinsic

This new intrinsic is capable of handling the full range of conversions
from OpenCL including rounding modes and possible saturation.  The
intention is that we'll emit this intrinsic directly from spirv_to_nir
and then lower it to ALU ops later.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

nir: Split NIR_INTRINSIC_TYPE into separate src/dest indices

We're about to introduce conversion ops which are going to want two
different types.  We may as well just split the one we have rather than
end up with three.  There are a couple places where this is mildly
inconvenient but most of the time I find it to actually be nicer.

Reviewed-by: Jesse Natalie <jenatali@microsoft.com>
Reviewed-by: Daniel Stone <daniels@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6945>

gallivm/nir: fix non-32 bit find lsb/msb

fixes piglit cl get-global-id

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6954>

llvmpipe/cs: add in shader shared size.

(can remove lavapipe setting this later).

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6954>

gallivm/nir: make sure to mask global reads.

Make the driver only read values for the active lanes,
otherwise it can cause unwanted oob accesses that aren't
the apps fault.

Reviewed-by: Roland Scheidegger <sroland@vmware.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6954>

intel/gen9: Enable MSC RAW Hazard Avoidance

Workaround # 22011374674
Applied to i965, iris and anv drivers
No performance impact is observed with WA.

Cc: mesa-stable
Signed-off-by: Anuj Phogat <anuj.phogat@gmail.com>
Reviewed-by: Kenneth Graunke <kenneth@whitecape.org>

radeonsi: restructure si_pipe_set_constant_buffer

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6948>

radeonsi: call nir_lower_bool_to_int32 last because it breaks nir_opt_if

The new place is where shader variants are generated.

This is a prerequisite for inlinable uniforms.

Reviewed-by: Connor Abbott <cwabbott0@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6948>

radeonsi: use staging buffer uploads for most VRAM buffers

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6948>

radeonsi: move si_set_active_descriptors_for_shader into si_update_common_shader_state

Reviewed-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6948>

radeonsi: kill disabled clip distances and planes at per-channel granularity

Apps often enable only 1 plane for gl_ClipVertex, which means 1 scalar
clip distance.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6948>

radeonsi: simplify NGG culling enablement and add radeonsi_shader_culling option

Add a vertex count threshold into si_shader_selector to simplify
the draw_vbo code.

The new option is supposed to be used in 00-mesa-defaults.conf and should be
tweaked for best performance unlike the AMD_DEBUG experimental options.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6948>

intel/blorp: Conditionally clear full surface depth and stencil

We should set "Full Surface Depth and Stencil Clear" field of WM_HZ_OP
3DSTATE packet, only when application requires the entire depth surface
to be cleared.

Signed-off-by: Sagar Ghuge <sagar.ghuge@intel.com>
Reviewed-by: Tapani Pälli <tapani.palli@intel.com>
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6549>

anv: Skip HiZ and CCS ambiguates which preceed fast-clears

This gets rid of multiple HiZ ambiguate operations per frame in
Witcher 3.

v2:
- Fix typo (Tapani)

Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6549>

anv: Use more temp vars in cmd_buffer_begin_subpass

This is a mostly cosmetic change but there is one subtle functional
issue:  If we ever render to a 3D depth image, we are now handling the
base layer and number of layers correctly.  I'm not sure rendering to 3D
depth is even allowed but we can theoretically handle it now.

Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6549>

anv: Allow HiZ clears for multi-view

Now that we're enabling HiZ on multi-layer images, there's no reason why
we can't enable HiZ clears for multi-view.  The only reason I can think
of why we didn't before was because no one thought to and the old code
didn't.  Enabling this means that an attachment will get HiZ cleared if
and only if att_state->fast_clear.

Reviewed-by: Sagar Ghuge <sagar.ghuge@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6549>

radeonsi: support for external buffers (ext_external_objects)

So far, the callback to create a resource from a memory object had code
for importing textures only. Modified it to allow importing buffers too.

Fixes the following piglit tests:
- ext_external_objects/vk-buf-exchange
- ext_external_objects/vk-pix-buf-update-errors
- ext_external_objects/vk-vert-buf-update-errors
- ext_external_objects/vk-vert-buf-reuse

v2: Used si_alloc_buffer_struct instead of CALLOC
v3: Fixed indentation issue, removed free in case of unsuccessful
allocation, joined two if conditions together

Signed-off-by: Eleni Maria Stea <estea@igalia.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6364>

radv/aco: lower IO for all stages outside of ACO

Lowering IO for VS, TCS, TES and GS still have to be done for LLVM.

No fossils-db change on NAVI10.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6897>

radv: gather output usage mask from store_output for VS, TES and GS

IO are now lowered before the shader info pass is called and the
output usage masks have to be gathered from store_output instead.

This is currently only used by ACO.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Rhys Perry <pendingchaos02@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6897>

ttn: Fix number of components for IF/UIF

NIR if statements only take one component, but TGSI registers are vec4.
We're supposed to compare the x component, per
https://docs.mesa3d.org/gallium/tgsi.html#opcode-IF.

Fixes: f103bded ("ttn: Use nir control flow insertion helpers")
Reviewed-by: Jason Ekstrand <jason@jlekstrand.net>
Tested-by: Leo Liu <leo.liu@amd.com>
Closes: #3585
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6956>

ac/nir: fix nir_intrinsic_shared_atomic_fadd

This was completely broken.

Fixes dEQP-VK.glsl.atomic_operations.add_float32_compute_shared.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6936>

ac/llvm: fix invalid use of unreachable in ac_build_atomic_rmw()

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6936>

radv: fix gathering writes_memory for global store/atomic operations

Because global operations are lowered before the shader info pass now
we have to adjust the gathering code.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/3578
Fixes: 1588644543d ("radv: lower deref operations for global memory for both backends")
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Reviewed-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6934>

lavapipe: rename vallium to lavapipe

Just a cooler name, and a lot easier to search for. thanks Marek

Acked-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6921>

radeonsi: Fix dead lock with aux_context_lock in si_screen_clear_buffer.

After disable SDMA on Arcturus(gfx9), dead lock with aux_context_lock is
detected since si_screen_clear_buffer is called recursively before
release lock.

The call trace is:
si_clear_render_target->si_compute_clear_render_target->
si_launch_grid_internal->si_launch_grid->si_emit_cache_flush->
si_prim_discard_signal_next_compute_ib_start->u_suballocator_alloc->
si_resource_create->si_buffer_create->si_alloc_resource->
si_screen_clear_buffer->simple_mtx_lock->
si_sdma_clear_buffer->si_pipe_clear_buffer->
si_clear_buffer->si_compute_do_clear_or_copy->
si_launch_grid_internal->si_launch_grid->si_emit_cache_flush->
si_prim_discard_signal_next_compute_ib_start->u_suballocator_alloc->
si_resource_create->si_buffer_create->si_alloc_resource->
si_screen_clear_buffer->simple_mtx_lock

Fixes: 07a49bf5976 "radeonsi: disable SDMA on gfx9"
Signed-off-by: James Zhu <James.Zhu@amd.com>
Reviewed-by: Marek Olšák <marek.olsak@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6941>

add one last 20.1 release to coincide with expected 20.2.1

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6943>

docs: update calendar and link releases notes for 20.1.9

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6943>

docs: add release notes for 20.1.9

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6943>

freedreno/ir3: Enable the i/o vectorizer on UBOs.

This will merge loads of UBO components together into vec4 loads.  At the
same time, it improves the alignment information on our loads, fixing the
regression from the vec3 loads fix.

shader-db results:
total instructions in shared programs: 12829370 -> 8755851 (-31.75%)
total cat6 in shared programs: 145840 -> 97027 (-33.47%)

Overall results from before the vec3 fix:
total instructions in shared programs: 8019997 -> 8755851 (9.18%)
total cat6 in shared programs: 87683 -> 97027 (10.66%)

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

nir: Make nir_lower_ubo_vec4() handle non-vec4-aligned loads.

It turns out I had missed a case in my enumeration of why everything
currently was vec4-aligned.

Fixes a simple testcase of loading from a vec3[2] array in freedreno with
IR3_SHADER_DEBUG=nouboopt.

Initial shader-db results look devastating:

total instructions in shared programs: 8019997 -> 12829370 (59.97%)
total cat6 in shared programs: 87683 -> 145840 (66.33%)

Hopefully this will recover once we introduce the i/o vectorizer, but that
was blocked on getting the vec3 case fixed.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

nir: Drop the high_offset argument to the load_store_vectorizer filter.

Nothing uses it, and it's not clear to me what it provides over
alignment/num_components/bit_size.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

nir: Make the load_store_vectorizer provide align_mul + align_offset.

It was passing an encoding of the two that wasn't good for ensuring "Don't
combine loads that would make us straddle a vec4 boundary" for
nir_lower_ubo_vec4.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

nir/gl_nir_lower_buffers: Set up align_mul/offset on UBOs.

nir_lower_to_explicit_io will give us good alignments if we have the
cast's alignment information known, and it's trivial: Just the offset of
the UBO variable that is at the base of the deref.  Otherwise, explicit io
assumes the load is aligned just to the size of a scalar value in it.

The change in freedreno is in the noise.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

nir/nir_lower_uniforms_to_ubo: Set better alignments on our new instructions.

The change on freedreno is in the noise.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

nir: Print the alignment information on casts.

I wanted it for debugging GL alignment.

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

nir: Document a bit about how align_mul/offset work.

Introduces a #define for the maximum valid align_mul that's used in the
load_store_vectorizer tests (currently, though it will be used more soon).

Reviewed-by: Rob Clark <robdclark@chromium.org>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

freedreno/ir3: Make sure we run the opt loop after lowering UBOs to vec4.

The lowering pass may introduce vector bcsels that we need to scalarize
back out.  It's unusual to have UBOs and not get any lowered to push
constants, so the flag was usually set anyway.

Fixes: 2b2524099379 ("freedreno/ir3: Replace our custom vec4 UBO intrinsic
with the shared lowering.")

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6612>

turnip: implement legacy API functions separately

Move legacy API functions to a separate file, and implement them by calling
the new API functions, like tu_CreateRenderPass was already doing.

Signed-off-by: Jonathan Marek <jonathan@marek.ca>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6920>

Revert "nir/lower_goto_if: Add a route::outside set"

This reverts commit d57573dcd4aa2160f3153df466b01f377484a3fa.  The
actual bug was an issue with prev_frontiers which has been properly
fixed in the previous commit.

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

nir/lower_goto_ifs: Always include level dom_frontiers in prev_frontier

When we come in from some other level or from the parent, we need to
ensure that the reach set is in prev_frontier but we also need to
consider the dominance frontier of our level.  Otherwise, we may end up
leaving out possible blocks when computing the reach of a level.

Acked-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

nir/lower_goto_ifs: Add asserts for SSA forks

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

nir/lower_goto_ifs: Use rzalloc

In particular, SSA forks weren't always getting properly initialized
which was causing asserts to fail.

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

spirv: Only run repair_ssa if structured

We shouldn't need it if we're unstructured and the pass assumes
structure so attempting to run it will assert-fail.

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

nir/dominance: Use _mesa_set_clear instead ofhand-rolling it

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

nir/lower_goto_ifs: Don't destroy SSA form in the process

There are two issues here:

 1. If there are any phi nodes, we'll make complete hash of them.  This
    isn't likely actually a problem because spirv_to_nir doesn't
    generate any actual phi nodes today.  However, if we start doing any
    other passes before this, we may have a problem.

 2. Even without phi nodes, we may still break SSA form.  This can
    happen if we ever have to stick a block inside a conditional to
    satisfy weird CFG constraints.  Doing so can cause it to no longer
    look like it dominates some of its uses even though, at runtime,
    it's guaranteed to be run first.

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

nir/validate: Improve the validation of blocks

This commit adds a number of new validation checks:

 1. We now check that every block pointer in the IR points to a block
    that actually exists in a block list that's reachable from the
    nir_function_impl.

 2. We assert that nir_function_impl::body is non-empty

 3. We assert that the start block has no predecessors.  This is
    important because we tend to put run-once code there.

 4. We now validate some stuff on the end block.

Reviewed-by: Karol Herbst <kherbst@redhat.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6750>

gallium/util: remove empty file u_half.h

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6774>

gallium/util: remove redundant util_float_to_half_rtz

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6774>

util: remove util_float_to_half and util_half_to_float wrappers

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6774>

util: move util_half_to_float code into _mesa_half_to_float_slow

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6774>

util: implement f16c - fast half<->float conversions

This also happens to fix bptc-float-modes on llvmpipe.

Acked-by: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6774>