[mlir] Make the vast majority of integration and runner tests work on Windows
authorMarkus Böck <markus.boeck02@gmail.com>
Mon, 13 Feb 2023 16:32:08 +0000 (17:32 +0100)
committerMarkus Böck <markus.boeck02@gmail.com>
Mon, 13 Feb 2023 21:24:20 +0000 (22:24 +0100)
commit161b9d741a3c25f7bd79620598c5a2acf3f0f377
tree1c22395ad5d670e559636339ebe803cab30b6abb
parent19b1e27fcd56a2a61525899a80bddffd8af9ba61
[mlir] Make the vast majority of integration and runner tests work on Windows

This patch contains the changes required to make the vast majority of integration and runner tests run on Windows.
Historically speaking, the JIT support for Windows has been lacking behind, but recent versions of ORC JIT have now caught up and works for basically all examples in repo.

Sadly due to these tests previously not working on Windows, basically all of them are making unix-like assumptions about things like filenames, paths, shell syntax etc.
This patch fixes all these issues in one big swoop and enables Windows support for the vast majority of integration tests.

More specifically, following changes had to be done:
* The various JIT runners used paths to the runtime libraries that assumed a Unix toolchain layout and filenames. I abstracted the specific path and filename of these runtime libraries away by making the paths to the runtime libraries be passed from cmake into lit. This now also allows a much more convenient syntax: `--shared-libs=%mlir_c_runner_utils` instead of `--shared-libs=%mlir_lib_dir/lib/libmlir_c_runner_utils%shlibext`
* Some tests using python set environment variables using the `ENV=VALUE cmd` format. This works on Unix, but on Windows it has to prefixed using `env ENV=VALUE cmd`
* Some tests used C functions that are simply not available or exported on Windows (`fabsf`, `aligned_alloc`). These tests have either been adjusted or explicitly marked as `UNSUPPORTED`

Some tests remain disabled on Windows as before:
* In SparseTensor some tests have non-trivial logic for finding the runtime libraries which seems to be required for the use of emulators. I do not have the time to port these so I simply kept them disabled
* Some tests requiring special hardware which I simply cannot test remain disabled on Windows. These include usage of AVX512 or AMX

The tests for `mlir-vulkan-runner` and `mlir-spirv-runner` all work now as well and so do the vast majority of `mlir-cpu-runner`.

Differential Revision: https://reviews.llvm.org/D143925
250 files changed:
mlir/include/mlir/ExecutionEngine/RunnerUtils.h
mlir/test/CMakeLists.txt
mlir/test/Integration/Dialect/Arith/CPU/lit.local.cfg [deleted file]
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-addi-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-cmpi-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-compare-results-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-constants-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-max-min-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-muli-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-shli-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-shrsi-i16.mlir
mlir/test/Integration/Dialect/Arith/CPU/test-wide-int-emulation-shrui-i16.mlir
mlir/test/Integration/Dialect/Async/CPU/lit.local.cfg
mlir/test/Integration/Dialect/Async/CPU/microbench-linalg-async-parallel-for.mlir
mlir/test/Integration/Dialect/Async/CPU/microbench-scf-async-parallel-for.mlir
mlir/test/Integration/Dialect/Async/CPU/test-async-parallel-for-1d.mlir
mlir/test/Integration/Dialect/Async/CPU/test-async-parallel-for-2d.mlir
mlir/test/Integration/Dialect/Complex/CPU/correctness.mlir
mlir/test/Integration/Dialect/LLVMIR/CPU/X86/lit.local.cfg
mlir/test/Integration/Dialect/LLVMIR/CPU/X86/test-inline-asm-vector.mlir
mlir/test/Integration/Dialect/LLVMIR/CPU/X86/test-inline-asm.mlir
mlir/test/Integration/Dialect/LLVMIR/CPU/lit.local.cfg
mlir/test/Integration/Dialect/LLVMIR/CPU/test-vector-reductions-fp.mlir
mlir/test/Integration/Dialect/LLVMIR/CPU/test-vector-reductions-int.mlir
mlir/test/Integration/Dialect/LLVMIR/CPU/test-vp-intrinsic.mlir
mlir/test/Integration/Dialect/Linalg/CPU/matmul-vs-matvec.mlir
mlir/test/Integration/Dialect/Linalg/CPU/rank-reducing-subview.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-collapse-tensor.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-conv-1d-call.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-conv-1d-nwc-wcf-call.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-conv-2d-call.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-conv-2d-nhwc-hwcf-call.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-conv-3d-call.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-conv-3d-ndhwc-dhwcf-call.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-elementwise.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-expand-tensor.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-one-shot-bufferize.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-padtensor.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-subtensor-insert-multiple-uses.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-subtensor-insert.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-tensor-e2e.mlir
mlir/test/Integration/Dialect/Linalg/CPU/test-tensor-matmul.mlir
mlir/test/Integration/Dialect/Memref/cast-runtime-verification.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/concatenate.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/dense_output.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/dense_output_bf16.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/dense_output_f16.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/lit.local.cfg
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_abs.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_binary.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_cast.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_codegen_dim.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_codegen_foreach.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_complex32.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_complex64.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_complex_ops.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_constant_to_sparse_tensor.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conv_1d_nwc_wcf.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conv_2d.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conv_2d_nhwc_hwcf.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conv_3d.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conv_3d_ndhwc_dhwcf.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_dyn.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_ptr.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_sparse2dense.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_conversion_sparse2sparse.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_dot.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_expand.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_file_io.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_filter_conv2d.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_flatten.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_foreach_slices.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_index_dense.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_1d.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_2d.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_insert_3d.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matmul.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matrix_ops.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_matvec.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_mttkrp.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_out_mult_elt.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_out_reduction.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_out_simple.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_pack.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_quantized_matmul.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_re_im.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_reduce_custom.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_reduce_custom_prod.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_reductions.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_reductions_prod.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_reshape.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_rewrite_push_back.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_rewrite_sort.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_rewrite_sort_coo.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_matmul.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sampled_mm_fusion.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_scale.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_scf_nested.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_select.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sign.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sorted_coo.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_spmm.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_storage.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sum.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sum_bf16.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sum_c32.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_sum_f16.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_tanh.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_tensor_mul.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_tensor_ops.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_transpose.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_unary.mlir
mlir/test/Integration/Dialect/SparseTensor/CPU/sparse_vector_ops.mlir
mlir/test/Integration/Dialect/SparseTensor/python/test_SDDMM.py
mlir/test/Integration/Dialect/SparseTensor/python/test_SpMM.py
mlir/test/Integration/Dialect/SparseTensor/python/test_elementwise_add_sparse_output.py
mlir/test/Integration/Dialect/SparseTensor/python/test_output.py
mlir/test/Integration/Dialect/SparseTensor/python/test_stress.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_MTTKRP.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_SDDMM.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_SpMM.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_SpMV.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_Tensor.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_scalar_tensor_algebra.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_simple_tensor_algebra.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_tensor_complex.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_tensor_types.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_tensor_unary_ops.py
mlir/test/Integration/Dialect/SparseTensor/taco/test_true_dense_tensor_algebra.py
mlir/test/Integration/Dialect/SparseTensor/taco/unit_test_tensor_core.py
mlir/test/Integration/Dialect/SparseTensor/taco/unit_test_tensor_io.py
mlir/test/Integration/Dialect/SparseTensor/taco/unit_test_tensor_utils.py
mlir/test/Integration/Dialect/Standard/CPU/test-ceil-floor-pos-neg.mlir
mlir/test/Integration/Dialect/Standard/CPU/test_subview.mlir
mlir/test/Integration/Dialect/Vector/CPU/AMX/test-mulf-full.mlir
mlir/test/Integration/Dialect/Vector/CPU/AMX/test-mulf.mlir
mlir/test/Integration/Dialect/Vector/CPU/AMX/test-muli-ext.mlir
mlir/test/Integration/Dialect/Vector/CPU/AMX/test-muli-full.mlir
mlir/test/Integration/Dialect/Vector/CPU/AMX/test-muli.mlir
mlir/test/Integration/Dialect/Vector/CPU/AMX/test-tilezero-block.mlir
mlir/test/Integration/Dialect/Vector/CPU/AMX/test-tilezero.mlir
mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-dot.mlir
mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-inline-asm-vector-avx512.mlir
mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-mask-compress.mlir
mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-rsqrt.mlir
mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-sparse-dot-product.mlir
mlir/test/Integration/Dialect/Vector/CPU/X86Vector/test-vp2intersect-i32.mlir
mlir/test/Integration/Dialect/Vector/CPU/lit.local.cfg [deleted file]
mlir/test/Integration/Dialect/Vector/CPU/test-0-d-vectors.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-broadcast.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-compress.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-constant-mask.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-contraction.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-create-mask-v4i1.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-create-mask.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-expand.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-extract-strided-slice.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-flat-transpose-col.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-flat-transpose-row.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-fma.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-gather.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-index-vectors.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-insert-strided-slice.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-maskedload.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-maskedstore.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-matrix-multiply-col.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-matrix-multiply-row.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-outerproduct-f32.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-outerproduct-i64.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-print-int.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-realloc.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-f32-reassoc.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-f32.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-f64-reassoc.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-f64.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-i32.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-i4.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-i64.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-si4.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-reductions-ui4.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-scan.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-scatter.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-shape-cast.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-shuffle.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-sparse-dot-matvec.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-sparse-saxpy-jagged-matvec.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-transfer-read-1d.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-transfer-read-2d.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-transfer-read-3d.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-transfer-read.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-transfer-to-loops.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-transfer-write.mlir
mlir/test/Integration/Dialect/Vector/CPU/test-transpose.mlir
mlir/test/Integration/Dialect/Vector/GPU/CUDA/test-reduction-distribute.mlir
mlir/test/Integration/Dialect/Vector/GPU/CUDA/test-warp-distribute.mlir
mlir/test/Integration/GPU/CUDA/TensorCore/wmma-matmul-f16.mlir
mlir/test/Integration/GPU/CUDA/TensorCore/wmma-matmul-f32.mlir
mlir/test/Integration/GPU/CUDA/all-reduce-and.mlir
mlir/test/Integration/GPU/CUDA/all-reduce-max.mlir
mlir/test/Integration/GPU/CUDA/all-reduce-min.mlir
mlir/test/Integration/GPU/CUDA/all-reduce-op.mlir
mlir/test/Integration/GPU/CUDA/all-reduce-or.mlir
mlir/test/Integration/GPU/CUDA/all-reduce-region.mlir
mlir/test/Integration/GPU/CUDA/all-reduce-xor.mlir
mlir/test/Integration/GPU/CUDA/async.mlir
mlir/test/Integration/GPU/CUDA/gpu-to-cubin.mlir
mlir/test/Integration/GPU/CUDA/multiple-all-reduce.mlir
mlir/test/Integration/GPU/CUDA/printf.mlir
mlir/test/Integration/GPU/CUDA/shuffle.mlir
mlir/test/Integration/GPU/CUDA/two-modules.mlir
mlir/test/Integration/GPU/ROCM/gpu-to-hsaco.mlir
mlir/test/Integration/GPU/ROCM/printf.mlir
mlir/test/Integration/GPU/ROCM/two-modules.mlir
mlir/test/Integration/GPU/ROCM/vecadd.mlir
mlir/test/Integration/GPU/ROCM/vector-transferops.mlir
mlir/test/lit.cfg.py
mlir/test/lit.site.cfg.py.in
mlir/test/mlir-cpu-runner/X86Vector/math-polynomial-approx-avx2.mlir
mlir/test/mlir-cpu-runner/async-error.mlir
mlir/test/mlir-cpu-runner/async-func.mlir
mlir/test/mlir-cpu-runner/async-group.mlir
mlir/test/mlir-cpu-runner/async-value.mlir
mlir/test/mlir-cpu-runner/async.mlir
mlir/test/mlir-cpu-runner/bare-ptr-call-conv.mlir
mlir/test/mlir-cpu-runner/copy.mlir
mlir/test/mlir-cpu-runner/global-memref.mlir
mlir/test/mlir-cpu-runner/lit.local.cfg
mlir/test/mlir-cpu-runner/math-polynomial-approx.mlir
mlir/test/mlir-cpu-runner/memref-reinterpret-cast.mlir
mlir/test/mlir-cpu-runner/memref-reshape.mlir
mlir/test/mlir-cpu-runner/print.mlir
mlir/test/mlir-cpu-runner/sgemm-naive-codegen.mlir
mlir/test/mlir-cpu-runner/simple.mlir
mlir/test/mlir-cpu-runner/unranked-memref.mlir
mlir/test/mlir-cpu-runner/utils.mlir
mlir/test/mlir-cpu-runner/x86-varargs.mlir
mlir/test/mlir-spirv-cpu-runner/double.mlir
mlir/test/mlir-spirv-cpu-runner/lit.local.cfg
mlir/test/mlir-spirv-cpu-runner/mlir_test_spirv_cpu_runner_c_wrappers.cpp
mlir/test/mlir-spirv-cpu-runner/simple_add.mlir
mlir/test/mlir-vulkan-runner/addf.mlir
mlir/test/mlir-vulkan-runner/addi.mlir
mlir/test/mlir-vulkan-runner/addi8.mlir
mlir/test/mlir-vulkan-runner/mulf.mlir
mlir/test/mlir-vulkan-runner/smul_extended.mlir
mlir/test/mlir-vulkan-runner/subf.mlir
mlir/test/mlir-vulkan-runner/time.mlir
mlir/test/mlir-vulkan-runner/umul_extended.mlir