* Initial snapshort of vxsort implementation incorporating source code from Dan Shechter.
* Bug fix from Dan.
* Use bigger mark list for experiments.
* Give up if the mark list size is bigger than a reasonable fraction of the ephemeral space.
* Latest version from Dan, disable for ARM64.
* Fixes for Linux compile, ended up disabling vxsort for Linux for now.
* Experimenting with 32-bit sort - the variation that gathers mark list entries pertaining to the local heap by reading the mark lists from all the heaps appears to be too slow and scales very badly with increasing number of heaps.
* 32-bit sort - preserve failing case.
* Do the pointer compression/decompression in place, to improve performance, optionally write mark lists and associated information to binary files for further analysis.
* Introduce runtime check whether CPU supports AVX2 instruction set.
* Implement mark list growth.
* Integrate new version including AVX512 code path.
* Implement runtime test for AVX512 support.
* Move the files for the vectorized sort to their own directory, add stubs to call AVX2 or AVX512 flavor of the sort.
* Get rid of unneeded #include statement in two files.
* Address codereview feedback to specifically say AVX512F instead of just AVX512 as there are multiple subsets.
* Fix CMakeLists.tx files for non-x64 non-Windows targets, introduce separate max mark list sizes for WKS, remove dead code from grow_mark_list, add #ifdef to AVX512 detection to make the other architectures build.
* Instead of modifying the tool-generated header file corinfoinstructionset.h, modify InstructionSetDesc.txt that it is generated from, and run the tools that generates all the files from it.
* Move AVX2/AVX512 instruction set detection to GC side.
* Use vectorized packer, switch packed range from uint32_t to int32_t, because that makes the sorting a bit more efficient.
* Add GCConfig setting to turn vectorized sorting off, streamline ISA detection (but require initialization), rename to IsSupportedInstructionSet.
* Several small improvements:
- Don't waste time sorting the mark list if background GC is running as we are not going to use it.
- Use smaller max mark list size if we cannot use AVX2/AVX512 instruction sets
- Fix mark list overflow detection for server GC.
* Address codereview feedback - add constants for the thresholds above which we use AVX/AVX512F instruction sets.
Add space before parameter lists as per GC codebase coding conventions.
Improve some comments.
* Add license headers and entry in THIRD-PARTY_NOTICES.TXT for Dan's vectorized sorting code.
* Update license headers
* Address code review feedback:
- fix typo in comment in InitSupportedInstructionSet
- move test for full GC to beginning of sort_mark_list
- in WKS GC, we can use the tighter range shigh - slow for the surviving objects instead of the full ephemeral range.
- make the description for the new config setting GCEnabledInstructionSets more explicit by enumerating the legal values and their meanings.
* Snapshot for Linux changes
* Add more definitions to immintrinh.h
* Fix cmake warnings about mismatched endif clauses.
* Disable Linux support for now due to multiple compile & link errors.
* Address code review feedback:
- add instructions to bitonic_gen.py
- centralize range and instruction set checks in do_vxsort
- add parentheses around expressions.
- removed some printfs, converted others to dprintf
- strengthened assert
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+License notice for vectorized sorting code
+------------------------------------------
+
+MIT License
+
+Copyright (c) 2020 Dan Shechter
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
windows/gcenv.windows.cpp)
endif(CLR_CMAKE_HOST_UNIX)
+if (CLR_CMAKE_TARGET_ARCH_AMD64 AND CLR_CMAKE_TARGET_WIN32)
+ set ( GC_SOURCES
+ ${GC_SOURCES}
+ vxsort/isa_detection_dummy.cpp
+ vxsort/do_vxsort_avx2.cpp
+ vxsort/do_vxsort_avx512.cpp
+ vxsort/machine_traits.avx2.cpp
+ vxsort/smallsort/bitonic_sort.AVX2.int64_t.generated.cpp
+ vxsort/smallsort/bitonic_sort.AVX2.int32_t.generated.cpp
+ vxsort/smallsort/bitonic_sort.AVX512.int64_t.generated.cpp
+ vxsort/smallsort/bitonic_sort.AVX512.int32_t.generated.cpp
+)
+endif (CLR_CMAKE_TARGET_ARCH_AMD64 AND CLR_CMAKE_TARGET_WIN32)
+
if (CLR_CMAKE_TARGET_WIN32)
set(GC_HEADERS
env/common.h
handletable.inl
handletablepriv.h
objecthandle.h
- softwarewritewatch.h)
+ softwarewritewatch.h
+ vxsort/do_vxsort.h)
endif(CLR_CMAKE_TARGET_WIN32)
if(CLR_CMAKE_HOST_WIN32)
#include "gcpriv.h"
+#if defined(TARGET_AMD64) && defined(TARGET_WINDOWS)
+#define USE_VXSORT
+#else
#define USE_INTROSORT
+#endif
// We just needed a simple random number generator for testing.
class gc_rand
#ifdef USE_INTROSORT
#define _sort introsort::sort
+#elif defined(USE_VXSORT)
+// in this case we have do_vxsort which takes an additional range that
+// all items to be sorted are contained in
+// so do not #define _sort
#else //USE_INTROSORT
#define _sort qsort1
void qsort1(uint8_t** low, uint8_t** high, unsigned int depth);
#endif //PARALLEL_MARK_LIST_SORT
size_t gc_heap::mark_list_size;
+bool gc_heap::mark_list_overflow;
#endif //MARK_LIST
seg_mapping* seg_mapping_table;
}
}
-#ifdef USE_INTROSORT
+// vxsort uses introsort as a fallback if the AVX2 instruction set is not supported
+#if defined(USE_INTROSORT) || defined(USE_VXSORT)
class introsort
{
};
-#endif //USE_INTROSORT
+#endif //defined(USE_INTROSORT) || defined(USE_VXSORT)
+
+#ifdef USE_VXSORT
+static void do_vxsort (uint8_t** item_array, ptrdiff_t item_count, uint8_t* range_low, uint8_t* range_high)
+{
+ // above this threshold, using AVX2 for sorting will likely pay off
+ // despite possible downclocking on some devices
+ const size_t AVX2_THRESHOLD_SIZE = 8 * 1024;
+
+ // above this threshold, using AVX51F for sorting will likely pay off
+ // despite possible downclocking on current devices
+ const size_t AVX512F_THRESHOLD_SIZE = 128 * 1024;
+
+ if (item_count <= 1)
+ return;
+
+ if (IsSupportedInstructionSet (InstructionSet::AVX2) && (item_count > AVX2_THRESHOLD_SIZE))
+ {
+ // is the range small enough for a 32-bit sort?
+ // the 32-bit sort is almost twice as fast
+ ptrdiff_t range = range_high - range_low;
+ assert(sizeof(uint8_t*) == (1 << 3));
+ ptrdiff_t scaled_range = range >> 3;
+ if ((uint32_t)scaled_range == scaled_range)
+ {
+ dprintf (3, ("Sorting mark lists as 32-bit offsets"));
+
+ do_pack_avx2 (item_array, item_count, range_low);
+
+ int32_t* item_array_32 = (int32_t*)item_array;
+
+ // use AVX512F only if the list is large enough to pay for downclocking impact
+ if (IsSupportedInstructionSet (InstructionSet::AVX512F) && (item_count > AVX512F_THRESHOLD_SIZE))
+ {
+ do_vxsort_avx512 (item_array_32, &item_array_32[item_count - 1]);
+ }
+ else
+ {
+ do_vxsort_avx2 (item_array_32, &item_array_32[item_count - 1]);
+ }
+
+ do_unpack_avx2 (item_array_32, item_count, range_low);
+ }
+ else
+ {
+ dprintf(3, ("Sorting mark lists"));
+
+ // use AVX512F only if the list is large enough to pay for downclocking impact
+ if (IsSupportedInstructionSet (InstructionSet::AVX512F) && (item_count > AVX512F_THRESHOLD_SIZE))
+ {
+ do_vxsort_avx512 (item_array, &item_array[item_count - 1]);
+ }
+ else
+ {
+ do_vxsort_avx2 (item_array, &item_array[item_count - 1]);
+ }
+ }
+ }
+ else
+ {
+ dprintf (3, ("Sorting mark lists"));
+ introsort::sort (item_array, &item_array[item_count - 1], 0);
+ }
+#ifdef _DEBUG
+ // check the array is sorted
+ for (ptrdiff_t i = 0; i < item_count - 1; i++)
+ {
+ assert (item_array[i] <= item_array[i + 1]);
+ }
+ // check that the ends of the array are indeed in range
+ // together with the above this implies all elements are in range
+ assert ((range_low <= item_array[0]) && (item_array[item_count - 1] <= range_high));
+#endif
+}
+#endif //USE_VXSORT
#ifdef MULTIPLE_HEAPS
#ifdef PARALLEL_MARK_LIST_SORT
+NOINLINE
void gc_heap::sort_mark_list()
{
+ if (settings.condemned_generation >= max_generation)
+ {
+ return;
+ }
+
// if this heap had a mark list overflow, we don't do anything
if (mark_list_index > mark_list_end)
{
// printf("sort_mark_list: overflow on heap %d\n", heap_number);
+ mark_list_overflow = true;
+ return;
+ }
+
+#ifdef BACKGROUND_GC
+ // we are not going to use the mark list if background GC is running
+ // so let's not waste time sorting it
+ if (gc_heap::background_running_p())
+ {
+ mark_list_index = mark_list_end + 1;
return;
}
+#endif //BACKGROUND_GC
// if any other heap had a mark list overflow, we fake one too,
// so we don't use an incomplete mark list by mistake
// unsigned long start = GetCycleCount32();
+ // compute total mark list size and total ephemeral size
+ size_t total_mark_list_size = 0;
+ size_t total_ephemeral_size = 0;
+ uint8_t* low = (uint8_t*)~0;
+ uint8_t* high = 0;
+ for (int i = 0; i < n_heaps; i++)
+ {
+ gc_heap* hp = g_heaps[i];
+ size_t ephemeral_size = heap_segment_allocated (hp->ephemeral_heap_segment) - hp->gc_low;
+ total_ephemeral_size += ephemeral_size;
+ total_mark_list_size += (hp->mark_list_index - hp->mark_list);
+ low = min (low, hp->gc_low);
+ high = max (high, heap_segment_allocated (hp->ephemeral_heap_segment));
+ }
+
+ // give up if this is not an ephemeral GC or the mark list size is unreasonably large
+ if (total_mark_list_size > (total_ephemeral_size / 256))
+ {
+ mark_list_index = mark_list_end + 1;
+ // let's not count this as a mark list overflow
+ mark_list_overflow = false;
+ return;
+ }
+
+#ifdef USE_VXSORT
+ ptrdiff_t item_count = mark_list_index - mark_list;
+//#define WRITE_SORT_DATA
+#if defined(_DEBUG) || defined(WRITE_SORT_DATA)
+ // in debug, make a copy of the mark list
+ // for checking and debugging purposes
+ uint8_t** mark_list_copy = &g_mark_list_copy[heap_number * mark_list_size];
+ uint8_t** mark_list_copy_index = &mark_list_copy[item_count];
+ for (ptrdiff_t i = 0; i < item_count; i++)
+ {
+ uint8_t* item = mark_list[i];
+ mark_list_copy[i] = item;
+ }
+#endif // defined(_DEBUG) || defined(WRITE_SORT_DATA)
+
+ ptrdiff_t start = get_cycle_count();
+
+ do_vxsort (mark_list, item_count, low, high);
+
+ ptrdiff_t elapsed_cycles = get_cycle_count() - start;
+
+#ifdef WRITE_SORT_DATA
+ char file_name[256];
+ sprintf_s (file_name, _countof(file_name), "sort_data_gc%d_heap%d", settings.gc_index, heap_number);
+
+ FILE* f;
+ errno_t err = fopen_s (&f, file_name, "wb");
+
+ if (err == 0)
+ {
+ size_t magic = 'SDAT';
+ if (fwrite (&magic, sizeof(magic), 1, f) != 1)
+ dprintf (3, ("fwrite failed\n"));
+ if (fwrite (&elapsed_cycles, sizeof(elapsed_cycles), 1, f) != 1)
+ dprintf (3, ("fwrite failed\n"));
+ if (fwrite (&low, sizeof(low), 1, f) != 1)
+ dprintf (3, ("fwrite failed\n"));
+ if (fwrite (&item_count, sizeof(item_count), 1, f) != 1)
+ dprintf (3, ("fwrite failed\n"));
+ if (fwrite (mark_list_copy, sizeof(mark_list_copy[0]), item_count, f) != item_count)
+ dprintf (3, ("fwrite failed\n"));
+ if (fwrite (&magic, sizeof(magic), 1, f) != 1)
+ dprintf (3, ("fwrite failed\n"));
+ if (fclose (f) != 0)
+ dprintf (3, ("fclose failed\n"));
+ }
+#endif
+
+#ifdef _DEBUG
+ // in debug, sort the copy as well using the proven sort, so we can check we got the right result
+ if (mark_list_copy_index > mark_list_copy)
+ {
+ introsort::sort (mark_list_copy, mark_list_copy_index - 1, 0);
+ }
+ for (ptrdiff_t i = 0; i < item_count; i++)
+ {
+ uint8_t* item = mark_list[i];
+ assert (mark_list_copy[i] == item);
+ }
+#endif //_DEBUG
+
+#else //USE_VXSORT
dprintf (3, ("Sorting mark lists"));
if (mark_list_index > mark_list)
- _sort (mark_list, mark_list_index - 1, 0);
+ {
+ introsort::sort (mark_list, mark_list_index - 1, 0);
+ }
+#endif
// printf("first phase of sort_mark_list for heap %d took %u cycles to sort %u entries\n", this->heap_number, GetCycleCount32() - start, mark_list_index - mark_list);
// start = GetCycleCount32();
}
#endif // PARALLEL_MARK_LIST_SORT
#endif //MULTIPLE_HEAPS
+
+void gc_heap::grow_mark_list ()
+{
+ // with vectorized sorting, we can use bigger mark lists
+#ifdef USE_VXSORT
+#ifdef MULTIPLE_HEAPS
+ const size_t MAX_MARK_LIST_SIZE = IsSupportedInstructionSet (InstructionSet::AVX2) ? 1000 * 1024 : 200 * 1024;
+#else //MULTIPLE_HEAPS
+ const size_t MAX_MARK_LIST_SIZE = IsSupportedInstructionSet (InstructionSet::AVX2) ? 32 * 1024 : 16 * 1024;
+#endif //MULTIPLE_HEAPS
+#else
+#ifdef MULTIPLE_HEAPS
+ const size_t MAX_MARK_LIST_SIZE = 200 * 1024;
+#else //MULTIPLE_HEAPS
+ const size_t MAX_MARK_LIST_SIZE = 16 * 1024;
+#endif //MULTIPLE_HEAPS
+#endif
+
+ size_t new_mark_list_size = min (mark_list_size * 2, MAX_MARK_LIST_SIZE);
+ if (new_mark_list_size == mark_list_size)
+ return;
+
+#ifdef MULTIPLE_HEAPS
+ uint8_t** new_mark_list = make_mark_list (new_mark_list_size * n_heaps);
+
+#ifdef PARALLEL_MARK_LIST_SORT
+ uint8_t** new_mark_list_copy = make_mark_list (new_mark_list_size * n_heaps);
+#endif //PARALLEL_MARK_LIST_SORT
+
+ if (new_mark_list != nullptr
+#ifdef PARALLEL_MARK_LIST_SORT
+ && new_mark_list_copy != nullptr
+#endif //PARALLEL_MARK_LIST_SORT
+ )
+ {
+ delete[] g_mark_list;
+ g_mark_list = new_mark_list;
+#ifdef PARALLEL_MARK_LIST_SORT
+ delete[] g_mark_list_copy;
+ g_mark_list_copy = new_mark_list_copy;
+#endif //PARALLEL_MARK_LIST_SORT
+ mark_list_size = new_mark_list_size;
+ }
+ else
+ {
+ delete[] new_mark_list;
+#ifdef PARALLEL_MARK_LIST_SORT
+ delete[] new_mark_list_copy;
+#endif //PARALLEL_MARK_LIST_SORT
+ }
+
+#else //MULTIPLE_HEAPS
+ uint8_t** new_mark_list = make_mark_list (new_mark_list_size);
+ if (new_mark_list != nullptr)
+ {
+ delete[] mark_list;
+ g_mark_list = new_mark_list;
+ mark_list_size = new_mark_list_size;
+ }
+#endif //MULTIPLE_HEAPS
+}
#endif //MARK_LIST
class seg_free_spaces
static_cast<int>(GCEventStatus::GetEnabledKeywords(GCEventProvider_Private)));
#endif // __linux__
+#ifdef USE_VXSORT
+ InitSupportedInstructionSet ((int32_t)GCConfig::GetGCEnabledInstructionSets());
+#endif
+
if (!init_semi_shared())
{
hres = E_FAIL;
#ifdef MARK_LIST
#ifdef MULTIPLE_HEAPS
- mark_list_size = min (150*1024, max (8192, soh_segment_size/(2*10*32)));
+ mark_list_size = min (100*1024, max (8192, soh_segment_size/(2*10*32)));
g_mark_list = make_mark_list (mark_list_size*n_heaps);
min_balance_threshold = alloc_quantum_balance_units * CLR_SIZE * 2;
(mark_list_index - &mark_list[0]), ((mark_list_end - &mark_list[0]))));
if (mark_list_index >= (mark_list_end + 1))
+ {
mark_list_index = mark_list_end + 1;
+#ifndef MULTIPLE_HEAPS // in Server GC, we check for mark list overflow in sort_mark_list
+ mark_list_overflow = true;
+#endif
+ }
#else
dprintf (3, ("mark_list length: %Id",
(mark_list_index - &mark_list[0])));
)
{
#ifndef MULTIPLE_HEAPS
- _sort (&mark_list[0], mark_list_index-1, 0);
+#ifdef USE_VXSORT
+ do_vxsort (mark_list, mark_list_index - mark_list, slow, shigh);
+#else //USE_VXSORT
+ _sort (&mark_list[0], mark_list_index - 1, 0);
+#endif //USE_VXSORT
+
//printf ("using mark list at GC #%d", dd_collection_count (dynamic_data_of (0)));
//verify_qsort_array (&mark_list[0], mark_list_index-1);
#endif //!MULTIPLE_HEAPS
#else
record_interesting_info_per_heap();
#endif //MULTIPLE_HEAPS
+ if (mark_list_overflow)
+ {
+ grow_mark_list();
+ mark_list_overflow = false;
+ }
+
record_global_mechanisms();
#endif //GC_CONFIG_DRIVEN
}
INT_CONFIG (GCHeapHardLimitSOHPercent, "GCHeapHardLimitSOHPercent", NULL, 0, "Specifies the GC heap SOH usage as a percentage of the total memory") \
INT_CONFIG (GCHeapHardLimitLOHPercent, "GCHeapHardLimitLOHPercent", NULL, 0, "Specifies the GC heap LOH usage as a percentage of the total memory") \
INT_CONFIG (GCHeapHardLimitPOHPercent, "GCHeapHardLimitPOHPercent", NULL, 0, "Specifies the GC heap POH usage as a percentage of the total memory") \
+ INT_CONFIG (GCEnabledInstructionSets, "GCEnabledInstructionSets", NULL, -1, "Specifies whether GC can use AVX2 or AVX512F - 0 for neither, 1 for AVX2, 3 for AVX512F")\
// This class is responsible for retreiving configuration information
// for how the GC should operate.
#endif
#endif //MULTIPLE_HEAPS
+#ifdef MARK_LIST
+ PER_HEAP_ISOLATED
+ void grow_mark_list();
+#endif //MARK_LIST
+
#ifdef BACKGROUND_GC
PER_HEAP
PER_HEAP_ISOLATED
size_t mark_list_size;
+ PER_HEAP_ISOLATED
+ bool mark_list_overflow;
+
PER_HEAP
uint8_t** mark_list_end;
#define SERVER_GC 1
+#if defined(TARGET_AMD64) && defined(TARGET_WINDOWS)
+#include "vxsort/do_vxsort.h"
+#endif
+
namespace SVR {
#include "gcimpl.h"
#include "gc.cpp"
#undef SERVER_GC
#endif
+#if defined(TARGET_AMD64) && defined(TARGET_WINDOWS)
+#include "vxsort/do_vxsort.h"
+#endif
+
namespace WKS {
#include "gcimpl.h"
#include "gc.cpp"
../softwarewritewatch.cpp
)
+if (CLR_CMAKE_TARGET_ARCH_AMD64 AND CLR_CMAKE_TARGET_WIN32)
+ set ( SOURCES
+ ${SOURCES}
+ ../vxsort/isa_detection_dummy.cpp
+ ../vxsort/do_vxsort_avx2.cpp
+ ../vxsort/do_vxsort_avx512.cpp
+ ../vxsort/machine_traits.avx2.cpp
+ ../vxsort/smallsort/bitonic_sort.AVX2.int64_t.generated.cpp
+ ../vxsort/smallsort/bitonic_sort.AVX2.int32_t.generated.cpp
+ ../vxsort/smallsort/bitonic_sort.AVX512.int64_t.generated.cpp
+ ../vxsort/smallsort/bitonic_sort.AVX512.int32_t.generated.cpp
+)
+endif (CLR_CMAKE_TARGET_ARCH_AMD64 AND CLR_CMAKE_TARGET_WIN32)
+
if(CLR_CMAKE_TARGET_WIN32)
set (GC_LINK_LIBRARIES
${STATIC_MT_CRT_LIB}
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#ifndef VXSORT_ALIGNNMENT_H
+#define VXSORT_ALIGNNMENT_H
+
+//#include <cstdint>
+
+namespace vxsort {
+
+using namespace std;
+
+template <int N>
+struct alignment_hint {
+ public:
+ static const size_t ALIGN = N;
+ static const int8_t REALIGN = 0x66;
+
+ alignment_hint() : left_align(REALIGN), right_align(REALIGN) {}
+ alignment_hint realign_left() {
+ alignment_hint copy = *this;
+ copy.left_align = REALIGN;
+ return copy;
+ }
+
+ alignment_hint realign_right() {
+ alignment_hint copy = *this;
+ copy.right_align = REALIGN;
+ return copy;
+ }
+
+ static bool is_aligned(void* p) { return (size_t)p % ALIGN == 0; }
+
+ int left_align : 8;
+ int right_align : 8;
+};
+
+}
+#endif // VXSORT_ALIGNNMENT_H
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#ifndef VXSORT_DEFS_H
+#define VXSORT_DEFS_H
+
+#if _MSC_VER
+#ifdef _M_X86
+#define ARCH_X86
+#endif
+#ifdef _M_X64
+#define ARCH_X64
+#endif
+#ifdef _M_ARM64
+#define ARCH_ARM
+#endif
+#else
+#ifdef __i386__
+#define ARCH_X86
+#endif
+#ifdef __amd64__
+#define ARCH_X64
+#endif
+#ifdef __arm__
+#define ARCH_ARM
+#endif
+#endif
+
+#ifdef _MSC_VER
+#ifdef __clang__
+#define mess_up_cmov()
+#define INLINE __attribute__((always_inline))
+#define NOINLINE __attribute__((noinline))
+#else
+// MSVC
+#include <intrin.h>
+#define mess_up_cmov() _ReadBarrier();
+#define INLINE __forceinline
+#define NOINLINE __declspec(noinline)
+#endif
+#else
+// GCC + Clang
+#define mess_up_cmov()
+#define INLINE __attribute__((always_inline))
+#define NOINLINE __attribute__((noinline))
+#endif
+
+#endif // VXSORT_DEFS_H
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+// Enum for the IsSupportedInstructionSet method
+enum class InstructionSet
+{
+ AVX2 = 0,
+ AVX512F = 1,
+};
+
+void InitSupportedInstructionSet (int32_t configSetting);
+bool IsSupportedInstructionSet (InstructionSet instructionSet);
+
+void do_vxsort_avx2 (uint8_t** low, uint8_t** high);
+void do_vxsort_avx2 (int32_t* low, int32_t* high);
+
+void do_pack_avx2 (uint8_t** mem, size_t len, uint8_t* base);
+void do_unpack_avx2 (int32_t* mem, size_t len, uint8_t* base);
+
+void do_vxsort_avx512 (uint8_t** low, uint8_t** high);
+void do_vxsort_avx512 (int32_t* low, int32_t* high);
+
+void do_pack_avx512 (uint8_t** mem, size_t len, uint8_t* base);
+void do_unpack_avx512 (int32_t* mem, size_t len, uint8_t* base);
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+
+#include "vxsort_targets_enable_avx2.h"
+
+namespace std
+{
+ template <class _Ty>
+ class numeric_limits
+ {
+ public:
+ static _Ty Max()
+ {
+ return _Ty();
+ }
+ static _Ty Min()
+ {
+ return _Ty();
+ }
+ };
+ template <>
+ class numeric_limits<int32_t>
+ {
+ public:
+ static int32_t Max()
+ {
+ return 0x7fffffff;
+ }
+ static int32_t Min()
+ {
+ return -0x7fffffff-1;
+ }
+ };
+ template <>
+ class numeric_limits<int64_t>
+ {
+ public:
+ static int64_t Max()
+ {
+ return 0x7fffffffffffffffi64;
+ }
+
+ static int64_t Min()
+ {
+ return -0x7fffffffffffffffi64-1;
+ }
+ };
+}
+
+#ifndef max
+template <typename T>
+T max (T a, T b)
+{
+ if (a > b) return a; else return b;
+}
+#endif
+#include "vxsort.h"
+#include "machine_traits.avx2.h"
+#include "packer.h"
+
+void do_vxsort_avx2 (uint8_t** low, uint8_t** high)
+{
+ auto sorter = vxsort::vxsort<int64_t, vxsort::vector_machine::AVX2, 8>();
+ sorter.sort ((int64_t*)low, (int64_t*)high);
+}
+
+void do_vxsort_avx2 (int32_t* low, int32_t* high)
+{
+ auto sorter = vxsort::vxsort<int32_t, vxsort::vector_machine::AVX2, 8>();
+ sorter.sort (low, high);
+}
+
+void do_pack_avx2 (uint8_t** mem, size_t len, uint8_t* base)
+{
+ auto packer = vxsort::packer<int64_t, int32_t, vxsort::vector_machine::AVX2, 3>();
+ packer.pack ((int64_t*)mem, len, (int64_t)base);
+}
+
+void do_unpack_avx2 (int32_t* mem, size_t len, uint8_t* base)
+{
+ auto packer = vxsort::packer<int64_t, int32_t, vxsort::vector_machine::AVX2, 3>();
+ packer.unpack (mem, len, (int64_t)base);
+}
+#include "vxsort_targets_disable.h"
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+
+#include "vxsort_targets_enable_avx512.h"
+
+namespace std
+{
+ template <class _Ty>
+ class numeric_limits
+ {
+ public:
+ static _Ty Max()
+ {
+ return _Ty();
+ }
+ static _Ty Min()
+ {
+ return _Ty();
+ }
+ };
+ template <>
+ class numeric_limits<int32_t>
+ {
+ public:
+ static int32_t Max()
+ {
+ return 0x7fffffff;
+ }
+ static int32_t Min()
+ {
+ return -0x7fffffff - 1;
+ }
+ };
+ template <>
+ class numeric_limits<int64_t>
+ {
+ public:
+ static int64_t Max()
+ {
+ return 0x7fffffffffffffffi64;
+ }
+
+ static int64_t Min()
+ {
+ return -0x7fffffffffffffffi64 - 1;
+ }
+ };
+}
+
+#ifndef max
+template <typename T>
+T max (T a, T b)
+{
+ if (a > b) return a; else return b;
+}
+#endif
+
+#include "vxsort.h"
+#include "machine_traits.avx512.h"
+
+void do_vxsort_avx512 (uint8_t** low, uint8_t** high)
+{
+ auto sorter = vxsort::vxsort<int64_t, vxsort::vector_machine::AVX512, 8>();
+ sorter.sort ((int64_t*)low, (int64_t*)high);
+}
+
+void do_vxsort_avx512 (int32_t* low, int32_t* high)
+{
+ auto sorter = vxsort::vxsort<int32_t, vxsort::vector_machine::AVX512, 8>();
+ sorter.sort (low, high);
+}
+
+#include "vxsort_targets_disable.h"
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+#include <intrin.h>
+
+#include "do_vxsort.h"
+
+enum class SupportedISA
+{
+ None = 0,
+ AVX2 = 1 << (int)InstructionSet::AVX2,
+ AVX512F = 1 << (int)InstructionSet::AVX512F
+};
+
+#if defined(TARGET_AMD64) && defined(TARGET_WINDOWS)
+
+static DWORD64 GetEnabledXStateFeaturesHelper()
+{
+ LIMITED_METHOD_CONTRACT;
+
+ // On Windows we have an api(GetEnabledXStateFeatures) to check if AVX is supported
+ typedef DWORD64(WINAPI* PGETENABLEDXSTATEFEATURES)();
+ PGETENABLEDXSTATEFEATURES pfnGetEnabledXStateFeatures = NULL;
+
+ HMODULE hMod = WszLoadLibraryEx(WINDOWS_KERNEL32_DLLNAME_W, NULL, LOAD_LIBRARY_SEARCH_SYSTEM32);
+ if (hMod == NULL)
+ return 0;
+
+ pfnGetEnabledXStateFeatures = (PGETENABLEDXSTATEFEATURES)GetProcAddress(hMod, "GetEnabledXStateFeatures");
+
+ if (pfnGetEnabledXStateFeatures == NULL)
+ {
+ return 0;
+ }
+
+ DWORD64 FeatureMask = pfnGetEnabledXStateFeatures();
+
+ return FeatureMask;
+}
+
+SupportedISA DetermineSupportedISA()
+{
+ // register definitions to make the following code more readable
+ enum reg
+ {
+ EAX = 0,
+ EBX = 1,
+ ECX = 2,
+ EDX = 3,
+ COUNT = 4
+ };
+
+ // bit definitions to make code more readable
+ enum bits
+ {
+ OCXSAVE = 1<<27,
+ AVX = 1<<28,
+ AVX2 = 1<<5,
+ AVX512F=1<<16,
+ };
+ int reg[COUNT];
+
+ __cpuid(reg, 0);
+ if (reg[EAX] < 7)
+ return SupportedISA::None;
+
+ __cpuid(reg, 1);
+
+ // both AVX and OCXSAVE feature flags must be enabled
+ if ((reg[ECX] & (OCXSAVE|AVX)) != (OCXSAVE | AVX))
+ return SupportedISA::None;
+
+ // get xcr0 register
+ DWORD64 xcr0 = _xgetbv(0);
+
+ // get OS XState info
+ DWORD64 FeatureMask = GetEnabledXStateFeaturesHelper();
+
+ // get processor extended feature flag info
+ __cpuid(reg, 7);
+
+ // check if both AVX2 and AVX512F are supported by both processor and OS
+ if ((reg[EBX] & (AVX2 | AVX512F)) == (AVX2 | AVX512F) &&
+ (xcr0 & 0xe6) == 0xe6 &&
+ (FeatureMask & (XSTATE_MASK_AVX | XSTATE_MASK_AVX512)) == (XSTATE_MASK_AVX | XSTATE_MASK_AVX512))
+ {
+ return (SupportedISA)((int)SupportedISA::AVX2 | (int)SupportedISA::AVX512F);
+ }
+
+ // check if AVX2 is supported by both processor and OS
+ if ((reg[EBX] & AVX2) &&
+ (xcr0 & 0x06) == 0x06 &&
+ (FeatureMask & XSTATE_MASK_AVX) == XSTATE_MASK_AVX)
+ {
+ return SupportedISA::AVX2;
+ }
+
+ return SupportedISA::None;
+}
+
+#elif defined(TARGET_UNIX)
+
+SupportedISA DetermineSupportedISA()
+{
+ __builtin_cpu_init();
+ if (__builtin_cpu_supports("avx2"))
+ {
+ if (__builtin_cpu_supports("avx512f"))
+ return (SupportedISA)((int)SupportedISA::AVX2 | (int)SupportedISA::AVX512F);
+ else
+ return SupportedISA::AVX2;
+ }
+ else
+ {
+ return SupportedISA::None;
+ }
+}
+
+#endif // defined(TARGET_UNIX)
+
+static bool s_initialized;
+static SupportedISA s_supportedISA;
+
+bool IsSupportedInstructionSet (InstructionSet instructionSet)
+{
+ assert(s_initialized);
+ assert(instructionSet == InstructionSet::AVX2 || instructionSet == InstructionSet::AVX512F);
+ return ((int)s_supportedISA & (1 << (int)instructionSet)) != 0;
+}
+
+void InitSupportedInstructionSet (int32_t configSetting)
+{
+ s_supportedISA = (SupportedISA)((int)DetermineSupportedISA() & configSetting);
+ // we are assuming that AVX2 can be used if AVX512F can,
+ // so if AVX2 is disabled, we need to disable AVX512F as well
+ if (!((int)s_supportedISA & (int)SupportedISA::AVX2))
+ s_supportedISA = SupportedISA::None;
+ s_initialized = true;
+}
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+
+#include "do_vxsort.h"
+
+#if defined(TARGET_AMD64) && defined(TARGET_WINDOWS)
+
+void InitSupportedInstructionSet (int32_t)
+{
+}
+
+bool IsSupportedInstructionSet (InstructionSet)
+{
+ return false;
+}
+#endif // defined(TARGET_AMD64) && defined(TARGET_WINDOWS)
+
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+//#include <cstdint>
+
+#include "machine_traits.avx2.h"
+
+namespace vxsort {
+
+alignas(128) const int8_t perm_table_64[128] = {
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b0000 (0)
+ 2, 3, 4, 5, 6, 7, 0, 1, // 0b0001 (1)
+ 0, 1, 4, 5, 6, 7, 2, 3, // 0b0010 (2)
+ 4, 5, 6, 7, 0, 1, 2, 3, // 0b0011 (3)
+ 0, 1, 2, 3, 6, 7, 4, 5, // 0b0100 (4)
+ 2, 3, 6, 7, 0, 1, 4, 5, // 0b0101 (5)
+ 0, 1, 6, 7, 2, 3, 4, 5, // 0b0110 (6)
+ 6, 7, 0, 1, 2, 3, 4, 5, // 0b0111 (7)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b1000 (8)
+ 2, 3, 4, 5, 0, 1, 6, 7, // 0b1001 (9)
+ 0, 1, 4, 5, 2, 3, 6, 7, // 0b1010 (10)
+ 4, 5, 0, 1, 2, 3, 6, 7, // 0b1011 (11)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b1100 (12)
+ 2, 3, 0, 1, 4, 5, 6, 7, // 0b1101 (13)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b1110 (14)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b1111 (15)
+};
+
+alignas(2048) const int8_t perm_table_32[2048] = {
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b00000000 (0)
+ 1, 2, 3, 4, 5, 6, 7, 0, // 0b00000001 (1)
+ 0, 2, 3, 4, 5, 6, 7, 1, // 0b00000010 (2)
+ 2, 3, 4, 5, 6, 7, 0, 1, // 0b00000011 (3)
+ 0, 1, 3, 4, 5, 6, 7, 2, // 0b00000100 (4)
+ 1, 3, 4, 5, 6, 7, 0, 2, // 0b00000101 (5)
+ 0, 3, 4, 5, 6, 7, 1, 2, // 0b00000110 (6)
+ 3, 4, 5, 6, 7, 0, 1, 2, // 0b00000111 (7)
+ 0, 1, 2, 4, 5, 6, 7, 3, // 0b00001000 (8)
+ 1, 2, 4, 5, 6, 7, 0, 3, // 0b00001001 (9)
+ 0, 2, 4, 5, 6, 7, 1, 3, // 0b00001010 (10)
+ 2, 4, 5, 6, 7, 0, 1, 3, // 0b00001011 (11)
+ 0, 1, 4, 5, 6, 7, 2, 3, // 0b00001100 (12)
+ 1, 4, 5, 6, 7, 0, 2, 3, // 0b00001101 (13)
+ 0, 4, 5, 6, 7, 1, 2, 3, // 0b00001110 (14)
+ 4, 5, 6, 7, 0, 1, 2, 3, // 0b00001111 (15)
+ 0, 1, 2, 3, 5, 6, 7, 4, // 0b00010000 (16)
+ 1, 2, 3, 5, 6, 7, 0, 4, // 0b00010001 (17)
+ 0, 2, 3, 5, 6, 7, 1, 4, // 0b00010010 (18)
+ 2, 3, 5, 6, 7, 0, 1, 4, // 0b00010011 (19)
+ 0, 1, 3, 5, 6, 7, 2, 4, // 0b00010100 (20)
+ 1, 3, 5, 6, 7, 0, 2, 4, // 0b00010101 (21)
+ 0, 3, 5, 6, 7, 1, 2, 4, // 0b00010110 (22)
+ 3, 5, 6, 7, 0, 1, 2, 4, // 0b00010111 (23)
+ 0, 1, 2, 5, 6, 7, 3, 4, // 0b00011000 (24)
+ 1, 2, 5, 6, 7, 0, 3, 4, // 0b00011001 (25)
+ 0, 2, 5, 6, 7, 1, 3, 4, // 0b00011010 (26)
+ 2, 5, 6, 7, 0, 1, 3, 4, // 0b00011011 (27)
+ 0, 1, 5, 6, 7, 2, 3, 4, // 0b00011100 (28)
+ 1, 5, 6, 7, 0, 2, 3, 4, // 0b00011101 (29)
+ 0, 5, 6, 7, 1, 2, 3, 4, // 0b00011110 (30)
+ 5, 6, 7, 0, 1, 2, 3, 4, // 0b00011111 (31)
+ 0, 1, 2, 3, 4, 6, 7, 5, // 0b00100000 (32)
+ 1, 2, 3, 4, 6, 7, 0, 5, // 0b00100001 (33)
+ 0, 2, 3, 4, 6, 7, 1, 5, // 0b00100010 (34)
+ 2, 3, 4, 6, 7, 0, 1, 5, // 0b00100011 (35)
+ 0, 1, 3, 4, 6, 7, 2, 5, // 0b00100100 (36)
+ 1, 3, 4, 6, 7, 0, 2, 5, // 0b00100101 (37)
+ 0, 3, 4, 6, 7, 1, 2, 5, // 0b00100110 (38)
+ 3, 4, 6, 7, 0, 1, 2, 5, // 0b00100111 (39)
+ 0, 1, 2, 4, 6, 7, 3, 5, // 0b00101000 (40)
+ 1, 2, 4, 6, 7, 0, 3, 5, // 0b00101001 (41)
+ 0, 2, 4, 6, 7, 1, 3, 5, // 0b00101010 (42)
+ 2, 4, 6, 7, 0, 1, 3, 5, // 0b00101011 (43)
+ 0, 1, 4, 6, 7, 2, 3, 5, // 0b00101100 (44)
+ 1, 4, 6, 7, 0, 2, 3, 5, // 0b00101101 (45)
+ 0, 4, 6, 7, 1, 2, 3, 5, // 0b00101110 (46)
+ 4, 6, 7, 0, 1, 2, 3, 5, // 0b00101111 (47)
+ 0, 1, 2, 3, 6, 7, 4, 5, // 0b00110000 (48)
+ 1, 2, 3, 6, 7, 0, 4, 5, // 0b00110001 (49)
+ 0, 2, 3, 6, 7, 1, 4, 5, // 0b00110010 (50)
+ 2, 3, 6, 7, 0, 1, 4, 5, // 0b00110011 (51)
+ 0, 1, 3, 6, 7, 2, 4, 5, // 0b00110100 (52)
+ 1, 3, 6, 7, 0, 2, 4, 5, // 0b00110101 (53)
+ 0, 3, 6, 7, 1, 2, 4, 5, // 0b00110110 (54)
+ 3, 6, 7, 0, 1, 2, 4, 5, // 0b00110111 (55)
+ 0, 1, 2, 6, 7, 3, 4, 5, // 0b00111000 (56)
+ 1, 2, 6, 7, 0, 3, 4, 5, // 0b00111001 (57)
+ 0, 2, 6, 7, 1, 3, 4, 5, // 0b00111010 (58)
+ 2, 6, 7, 0, 1, 3, 4, 5, // 0b00111011 (59)
+ 0, 1, 6, 7, 2, 3, 4, 5, // 0b00111100 (60)
+ 1, 6, 7, 0, 2, 3, 4, 5, // 0b00111101 (61)
+ 0, 6, 7, 1, 2, 3, 4, 5, // 0b00111110 (62)
+ 6, 7, 0, 1, 2, 3, 4, 5, // 0b00111111 (63)
+ 0, 1, 2, 3, 4, 5, 7, 6, // 0b01000000 (64)
+ 1, 2, 3, 4, 5, 7, 0, 6, // 0b01000001 (65)
+ 0, 2, 3, 4, 5, 7, 1, 6, // 0b01000010 (66)
+ 2, 3, 4, 5, 7, 0, 1, 6, // 0b01000011 (67)
+ 0, 1, 3, 4, 5, 7, 2, 6, // 0b01000100 (68)
+ 1, 3, 4, 5, 7, 0, 2, 6, // 0b01000101 (69)
+ 0, 3, 4, 5, 7, 1, 2, 6, // 0b01000110 (70)
+ 3, 4, 5, 7, 0, 1, 2, 6, // 0b01000111 (71)
+ 0, 1, 2, 4, 5, 7, 3, 6, // 0b01001000 (72)
+ 1, 2, 4, 5, 7, 0, 3, 6, // 0b01001001 (73)
+ 0, 2, 4, 5, 7, 1, 3, 6, // 0b01001010 (74)
+ 2, 4, 5, 7, 0, 1, 3, 6, // 0b01001011 (75)
+ 0, 1, 4, 5, 7, 2, 3, 6, // 0b01001100 (76)
+ 1, 4, 5, 7, 0, 2, 3, 6, // 0b01001101 (77)
+ 0, 4, 5, 7, 1, 2, 3, 6, // 0b01001110 (78)
+ 4, 5, 7, 0, 1, 2, 3, 6, // 0b01001111 (79)
+ 0, 1, 2, 3, 5, 7, 4, 6, // 0b01010000 (80)
+ 1, 2, 3, 5, 7, 0, 4, 6, // 0b01010001 (81)
+ 0, 2, 3, 5, 7, 1, 4, 6, // 0b01010010 (82)
+ 2, 3, 5, 7, 0, 1, 4, 6, // 0b01010011 (83)
+ 0, 1, 3, 5, 7, 2, 4, 6, // 0b01010100 (84)
+ 1, 3, 5, 7, 0, 2, 4, 6, // 0b01010101 (85)
+ 0, 3, 5, 7, 1, 2, 4, 6, // 0b01010110 (86)
+ 3, 5, 7, 0, 1, 2, 4, 6, // 0b01010111 (87)
+ 0, 1, 2, 5, 7, 3, 4, 6, // 0b01011000 (88)
+ 1, 2, 5, 7, 0, 3, 4, 6, // 0b01011001 (89)
+ 0, 2, 5, 7, 1, 3, 4, 6, // 0b01011010 (90)
+ 2, 5, 7, 0, 1, 3, 4, 6, // 0b01011011 (91)
+ 0, 1, 5, 7, 2, 3, 4, 6, // 0b01011100 (92)
+ 1, 5, 7, 0, 2, 3, 4, 6, // 0b01011101 (93)
+ 0, 5, 7, 1, 2, 3, 4, 6, // 0b01011110 (94)
+ 5, 7, 0, 1, 2, 3, 4, 6, // 0b01011111 (95)
+ 0, 1, 2, 3, 4, 7, 5, 6, // 0b01100000 (96)
+ 1, 2, 3, 4, 7, 0, 5, 6, // 0b01100001 (97)
+ 0, 2, 3, 4, 7, 1, 5, 6, // 0b01100010 (98)
+ 2, 3, 4, 7, 0, 1, 5, 6, // 0b01100011 (99)
+ 0, 1, 3, 4, 7, 2, 5, 6, // 0b01100100 (100)
+ 1, 3, 4, 7, 0, 2, 5, 6, // 0b01100101 (101)
+ 0, 3, 4, 7, 1, 2, 5, 6, // 0b01100110 (102)
+ 3, 4, 7, 0, 1, 2, 5, 6, // 0b01100111 (103)
+ 0, 1, 2, 4, 7, 3, 5, 6, // 0b01101000 (104)
+ 1, 2, 4, 7, 0, 3, 5, 6, // 0b01101001 (105)
+ 0, 2, 4, 7, 1, 3, 5, 6, // 0b01101010 (106)
+ 2, 4, 7, 0, 1, 3, 5, 6, // 0b01101011 (107)
+ 0, 1, 4, 7, 2, 3, 5, 6, // 0b01101100 (108)
+ 1, 4, 7, 0, 2, 3, 5, 6, // 0b01101101 (109)
+ 0, 4, 7, 1, 2, 3, 5, 6, // 0b01101110 (110)
+ 4, 7, 0, 1, 2, 3, 5, 6, // 0b01101111 (111)
+ 0, 1, 2, 3, 7, 4, 5, 6, // 0b01110000 (112)
+ 1, 2, 3, 7, 0, 4, 5, 6, // 0b01110001 (113)
+ 0, 2, 3, 7, 1, 4, 5, 6, // 0b01110010 (114)
+ 2, 3, 7, 0, 1, 4, 5, 6, // 0b01110011 (115)
+ 0, 1, 3, 7, 2, 4, 5, 6, // 0b01110100 (116)
+ 1, 3, 7, 0, 2, 4, 5, 6, // 0b01110101 (117)
+ 0, 3, 7, 1, 2, 4, 5, 6, // 0b01110110 (118)
+ 3, 7, 0, 1, 2, 4, 5, 6, // 0b01110111 (119)
+ 0, 1, 2, 7, 3, 4, 5, 6, // 0b01111000 (120)
+ 1, 2, 7, 0, 3, 4, 5, 6, // 0b01111001 (121)
+ 0, 2, 7, 1, 3, 4, 5, 6, // 0b01111010 (122)
+ 2, 7, 0, 1, 3, 4, 5, 6, // 0b01111011 (123)
+ 0, 1, 7, 2, 3, 4, 5, 6, // 0b01111100 (124)
+ 1, 7, 0, 2, 3, 4, 5, 6, // 0b01111101 (125)
+ 0, 7, 1, 2, 3, 4, 5, 6, // 0b01111110 (126)
+ 7, 0, 1, 2, 3, 4, 5, 6, // 0b01111111 (127)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b10000000 (128)
+ 1, 2, 3, 4, 5, 6, 0, 7, // 0b10000001 (129)
+ 0, 2, 3, 4, 5, 6, 1, 7, // 0b10000010 (130)
+ 2, 3, 4, 5, 6, 0, 1, 7, // 0b10000011 (131)
+ 0, 1, 3, 4, 5, 6, 2, 7, // 0b10000100 (132)
+ 1, 3, 4, 5, 6, 0, 2, 7, // 0b10000101 (133)
+ 0, 3, 4, 5, 6, 1, 2, 7, // 0b10000110 (134)
+ 3, 4, 5, 6, 0, 1, 2, 7, // 0b10000111 (135)
+ 0, 1, 2, 4, 5, 6, 3, 7, // 0b10001000 (136)
+ 1, 2, 4, 5, 6, 0, 3, 7, // 0b10001001 (137)
+ 0, 2, 4, 5, 6, 1, 3, 7, // 0b10001010 (138)
+ 2, 4, 5, 6, 0, 1, 3, 7, // 0b10001011 (139)
+ 0, 1, 4, 5, 6, 2, 3, 7, // 0b10001100 (140)
+ 1, 4, 5, 6, 0, 2, 3, 7, // 0b10001101 (141)
+ 0, 4, 5, 6, 1, 2, 3, 7, // 0b10001110 (142)
+ 4, 5, 6, 0, 1, 2, 3, 7, // 0b10001111 (143)
+ 0, 1, 2, 3, 5, 6, 4, 7, // 0b10010000 (144)
+ 1, 2, 3, 5, 6, 0, 4, 7, // 0b10010001 (145)
+ 0, 2, 3, 5, 6, 1, 4, 7, // 0b10010010 (146)
+ 2, 3, 5, 6, 0, 1, 4, 7, // 0b10010011 (147)
+ 0, 1, 3, 5, 6, 2, 4, 7, // 0b10010100 (148)
+ 1, 3, 5, 6, 0, 2, 4, 7, // 0b10010101 (149)
+ 0, 3, 5, 6, 1, 2, 4, 7, // 0b10010110 (150)
+ 3, 5, 6, 0, 1, 2, 4, 7, // 0b10010111 (151)
+ 0, 1, 2, 5, 6, 3, 4, 7, // 0b10011000 (152)
+ 1, 2, 5, 6, 0, 3, 4, 7, // 0b10011001 (153)
+ 0, 2, 5, 6, 1, 3, 4, 7, // 0b10011010 (154)
+ 2, 5, 6, 0, 1, 3, 4, 7, // 0b10011011 (155)
+ 0, 1, 5, 6, 2, 3, 4, 7, // 0b10011100 (156)
+ 1, 5, 6, 0, 2, 3, 4, 7, // 0b10011101 (157)
+ 0, 5, 6, 1, 2, 3, 4, 7, // 0b10011110 (158)
+ 5, 6, 0, 1, 2, 3, 4, 7, // 0b10011111 (159)
+ 0, 1, 2, 3, 4, 6, 5, 7, // 0b10100000 (160)
+ 1, 2, 3, 4, 6, 0, 5, 7, // 0b10100001 (161)
+ 0, 2, 3, 4, 6, 1, 5, 7, // 0b10100010 (162)
+ 2, 3, 4, 6, 0, 1, 5, 7, // 0b10100011 (163)
+ 0, 1, 3, 4, 6, 2, 5, 7, // 0b10100100 (164)
+ 1, 3, 4, 6, 0, 2, 5, 7, // 0b10100101 (165)
+ 0, 3, 4, 6, 1, 2, 5, 7, // 0b10100110 (166)
+ 3, 4, 6, 0, 1, 2, 5, 7, // 0b10100111 (167)
+ 0, 1, 2, 4, 6, 3, 5, 7, // 0b10101000 (168)
+ 1, 2, 4, 6, 0, 3, 5, 7, // 0b10101001 (169)
+ 0, 2, 4, 6, 1, 3, 5, 7, // 0b10101010 (170)
+ 2, 4, 6, 0, 1, 3, 5, 7, // 0b10101011 (171)
+ 0, 1, 4, 6, 2, 3, 5, 7, // 0b10101100 (172)
+ 1, 4, 6, 0, 2, 3, 5, 7, // 0b10101101 (173)
+ 0, 4, 6, 1, 2, 3, 5, 7, // 0b10101110 (174)
+ 4, 6, 0, 1, 2, 3, 5, 7, // 0b10101111 (175)
+ 0, 1, 2, 3, 6, 4, 5, 7, // 0b10110000 (176)
+ 1, 2, 3, 6, 0, 4, 5, 7, // 0b10110001 (177)
+ 0, 2, 3, 6, 1, 4, 5, 7, // 0b10110010 (178)
+ 2, 3, 6, 0, 1, 4, 5, 7, // 0b10110011 (179)
+ 0, 1, 3, 6, 2, 4, 5, 7, // 0b10110100 (180)
+ 1, 3, 6, 0, 2, 4, 5, 7, // 0b10110101 (181)
+ 0, 3, 6, 1, 2, 4, 5, 7, // 0b10110110 (182)
+ 3, 6, 0, 1, 2, 4, 5, 7, // 0b10110111 (183)
+ 0, 1, 2, 6, 3, 4, 5, 7, // 0b10111000 (184)
+ 1, 2, 6, 0, 3, 4, 5, 7, // 0b10111001 (185)
+ 0, 2, 6, 1, 3, 4, 5, 7, // 0b10111010 (186)
+ 2, 6, 0, 1, 3, 4, 5, 7, // 0b10111011 (187)
+ 0, 1, 6, 2, 3, 4, 5, 7, // 0b10111100 (188)
+ 1, 6, 0, 2, 3, 4, 5, 7, // 0b10111101 (189)
+ 0, 6, 1, 2, 3, 4, 5, 7, // 0b10111110 (190)
+ 6, 0, 1, 2, 3, 4, 5, 7, // 0b10111111 (191)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b11000000 (192)
+ 1, 2, 3, 4, 5, 0, 6, 7, // 0b11000001 (193)
+ 0, 2, 3, 4, 5, 1, 6, 7, // 0b11000010 (194)
+ 2, 3, 4, 5, 0, 1, 6, 7, // 0b11000011 (195)
+ 0, 1, 3, 4, 5, 2, 6, 7, // 0b11000100 (196)
+ 1, 3, 4, 5, 0, 2, 6, 7, // 0b11000101 (197)
+ 0, 3, 4, 5, 1, 2, 6, 7, // 0b11000110 (198)
+ 3, 4, 5, 0, 1, 2, 6, 7, // 0b11000111 (199)
+ 0, 1, 2, 4, 5, 3, 6, 7, // 0b11001000 (200)
+ 1, 2, 4, 5, 0, 3, 6, 7, // 0b11001001 (201)
+ 0, 2, 4, 5, 1, 3, 6, 7, // 0b11001010 (202)
+ 2, 4, 5, 0, 1, 3, 6, 7, // 0b11001011 (203)
+ 0, 1, 4, 5, 2, 3, 6, 7, // 0b11001100 (204)
+ 1, 4, 5, 0, 2, 3, 6, 7, // 0b11001101 (205)
+ 0, 4, 5, 1, 2, 3, 6, 7, // 0b11001110 (206)
+ 4, 5, 0, 1, 2, 3, 6, 7, // 0b11001111 (207)
+ 0, 1, 2, 3, 5, 4, 6, 7, // 0b11010000 (208)
+ 1, 2, 3, 5, 0, 4, 6, 7, // 0b11010001 (209)
+ 0, 2, 3, 5, 1, 4, 6, 7, // 0b11010010 (210)
+ 2, 3, 5, 0, 1, 4, 6, 7, // 0b11010011 (211)
+ 0, 1, 3, 5, 2, 4, 6, 7, // 0b11010100 (212)
+ 1, 3, 5, 0, 2, 4, 6, 7, // 0b11010101 (213)
+ 0, 3, 5, 1, 2, 4, 6, 7, // 0b11010110 (214)
+ 3, 5, 0, 1, 2, 4, 6, 7, // 0b11010111 (215)
+ 0, 1, 2, 5, 3, 4, 6, 7, // 0b11011000 (216)
+ 1, 2, 5, 0, 3, 4, 6, 7, // 0b11011001 (217)
+ 0, 2, 5, 1, 3, 4, 6, 7, // 0b11011010 (218)
+ 2, 5, 0, 1, 3, 4, 6, 7, // 0b11011011 (219)
+ 0, 1, 5, 2, 3, 4, 6, 7, // 0b11011100 (220)
+ 1, 5, 0, 2, 3, 4, 6, 7, // 0b11011101 (221)
+ 0, 5, 1, 2, 3, 4, 6, 7, // 0b11011110 (222)
+ 5, 0, 1, 2, 3, 4, 6, 7, // 0b11011111 (223)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b11100000 (224)
+ 1, 2, 3, 4, 0, 5, 6, 7, // 0b11100001 (225)
+ 0, 2, 3, 4, 1, 5, 6, 7, // 0b11100010 (226)
+ 2, 3, 4, 0, 1, 5, 6, 7, // 0b11100011 (227)
+ 0, 1, 3, 4, 2, 5, 6, 7, // 0b11100100 (228)
+ 1, 3, 4, 0, 2, 5, 6, 7, // 0b11100101 (229)
+ 0, 3, 4, 1, 2, 5, 6, 7, // 0b11100110 (230)
+ 3, 4, 0, 1, 2, 5, 6, 7, // 0b11100111 (231)
+ 0, 1, 2, 4, 3, 5, 6, 7, // 0b11101000 (232)
+ 1, 2, 4, 0, 3, 5, 6, 7, // 0b11101001 (233)
+ 0, 2, 4, 1, 3, 5, 6, 7, // 0b11101010 (234)
+ 2, 4, 0, 1, 3, 5, 6, 7, // 0b11101011 (235)
+ 0, 1, 4, 2, 3, 5, 6, 7, // 0b11101100 (236)
+ 1, 4, 0, 2, 3, 5, 6, 7, // 0b11101101 (237)
+ 0, 4, 1, 2, 3, 5, 6, 7, // 0b11101110 (238)
+ 4, 0, 1, 2, 3, 5, 6, 7, // 0b11101111 (239)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b11110000 (240)
+ 1, 2, 3, 0, 4, 5, 6, 7, // 0b11110001 (241)
+ 0, 2, 3, 1, 4, 5, 6, 7, // 0b11110010 (242)
+ 2, 3, 0, 1, 4, 5, 6, 7, // 0b11110011 (243)
+ 0, 1, 3, 2, 4, 5, 6, 7, // 0b11110100 (244)
+ 1, 3, 0, 2, 4, 5, 6, 7, // 0b11110101 (245)
+ 0, 3, 1, 2, 4, 5, 6, 7, // 0b11110110 (246)
+ 3, 0, 1, 2, 4, 5, 6, 7, // 0b11110111 (247)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b11111000 (248)
+ 1, 2, 0, 3, 4, 5, 6, 7, // 0b11111001 (249)
+ 0, 2, 1, 3, 4, 5, 6, 7, // 0b11111010 (250)
+ 2, 0, 1, 3, 4, 5, 6, 7, // 0b11111011 (251)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b11111100 (252)
+ 1, 0, 2, 3, 4, 5, 6, 7, // 0b11111101 (253)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b11111110 (254)
+ 0, 1, 2, 3, 4, 5, 6, 7, // 0b11111111 (255)
+};
+
+}
+
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+//
+// Created by dans on 6/1/20.
+//
+
+#ifndef VXSORT_MACHINE_TRAITS_AVX2_H
+#define VXSORT_MACHINE_TRAITS_AVX2_H
+
+#include "vxsort_targets_enable_avx2.h"
+
+#include <immintrin.h>
+//#include <stdexcept>
+#include <assert.h>
+
+#include "defs.h"
+#include "machine_traits.h"
+
+#define i2d _mm256_castsi256_pd
+#define d2i _mm256_castpd_si256
+#define i2s _mm256_castsi256_ps
+#define s2i _mm256_castps_si256
+#define s2d _mm256_castps_pd
+#define d2s _mm256_castpd_ps
+
+namespace vxsort {
+extern const int8_t perm_table_64[128];
+extern const int8_t perm_table_32[2048];
+
+static void not_supported()
+{
+ assert(!"operation is unsupported");
+}
+
+#ifdef _DEBUG
+// in _DEBUG, we #define return to be something more complicated,
+// containing a statement, so #define away constexpr for _DEBUG
+#define constexpr
+#endif //_DEBUG
+
+template <>
+class vxsort_machine_traits<int32_t, AVX2> {
+ public:
+ typedef __m256i TV;
+ typedef uint32_t TMASK;
+
+ static constexpr bool supports_compress_writes() { return false; }
+
+ static INLINE TV load_vec(TV* p) { return _mm256_lddqu_si256(p); }
+
+ static INLINE void store_vec(TV* ptr, TV v) { _mm256_storeu_si256(ptr, v); }
+
+ static void store_compress_vec(TV* ptr, TV v, TMASK mask) { not_supported(); }
+
+ static INLINE TV partition_vector(TV v, int mask) {
+ assert(mask >= 0);
+ assert(mask <= 255);
+ return s2i(_mm256_permutevar8x32_ps(i2s(v), _mm256_cvtepu8_epi32(_mm_loadu_si128((__m128i*)(perm_table_32 + mask * 8)))));
+ }
+
+ static INLINE TV broadcast(int32_t pivot) { return _mm256_set1_epi32(pivot); }
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) { return _mm256_movemask_ps(i2s(_mm256_cmpgt_epi32(a, b))); }
+
+ static TV shift_right(TV v, int i) { return _mm256_srli_epi32(v, i); }
+ static TV shift_left(TV v, int i) { return _mm256_slli_epi32(v, i); }
+
+ static INLINE TV add(TV a, TV b) { return _mm256_add_epi32(a, b); }
+ static INLINE TV sub(TV a, TV b) { return _mm256_sub_epi32(a, b); };
+};
+
+template <>
+class vxsort_machine_traits<uint32_t, AVX2> {
+ public:
+ typedef __m256i TV;
+ typedef uint32_t TMASK;
+
+ static constexpr bool supports_compress_writes() { return false; }
+
+ static INLINE TV load_vec(TV* p) { return _mm256_lddqu_si256(p); }
+
+ static INLINE void store_vec(TV* ptr, TV v) { _mm256_storeu_si256(ptr, v); }
+
+ static void store_compress_vec(TV* ptr, TV v, TMASK mask) { not_supported(); }
+
+ static INLINE TV partition_vector(TV v, int mask) {
+ assert(mask >= 0);
+ assert(mask <= 255);
+ return s2i(_mm256_permutevar8x32_ps(i2s(v), _mm256_cvtepu8_epi32(_mm_loadu_si128((__m128i*)(perm_table_32 + mask * 8)))));
+ }
+
+ static INLINE TV broadcast(uint32_t pivot) { return _mm256_set1_epi32(pivot); }
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ __m256i top_bit = _mm256_set1_epi32(1U << 31);
+ return _mm256_movemask_ps(i2s(_mm256_cmpgt_epi32(_mm256_xor_si256(top_bit, a), _mm256_xor_si256(top_bit, b))));
+ }
+
+ static TV shift_right(TV v, int i) { return _mm256_srli_epi32(v, i); }
+ static TV shift_left(TV v, int i) { return _mm256_slli_epi32(v, i); }
+
+ static INLINE TV add(TV a, TV b) { return _mm256_add_epi32(a, b); }
+ static INLINE TV sub(TV a, TV b) { return _mm256_sub_epi32(a, b); };
+};
+
+template <>
+class vxsort_machine_traits<float, AVX2> {
+ public:
+ typedef __m256 TV;
+ typedef uint32_t TMASK;
+
+ static constexpr bool supports_compress_writes() { return false; }
+
+ static INLINE TV load_vec(TV* p) { return _mm256_loadu_ps((float*)p); }
+
+ static INLINE void store_vec(TV* ptr, TV v) { _mm256_storeu_ps((float*)ptr, v); }
+
+ static void store_compress_vec(TV* ptr, TV v, TMASK mask) { not_supported(); }
+
+ static INLINE TV partition_vector(TV v, int mask) {
+ assert(mask >= 0);
+ assert(mask <= 255);
+ return _mm256_permutevar8x32_ps(v, _mm256_cvtepu8_epi32(_mm_loadu_si128((__m128i*)(perm_table_32 + mask * 8))));
+ }
+
+ static INLINE TV broadcast(float pivot) { return _mm256_set1_ps(pivot); }
+
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ /// 0x0E: Greater-than (ordered, signaling) \n
+ /// 0x1E: Greater-than (ordered, non-signaling)
+ return _mm256_movemask_ps(_mm256_cmp_ps(a, b, _CMP_GT_OS));
+ }
+
+ static INLINE TV add(TV a, TV b) { return _mm256_add_ps(a, b); }
+ static INLINE TV sub(TV a, TV b) { return _mm256_sub_ps(a, b); };
+
+};
+
+template <>
+class vxsort_machine_traits<int64_t, AVX2> {
+ public:
+ typedef __m256i TV;
+ typedef uint32_t TMASK;
+
+ static constexpr bool supports_compress_writes() { return false; }
+
+ static INLINE TV load_vec(TV* p) { return _mm256_lddqu_si256(p); }
+
+ static INLINE void store_vec(TV* ptr, TV v) { _mm256_storeu_si256(ptr, v); }
+
+ static void store_compress_vec(TV* ptr, TV v, TMASK mask) { not_supported(); }
+
+ static INLINE TV partition_vector(TV v, int mask) {
+ assert(mask >= 0);
+ assert(mask <= 15);
+ return s2i(_mm256_permutevar8x32_ps(i2s(v), _mm256_cvtepu8_epi32(_mm_loadu_si128((__m128i*)(perm_table_64 + mask * 8)))));
+ }
+
+ static INLINE TV broadcast(int64_t pivot) { return _mm256_set1_epi64x(pivot); }
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) { return _mm256_movemask_pd(i2d(_mm256_cmpgt_epi64(a, b))); }
+
+ static TV shift_right(TV v, int i) { return _mm256_srli_epi64(v, i); }
+ static TV shift_left(TV v, int i) { return _mm256_slli_epi64(v, i); }
+
+ static INLINE TV add(TV a, TV b) { return _mm256_add_epi64(a, b); }
+ static INLINE TV sub(TV a, TV b) { return _mm256_sub_epi64(a, b); };
+
+
+
+ static INLINE TV pack_ordered(TV a, TV b) {
+ a = _mm256_permute4x64_epi64(_mm256_shuffle_epi32(a, _MM_PERM_DBCA), _MM_PERM_DBCA);
+ b = _mm256_permute4x64_epi64(_mm256_shuffle_epi32(b, _MM_PERM_DBCA), _MM_PERM_CADB);
+ return _mm256_blend_epi32(a, b, 0b11110000);
+ }
+
+ static INLINE TV pack_unordered(TV a, TV b) {
+ b = _mm256_shuffle_epi32(b, _MM_PERM_CDAB);
+ return _mm256_blend_epi32(a, b, 0b10101010);
+ }
+
+ static INLINE void unpack_ordered_signed(TV p, TV& u1, TV& u2) {
+ auto p01 = _mm256_extracti128_si256(p, 0);
+ auto p02 = _mm256_extracti128_si256(p, 1);
+
+ u1 = _mm256_cvtepi32_epi64(p01);
+ u2 = _mm256_cvtepi32_epi64(p02);
+
+ }
+
+ static INLINE void unpack_ordered_unsigned(TV p, TV& u1, TV& u2) {
+ auto p01 = _mm256_extracti128_si256(p, 0);
+ auto p02 = _mm256_extracti128_si256(p, 1);
+
+ u1 = _mm256_cvtepu32_epi64(p01);
+ u2 = _mm256_cvtepu32_epi64(p02);
+
+ }
+
+/*
+ template<>
+ static INLINE TV pack_ordered<int32_t>(TV a, TV b) {
+ a = _mm256_permute4x64_epi64(_mm256_shuffle_epi32(a, _MM_PERM_DBCA), _MM_PERM_DBCA);
+ b = _mm256_permute4x64_epi64(_mm256_shuffle_epi32(b, _MM_PERM_DBCA), _MM_PERM_CADB);
+ return _mm256_blend_epi32(a, b, 0b11110000);
+ }
+
+ template<>
+ static INLINE typename vxsort_machine_traits<int32_t, AVX2>::TV pack_unordered<int32_t>(TV a, TV b) {
+ b = _mm256_shuffle_epi32(b, _MM_PERM_CDAB);
+ return _mm256_blend_epi32(a, b, 0b10101010);
+ }
+
+ */
+
+
+
+};
+
+template <>
+class vxsort_machine_traits<uint64_t, AVX2> {
+ public:
+ typedef __m256i TV;
+ typedef uint32_t TMASK;
+
+ static constexpr bool supports_compress_writes() { return false; }
+
+ static INLINE TV load_vec(TV* p) { return _mm256_lddqu_si256(p); }
+
+ static INLINE void store_vec(TV* ptr, TV v) { _mm256_storeu_si256(ptr, v); }
+
+ static void store_compress_vec(TV* ptr, TV v, TMASK mask) { not_supported(); }
+
+ static INLINE TV partition_vector(TV v, int mask) {
+ assert(mask >= 0);
+ assert(mask <= 15);
+ return s2i(_mm256_permutevar8x32_ps(i2s(v), _mm256_cvtepu8_epi32(_mm_loadu_si128((__m128i*)(perm_table_64 + mask * 8)))));
+ }
+ static INLINE TV broadcast(int64_t pivot) { return _mm256_set1_epi64x(pivot); }
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ __m256i top_bit = _mm256_set1_epi64x(1LLU << 63);
+ return _mm256_movemask_pd(i2d(_mm256_cmpgt_epi64(_mm256_xor_si256(top_bit, a), _mm256_xor_si256(top_bit, b))));
+ }
+
+ static INLINE TV shift_right(TV v, int i) { return _mm256_srli_epi64(v, i); }
+ static INLINE TV shift_left(TV v, int i) { return _mm256_slli_epi64(v, i); }
+
+ static INLINE TV add(TV a, TV b) { return _mm256_add_epi64(a, b); }
+ static INLINE TV sub(TV a, TV b) { return _mm256_sub_epi64(a, b); };
+};
+
+template <>
+class vxsort_machine_traits<double, AVX2> {
+ public:
+ typedef __m256d TV;
+ typedef uint32_t TMASK;
+
+ static constexpr bool supports_compress_writes() { return false; }
+
+ static INLINE TV load_vec(TV* p) { return _mm256_loadu_pd((double*)p); }
+
+ static INLINE void store_vec(TV* ptr, TV v) { _mm256_storeu_pd((double*)ptr, v); }
+
+ static void store_compress_vec(TV* ptr, TV v, TMASK mask) { not_supported(); }
+
+ static INLINE TV partition_vector(TV v, int mask) {
+ assert(mask >= 0);
+ assert(mask <= 15);
+ return s2d(_mm256_permutevar8x32_ps(d2s(v), _mm256_cvtepu8_epi32(_mm_loadu_si128((__m128i*)(perm_table_64 + mask * 8)))));
+ }
+
+ static INLINE TV broadcast(double pivot) { return _mm256_set1_pd(pivot); }
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ /// 0x0E: Greater-than (ordered, signaling) \n
+ /// 0x1E: Greater-than (ordered, non-signaling)
+ return _mm256_movemask_pd(_mm256_cmp_pd(a, b, _CMP_GT_OS));
+ }
+
+ static INLINE TV add(TV a, TV b) { return _mm256_add_pd(a, b); }
+ static INLINE TV sub(TV a, TV b) { return _mm256_sub_pd(a, b); };
+};
+
+}
+
+#undef i2d
+#undef d2i
+#undef i2s
+#undef s2i
+#undef s2d
+#undef d2s
+
+#ifdef _DEBUG
+#undef constexpr
+#endif //_DEBUG
+
+#include "vxsort_targets_disable.h"
+
+
+#endif // VXSORT_VXSORT_AVX2_H
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+//
+// Created by dans on 6/1/20.
+//
+
+#ifndef VXSORT_MACHINE_TRAITS_AVX512_H
+#define VXSORT_MACHINE_TRAITS_AVX512_H
+
+#include "vxsort_targets_enable_avx512.h"
+
+#include <immintrin.h>
+#include "defs.h"
+#include "machine_traits.h"
+
+#ifdef _DEBUG
+// in _DEBUG, we #define return to be something more complicated,
+// containing a statement, so #define away constexpr for _DEBUG
+#define constexpr
+#endif //_DEBUG
+
+namespace vxsort {
+template <>
+class vxsort_machine_traits<int32_t, AVX512> {
+ public:
+ typedef __m512i TV;
+ typedef __mmask16 TMASK;
+
+ static constexpr bool supports_compress_writes() { return true; }
+
+ static INLINE TV load_vec(TV* p) {
+ return _mm512_loadu_si512(p);
+ }
+
+ static INLINE void store_vec(TV* ptr, TV v) {
+ _mm512_storeu_si512(ptr, v);
+ }
+
+ // Will never be called
+ static INLINE TV partition_vector(TV v, int mask) { return v; }
+
+
+ static void store_compress_vec(TV *ptr, TV v, TMASK mask) {
+ _mm512_mask_compressstoreu_epi32(ptr, mask, v);
+ }
+
+ static INLINE TV broadcast(int32_t pivot) {
+ return _mm512_set1_epi32(pivot);
+ }
+
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ return _mm512_cmp_epi32_mask(a, b, _MM_CMPINT_GT);
+ }
+};
+
+template <>
+class vxsort_machine_traits<uint32_t, AVX512> {
+ public:
+ typedef __m512i TV;
+ typedef __mmask16 TMASK;
+
+ static constexpr bool supports_compress_writes() { return true; }
+
+ static INLINE TV load_vec(TV* p) {
+ return _mm512_loadu_si512(p);
+ }
+
+ static INLINE void store_vec(TV* ptr, TV v) {
+ _mm512_storeu_si512(ptr, v);
+ }
+
+ // Will never be called
+ static INLINE TV partition_vector(TV v, int mask) { return v; }
+
+
+ static void store_compress_vec(TV *ptr, TV v, TMASK mask) {
+ _mm512_mask_compressstoreu_epi32(ptr, mask, v);
+ }
+
+ static INLINE TV broadcast(uint32_t pivot) {
+ return _mm512_set1_epi32(pivot);
+ }
+
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ return _mm512_cmp_epu32_mask(a, b, _MM_CMPINT_GT);
+ }
+};
+
+template <>
+class vxsort_machine_traits<float, AVX512> {
+ public:
+ typedef __m512 TV;
+ typedef __mmask16 TMASK;
+
+ static constexpr bool supports_compress_writes() { return true; }
+
+ static INLINE TV load_vec(TV* p) {
+ return _mm512_loadu_ps(p);
+ }
+
+ static INLINE void store_vec(TV* ptr, TV v) {
+ _mm512_storeu_ps(ptr, v);
+ }
+
+ // Will never be called
+ static INLINE TV partition_vector(TV v, int mask) { return v; }
+
+
+ static void store_compress_vec(TV *ptr, TV v, TMASK mask) {
+ _mm512_mask_compressstoreu_ps(ptr, mask, v);
+ }
+
+ static INLINE TV broadcast(float pivot) {
+ return _mm512_set1_ps(pivot);
+ }
+
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ return _mm512_cmp_ps_mask(a, b, _CMP_GT_OS);
+ }
+};
+
+template <>
+class vxsort_machine_traits<int64_t, AVX512> {
+ public:
+ typedef __m512i TV;
+ typedef __mmask8 TMASK;
+
+ static bool supports_compress_writes() { return true; }
+
+ static INLINE TV load_vec(TV* p) {
+ return _mm512_loadu_si512(p);
+ }
+
+ static INLINE void store_vec(TV* ptr, TV v) {
+ _mm512_storeu_si512(ptr, v);
+ }
+
+ // Will never be called
+ static INLINE TV partition_vector(TV v, int mask) { return v; }
+
+
+ static void store_compress_vec(TV *ptr, TV v, TMASK mask) {
+ _mm512_mask_compressstoreu_epi64(ptr, mask, v);
+ }
+
+ static INLINE TV broadcast(int64_t pivot) {
+ return _mm512_set1_epi64(pivot);
+ }
+
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ return _mm512_cmp_epi64_mask(a, b, _MM_CMPINT_GT);
+ }
+};
+
+template <>
+class vxsort_machine_traits<uint64_t, AVX512> {
+ public:
+ typedef __m512i TV;
+ typedef __mmask8 TMASK;
+
+ static constexpr bool supports_compress_writes() { return true; }
+
+ static INLINE TV load_vec(TV* p) {
+ return _mm512_loadu_si512(p);
+ }
+
+ static INLINE void store_vec(TV* ptr, TV v) {
+ _mm512_storeu_si512(ptr, v);
+ }
+
+ // Will never be called
+ static INLINE TV partition_vector(TV v, int mask) { return v; }
+
+
+ static void store_compress_vec(TV *ptr, TV v, TMASK mask) {
+ _mm512_mask_compressstoreu_epi64(ptr, mask, v);
+ }
+
+ static INLINE TV broadcast(uint64_t pivot) {
+ return _mm512_set1_epi64(pivot);
+ }
+
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ return _mm512_cmp_epu64_mask(a, b, _MM_CMPINT_GT);
+ }
+};
+
+template <>
+class vxsort_machine_traits<double, AVX512> {
+ public:
+ typedef __m512d TV;
+ typedef __mmask8 TMASK;
+
+ static constexpr bool supports_compress_writes() { return true; }
+
+ static INLINE TV load_vec(TV* p) {
+ return _mm512_loadu_pd(p);
+ }
+
+ static INLINE void store_vec(TV* ptr, TV v) {
+ _mm512_storeu_pd(ptr, v);
+ }
+
+ // Will never be called
+ static INLINE TV partition_vector(TV v, int mask) { return v; }
+
+
+ static void store_compress_vec(TV *ptr, TV v, TMASK mask) {
+ _mm512_mask_compressstoreu_pd(ptr, mask, v);
+ }
+
+ static INLINE TV broadcast(double pivot) {
+ return _mm512_set1_pd(pivot);
+ }
+
+ static INLINE TMASK get_cmpgt_mask(TV a, TV b) {
+ return _mm512_cmp_pd_mask(a, b, _CMP_GT_OS);
+ }
+};
+
+}
+
+#ifdef _DEBUG
+#undef constexpr
+#endif //_DEBUG
+
+#include "vxsort_targets_disable.h"
+
+#endif // VXSORT_VXSORT_AVX512_H
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+//
+// Created by dans on 6/1/20.
+//
+
+#ifndef VXSORT_MACHINE_TRAITS_H
+#define VXSORT_MACHINE_TRAITS_H
+
+//#include <cstdint>
+
+namespace vxsort {
+
+enum vector_machine {
+ NONE,
+ AVX2,
+ AVX512,
+ SVE,
+};
+
+template <typename T, vector_machine M>
+struct vxsort_machine_traits {
+ public:
+ typedef int TV;
+ typedef int TMASK;
+
+ static constexpr bool supports_compress_writes();
+
+ static TV load_vec(TV* ptr);
+ static void store_vec(TV* ptr, TV v);
+ static void store_compress_vec(TV* ptr, TV v, TMASK mask);
+ static TV partition_vector(TV v, int mask);
+ static TV broadcast(T pivot);
+ static TMASK get_cmpgt_mask(TV a, TV b);
+
+ static TV shift_right(TV v, int i);
+ static TV shift_left(TV v, int i);
+
+ static TV add(TV a, TV b);
+ static TV sub(TV a, TV b);
+
+ static TV pack_ordered(TV a, TV b);
+ static TV pack_unordered(TV a, TV b);
+
+ static void unpack_ordered_signed(TV p, TV& u1, TV& u2);
+ static void unpack_ordered_unsigned(TV p, TV& u1, TV& u2);
+
+
+};
+}
+
+#endif // VXSORT_MACHINE_TRAITS_H
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#ifndef VXSORT_PACKER_H
+#define VXSORT_PACKER_H
+
+#include "vxsort_targets_enable_avx2.h"
+
+//#include <cstdint>
+//#include <limits>
+//#include <type_traits>
+#//include <cassert>
+#include "alignment.h"
+#include "machine_traits.h"
+#include "machine_traits.avx2.h"
+#include "machine_traits.avx512.h"
+
+#include <immintrin.h>
+//#include <cstdio>
+
+namespace vxsort {
+
+template <typename TFrom, typename TTo, vector_machine M, int Shift = 0, int MinLength=1, bool RespectPackingOrder=false>
+class packer {
+ static_assert(Shift <= 31, "Shift must be in the range 0..31");
+ using MT = vxsort_machine_traits<TFrom, M>;
+ typedef typename MT::TV TV;
+ typedef typename std::make_unsigned<TFrom>::type TU;
+ static const int N = sizeof(TV) / sizeof(TFrom);
+ typedef alignment_hint<sizeof(TV)> AH;
+
+ static const size_t ALIGN = AH::ALIGN;
+ static const size_t ALIGN_MASK = ALIGN - 1;
+
+ static INLINE void pack_scalar(const TFrom offset, TFrom*& mem_read, TTo*& mem_write) {
+ auto d = *(mem_read++);
+ if (Shift > 0)
+ d >>= Shift;
+ d -= offset;
+ *(mem_write++) = (TTo) d;
+ }
+
+ static INLINE void unpack_scalar(const TFrom offset, TTo*& mem_read, TFrom*& mem_write) {
+ TFrom d = *(--mem_read);
+
+ d += offset;
+
+ if (Shift > 0)
+ d = (TFrom) (((TU) d) << Shift);
+
+ *(--mem_write) = d;
+ }
+
+ public:
+
+ static void pack(TFrom *mem, size_t len, TFrom base) {
+ TFrom offset = (base >> Shift) - std::numeric_limits<TTo>::Min();
+ auto baseVec = MT::broadcast(offset);
+
+ auto pre_aligned_mem = reinterpret_cast<TFrom*>(reinterpret_cast<size_t>(mem) & ~ALIGN_MASK);
+
+ auto mem_read = mem;
+ auto mem_write = (TTo *) mem;
+
+ // Include a "special" pass to handle very short scalar
+ // passes
+ if (MinLength < N && len < N) {
+ while (len--) {
+ pack_scalar(offset, mem_read, mem_write);
+ }
+ return;
+ }
+
+ // We have at least
+ // one vector worth of data to handle
+ // Let's try to align to vector size first
+
+ if (pre_aligned_mem < mem) {
+ const auto alignment_point = pre_aligned_mem + N;
+ len -= (alignment_point - mem_read);
+ while (mem_read < alignment_point) {
+ pack_scalar(offset, mem_read, mem_write);
+ }
+ }
+
+ assert(AH::is_aligned(mem_read));
+
+ auto memv_read = (TV *) mem_read;
+ auto memv_write = (TV *) mem_write;
+
+ auto lenv = len / N;
+ len -= (lenv * N);
+
+ while (lenv >= 2) {
+ assert(memv_read >= memv_write);
+
+ auto d01 = MT::load_vec(memv_read);
+ auto d02 = MT::load_vec(memv_read + 1);
+ if (Shift > 0) { // This is statically compiled in/out
+ d01 = MT::shift_right(d01, Shift);
+ d02 = MT::shift_right(d02, Shift);
+ }
+ d01 = MT::sub(d01, baseVec);
+ d02 = MT::sub(d02, baseVec);
+
+ auto packed_data = RespectPackingOrder ?
+ MT::pack_ordered(d01, d02) :
+ MT::pack_unordered(d01, d02);
+
+ MT::store_vec(memv_write, packed_data);
+
+ memv_read += 2;
+ memv_write++;
+ lenv -= 2;
+ }
+
+ len += lenv * N;
+
+ mem_read = (TFrom *) memv_read;
+ mem_write = (TTo *) memv_write;
+
+ while (len-- > 0) {
+ pack_scalar(offset, mem_read, mem_write);
+ }
+ }
+
+ static void unpack(TTo *mem, size_t len, TFrom base) {
+ TFrom offset = (base >> Shift) - std::numeric_limits<TTo>::Min();
+ auto baseVec = MT::broadcast(offset);
+
+ auto mem_read = mem + len;
+ auto mem_write = ((TFrom *) mem) + len;
+
+
+ // Include a "special" pass to handle very short scalar
+ // passers
+ if (MinLength < 2*N && len < 2*N) {
+ while (len--) {
+ unpack_scalar(offset, mem_read, mem_write);
+ }
+ return;
+ }
+
+ auto pre_aligned_mem = reinterpret_cast<TTo*>(reinterpret_cast<size_t>(mem_read) & ~ALIGN_MASK);
+
+ if (pre_aligned_mem < mem_read) {
+ len -= (mem_read - pre_aligned_mem);
+ while (mem_read > pre_aligned_mem) {
+ unpack_scalar(offset, mem_read, mem_write);
+ }
+ }
+
+ assert(AH::is_aligned(mem_read));
+
+ auto lenv = len / (N*2);
+ auto memv_read = ((TV *) mem_read) - 1;
+ auto memv_write = ((TV *) mem_write) - 2;
+ len -= lenv * N * 2;
+
+ while (lenv > 0) {
+ assert(memv_read <= memv_write);
+ TV d01, d02;
+
+ if (std::numeric_limits<TTo>::Min() < 0)
+ MT::unpack_ordered_signed(MT::load_vec(memv_read), d01, d02);
+ else
+ MT::unpack_ordered_unsigned(MT::load_vec(memv_read), d01, d02);
+
+ d01 = MT::add(d01, baseVec);
+ d02 = MT::add(d02, baseVec);
+
+ if (Shift > 0) { // This is statically compiled in/out
+ d01 = MT::shift_left(d01, Shift);
+ d02 = MT::shift_left(d02, Shift);
+ }
+
+ MT::store_vec(memv_write, d01);
+ MT::store_vec(memv_write + 1, d02);
+
+ memv_read -= 1;
+ memv_write -= 2;
+ lenv--;
+ }
+
+ mem_read = (TTo *) (memv_read + 1);
+ mem_write = (TFrom *) (memv_write + 2);
+
+ while (len-- > 0) {
+ unpack_scalar(offset, mem_read, mem_write);
+ }
+ }
+
+};
+
+}
+
+#include "vxsort_targets_disable.h"
+
+#endif // VXSORT_PACKER_H
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+#include "bitonic_sort.AVX2.int32_t.generated.h"
+
+using namespace vxsort;
+
+void vxsort::smallsort::bitonic<int32_t, vector_machine::AVX2 >::sort(int32_t *ptr, size_t length) {
+ const int N = 8;
+
+ switch(length / N) {
+ case 1: sort_01v(ptr); break;
+ case 2: sort_02v(ptr); break;
+ case 3: sort_03v(ptr); break;
+ case 4: sort_04v(ptr); break;
+ case 5: sort_05v(ptr); break;
+ case 6: sort_06v(ptr); break;
+ case 7: sort_07v(ptr); break;
+ case 8: sort_08v(ptr); break;
+ case 9: sort_09v(ptr); break;
+ case 10: sort_10v(ptr); break;
+ case 11: sort_11v(ptr); break;
+ case 12: sort_12v(ptr); break;
+ case 13: sort_13v(ptr); break;
+ case 14: sort_14v(ptr); break;
+ case 15: sort_15v(ptr); break;
+ case 16: sort_16v(ptr); break;
+ }
+}
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+/////////////////////////////////////////////////////////////////////////////
+////
+// This file was auto-generated by a tool at 2020-06-22 05:27:48
+//
+// It is recommended you DO NOT directly edit this file but instead edit
+// the code-generator that generated this source file instead.
+/////////////////////////////////////////////////////////////////////////////
+
+#ifndef BITONIC_SORT_AVX2_INT32_T_H
+#define BITONIC_SORT_AVX2_INT32_T_H
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx2"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx2")
+#endif
+#endif
+
+#include <immintrin.h>
+#include "bitonic_sort.h"
+
+#define i2d _mm256_castsi256_pd
+#define d2i _mm256_castpd_si256
+#define i2s _mm256_castsi256_ps
+#define s2i _mm256_castps_si256
+#define s2d _mm256_castps_pd
+#define d2s _mm256_castpd_ps
+
+namespace vxsort {
+namespace smallsort {
+template<> struct bitonic<int32_t, AVX2> {
+public:
+
+ static INLINE void sort_01v_ascending(__m256i& d01) {
+ __m256i min, max, s;
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xAA);
+
+ s = _mm256_shuffle_epi32(d01, 0x1B);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xCC);
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xAA);
+
+ s = d2i(_mm256_permute4x64_pd(i2d(_mm256_shuffle_epi32(d01, 0x1B)), 0x4E));
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xF0);
+
+ s = _mm256_shuffle_epi32(d01, 0x4E);
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xCC);
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xAA);
+}
+ static INLINE void sort_01v_merge_ascending(__m256i& d01) {
+ __m256i min, max, s;
+
+ s = d2i(_mm256_permute4x64_pd(i2d(d01), 0x4E));
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xF0);
+
+ s = _mm256_shuffle_epi32(d01, 0x4E);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xCC);
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(min, max, 0xAA);
+ }
+ static INLINE void sort_01v_descending(__m256i& d01) {
+ __m256i min, max, s;
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xAA);
+
+ s = _mm256_shuffle_epi32(d01, 0x1B);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xCC);
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xAA);
+
+ s = d2i(_mm256_permute4x64_pd(i2d(_mm256_shuffle_epi32(d01, 0x1B)), 0x4E));
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xF0);
+
+ s = _mm256_shuffle_epi32(d01, 0x4E);
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xCC);
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xAA);
+}
+ static INLINE void sort_01v_merge_descending(__m256i& d01) {
+ __m256i min, max, s;
+
+ s = d2i(_mm256_permute4x64_pd(i2d(d01), 0x4E));
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xF0);
+
+ s = _mm256_shuffle_epi32(d01, 0x4E);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xCC);
+
+ s = _mm256_shuffle_epi32(d01, 0xB1);
+
+ min = _mm256_min_epi32(s, d01);
+ max = _mm256_max_epi32(s, d01);
+ d01 = _mm256_blend_epi32(max, min, 0xAA);
+ }
+ static INLINE void sort_02v_ascending(__m256i& d01, __m256i& d02) {
+ __m256i tmp;
+
+ sort_01v_ascending(d01);
+ sort_01v_descending(d02);
+
+ tmp = d02;
+
+ d02 = _mm256_max_epi32(d01, d02);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_descending(__m256i& d01, __m256i& d02) {
+ __m256i tmp;
+
+ sort_01v_descending(d01);
+ sort_01v_ascending(d02);
+
+ tmp = d02;
+
+ d02 = _mm256_max_epi32(d01, d02);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_02v_merge_ascending(__m256i& d01, __m256i& d02) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d02, d01);
+
+ d02 = _mm256_max_epi32(d02, tmp);
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_merge_descending(__m256i& d01, __m256i& d02) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d02, d01);
+
+ d02 = _mm256_max_epi32(d02, tmp);
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_03v_ascending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_01v_descending(d03);
+
+ tmp = d03;
+
+ d03 = _mm256_max_epi32(d02, d03);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_descending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp;
+
+ sort_02v_descending(d01, d02);
+ sort_01v_ascending(d03);
+
+ tmp = d03;
+
+ d03 = _mm256_max_epi32(d02, d03);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_03v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d03, d01);
+
+ d03 = _mm256_max_epi32(d03, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d03, d01);
+
+ d03 = _mm256_max_epi32(d03, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_04v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_02v_descending(d03, d04);
+
+ tmp = d03;
+
+ d03 = _mm256_max_epi32(d02, d03);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ tmp = d04;
+
+ d04 = _mm256_max_epi32(d01, d04);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp;
+
+ sort_02v_descending(d01, d02);
+ sort_02v_ascending(d03, d04);
+
+ tmp = d03;
+
+ d03 = _mm256_max_epi32(d02, d03);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ tmp = d04;
+
+ d04 = _mm256_max_epi32(d01, d04);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d03, d01);
+
+ d03 = _mm256_max_epi32(d03, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d04, d02);
+
+ d04 = _mm256_max_epi32(d04, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d03, d01);
+
+ d03 = _mm256_max_epi32(d03, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d04, d02);
+
+ d04 = _mm256_max_epi32(d04, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_05v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_01v_descending(d05);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_01v_ascending(d05);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_05v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_06v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_02v_descending(d05, d06);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d06;
+
+ d06 = _mm256_max_epi32(d03, d06);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_02v_ascending(d05, d06);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d06;
+
+ d06 = _mm256_max_epi32(d03, d06);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d06, d02);
+
+ d06 = _mm256_max_epi32(d06, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d06, d02);
+
+ d06 = _mm256_max_epi32(d06, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_07v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_03v_descending(d05, d06, d07);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d06;
+
+ d06 = _mm256_max_epi32(d03, d06);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d07;
+
+ d07 = _mm256_max_epi32(d02, d07);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_03v_ascending(d05, d06, d07);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d06;
+
+ d06 = _mm256_max_epi32(d03, d06);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d07;
+
+ d07 = _mm256_max_epi32(d02, d07);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d06, d02);
+
+ d06 = _mm256_max_epi32(d06, tmp);
+
+ tmp = d03;
+
+ d03 = _mm256_min_epi32(d07, d03);
+
+ d07 = _mm256_max_epi32(d07, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d06, d02);
+
+ d06 = _mm256_max_epi32(d06, tmp);
+
+ tmp = d03;
+
+ d03 = _mm256_min_epi32(d07, d03);
+
+ d07 = _mm256_max_epi32(d07, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_08v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_04v_descending(d05, d06, d07, d08);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d06;
+
+ d06 = _mm256_max_epi32(d03, d06);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d07;
+
+ d07 = _mm256_max_epi32(d02, d07);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ tmp = d08;
+
+ d08 = _mm256_max_epi32(d01, d08);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_04v_ascending(d05, d06, d07, d08);
+
+ tmp = d05;
+
+ d05 = _mm256_max_epi32(d04, d05);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d06;
+
+ d06 = _mm256_max_epi32(d03, d06);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d07;
+
+ d07 = _mm256_max_epi32(d02, d07);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ tmp = d08;
+
+ d08 = _mm256_max_epi32(d01, d08);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d06, d02);
+
+ d06 = _mm256_max_epi32(d06, tmp);
+
+ tmp = d03;
+
+ d03 = _mm256_min_epi32(d07, d03);
+
+ d07 = _mm256_max_epi32(d07, tmp);
+
+ tmp = d04;
+
+ d04 = _mm256_min_epi32(d08, d04);
+
+ d08 = _mm256_max_epi32(d08, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp;
+
+ tmp = d01;
+
+ d01 = _mm256_min_epi32(d05, d01);
+
+ d05 = _mm256_max_epi32(d05, tmp);
+
+ tmp = d02;
+
+ d02 = _mm256_min_epi32(d06, d02);
+
+ d06 = _mm256_max_epi32(d06, tmp);
+
+ tmp = d03;
+
+ d03 = _mm256_min_epi32(d07, d03);
+
+ d07 = _mm256_max_epi32(d07, tmp);
+
+ tmp = d04;
+
+ d04 = _mm256_min_epi32(d08, d04);
+
+ d08 = _mm256_max_epi32(d08, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_09v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_descending(d09);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_ascending(d09);
+ }
+ static INLINE void sort_09v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_ascending(d09);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_descending(d09);
+ }
+ static INLINE void sort_10v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_descending(d09, d10);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_ascending(d09, d10);
+ }
+ static INLINE void sort_10v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_ascending(d09, d10);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_descending(d09, d10);
+ }
+ static INLINE void sort_11v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_descending(d09, d10, d11);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_ascending(d09, d10, d11);
+ }
+ static INLINE void sort_11v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_ascending(d09, d10, d11);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_descending(d09, d10, d11);
+ }
+ static INLINE void sort_12v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_descending(d09, d10, d11, d12);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_ascending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_12v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_ascending(d09, d10, d11, d12);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_descending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_13v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_descending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_ascending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_13v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_ascending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_descending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_14v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_descending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d14;
+
+ d14 = _mm256_max_epi32(d03, d14);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_ascending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_14v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_ascending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d14;
+
+ d14 = _mm256_max_epi32(d03, d14);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_descending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_15v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_descending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d14;
+
+ d14 = _mm256_max_epi32(d03, d14);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d15;
+
+ d15 = _mm256_max_epi32(d02, d15);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_ascending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_15v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_ascending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d14;
+
+ d14 = _mm256_max_epi32(d03, d14);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d15;
+
+ d15 = _mm256_max_epi32(d02, d15);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_descending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_16v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15, __m256i& d16) {
+ __m256i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d14;
+
+ d14 = _mm256_max_epi32(d03, d14);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d15;
+
+ d15 = _mm256_max_epi32(d02, d15);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ tmp = d16;
+
+ d16 = _mm256_max_epi32(d01, d16);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+ static INLINE void sort_16v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15, __m256i& d16) {
+ __m256i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+
+ d09 = _mm256_max_epi32(d08, d09);
+ d08 = _mm256_min_epi32(d08, tmp);
+
+ tmp = d10;
+
+ d10 = _mm256_max_epi32(d07, d10);
+ d07 = _mm256_min_epi32(d07, tmp);
+
+ tmp = d11;
+
+ d11 = _mm256_max_epi32(d06, d11);
+ d06 = _mm256_min_epi32(d06, tmp);
+
+ tmp = d12;
+
+ d12 = _mm256_max_epi32(d05, d12);
+ d05 = _mm256_min_epi32(d05, tmp);
+
+ tmp = d13;
+
+ d13 = _mm256_max_epi32(d04, d13);
+ d04 = _mm256_min_epi32(d04, tmp);
+
+ tmp = d14;
+
+ d14 = _mm256_max_epi32(d03, d14);
+ d03 = _mm256_min_epi32(d03, tmp);
+
+ tmp = d15;
+
+ d15 = _mm256_max_epi32(d02, d15);
+ d02 = _mm256_min_epi32(d02, tmp);
+
+ tmp = d16;
+
+ d16 = _mm256_max_epi32(d01, d16);
+ d01 = _mm256_min_epi32(d01, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+
+ static NOINLINE void sort_01v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ sort_01v_ascending(d01);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+}
+
+ static NOINLINE void sort_02v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ sort_02v_ascending(d01, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+}
+
+ static NOINLINE void sort_03v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ sort_03v_ascending(d01, d02, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+}
+
+ static NOINLINE void sort_04v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ sort_04v_ascending(d01, d02, d03, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+}
+
+ static NOINLINE void sort_05v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ sort_05v_ascending(d01, d02, d03, d04, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+}
+
+ static NOINLINE void sort_06v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ sort_06v_ascending(d01, d02, d03, d04, d05, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+}
+
+ static NOINLINE void sort_07v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ sort_07v_ascending(d01, d02, d03, d04, d05, d06, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+}
+
+ static NOINLINE void sort_08v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+}
+
+ static NOINLINE void sort_09v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ sort_09v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+}
+
+ static NOINLINE void sort_10v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ sort_10v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+}
+
+ static NOINLINE void sort_11v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ sort_11v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+}
+
+ static NOINLINE void sort_12v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ sort_12v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+}
+
+ static NOINLINE void sort_13v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ sort_13v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+}
+
+ static NOINLINE void sort_14v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ __m256i d14 = _mm256_lddqu_si256((__m256i const *) ptr + 13);;
+ sort_14v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 13, d14);
+}
+
+ static NOINLINE void sort_15v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ __m256i d14 = _mm256_lddqu_si256((__m256i const *) ptr + 13);;
+ __m256i d15 = _mm256_lddqu_si256((__m256i const *) ptr + 14);;
+ sort_15v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 13, d14);
+ _mm256_storeu_si256((__m256i *) ptr + 14, d15);
+}
+
+ static NOINLINE void sort_16v(int32_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ __m256i d14 = _mm256_lddqu_si256((__m256i const *) ptr + 13);;
+ __m256i d15 = _mm256_lddqu_si256((__m256i const *) ptr + 14);;
+ __m256i d16 = _mm256_lddqu_si256((__m256i const *) ptr + 15);;
+ sort_16v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15, d16);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 13, d14);
+ _mm256_storeu_si256((__m256i *) ptr + 14, d15);
+ _mm256_storeu_si256((__m256i *) ptr + 15, d16);
+}
+ static void sort(int32_t *ptr, size_t length);
+
+};
+}
+}
+
+#undef i2d
+#undef d2i
+#undef i2s
+#undef s2i
+#undef s2d
+#undef d2s
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute pop
+#else
+#pragma GCC pop_options
+#endif
+#endif
+#endif
+
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+#include "bitonic_sort.AVX2.int64_t.generated.h"
+
+using namespace vxsort;
+
+void vxsort::smallsort::bitonic<int64_t, vector_machine::AVX2 >::sort(int64_t *ptr, size_t length) {
+ const int N = 4;
+
+ switch(length / N) {
+ case 1: sort_01v(ptr); break;
+ case 2: sort_02v(ptr); break;
+ case 3: sort_03v(ptr); break;
+ case 4: sort_04v(ptr); break;
+ case 5: sort_05v(ptr); break;
+ case 6: sort_06v(ptr); break;
+ case 7: sort_07v(ptr); break;
+ case 8: sort_08v(ptr); break;
+ case 9: sort_09v(ptr); break;
+ case 10: sort_10v(ptr); break;
+ case 11: sort_11v(ptr); break;
+ case 12: sort_12v(ptr); break;
+ case 13: sort_13v(ptr); break;
+ case 14: sort_14v(ptr); break;
+ case 15: sort_15v(ptr); break;
+ case 16: sort_16v(ptr); break;
+ }
+}
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+/////////////////////////////////////////////////////////////////////////////
+////
+// This file was auto-generated by a tool at 2020-06-22 05:27:48
+//
+// It is recommended you DO NOT directly edit this file but instead edit
+// the code-generator that generated this source file instead.
+/////////////////////////////////////////////////////////////////////////////
+
+#ifndef BITONIC_SORT_AVX2_INT64_T_H
+#define BITONIC_SORT_AVX2_INT64_T_H
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx2"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx2")
+#endif
+#endif
+
+#include <immintrin.h>
+#include "bitonic_sort.h"
+
+#define i2d _mm256_castsi256_pd
+#define d2i _mm256_castpd_si256
+#define i2s _mm256_castsi256_ps
+#define s2i _mm256_castps_si256
+#define s2d _mm256_castps_pd
+#define d2s _mm256_castpd_ps
+
+namespace vxsort {
+namespace smallsort {
+template<> struct bitonic<int64_t, AVX2> {
+public:
+
+ static INLINE void sort_01v_ascending(__m256i& d01) {
+ __m256i min, max, s, cmp;
+
+ s = d2i(_mm256_shuffle_pd(i2d(d01), i2d(d01), 0x5));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(min), i2d(max), 0xA));
+
+ s = d2i(_mm256_permute4x64_pd(i2d(d01), 0x1B));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(min), i2d(max), 0xC));
+
+ s = d2i(_mm256_shuffle_pd(i2d(d01), i2d(d01), 0x5));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(min), i2d(max), 0xA));
+}
+ static INLINE void sort_01v_merge_ascending(__m256i& d01) {
+ __m256i min, max, s, cmp;
+
+ s = d2i(_mm256_permute4x64_pd(i2d(d01), 0x4E));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(min), i2d(max), 0xC));
+
+ s = d2i(_mm256_shuffle_pd(i2d(d01), i2d(d01), 0x5));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(min), i2d(max), 0xA));
+ }
+ static INLINE void sort_01v_descending(__m256i& d01) {
+ __m256i min, max, s, cmp;
+
+ s = d2i(_mm256_shuffle_pd(i2d(d01), i2d(d01), 0x5));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(max), i2d(min), 0xA));
+
+ s = d2i(_mm256_permute4x64_pd(i2d(d01), 0x1B));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(max), i2d(min), 0xC));
+
+ s = d2i(_mm256_shuffle_pd(i2d(d01), i2d(d01), 0x5));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(max), i2d(min), 0xA));
+}
+ static INLINE void sort_01v_merge_descending(__m256i& d01) {
+ __m256i min, max, s, cmp;
+
+ s = d2i(_mm256_permute4x64_pd(i2d(d01), 0x4E));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(max), i2d(min), 0xC));
+
+ s = d2i(_mm256_shuffle_pd(i2d(d01), i2d(d01), 0x5));
+ cmp = _mm256_cmpgt_epi64(s, d01);
+ min = d2i(_mm256_blendv_pd(i2d(s), i2d(d01), i2d(cmp)));
+ max = d2i(_mm256_blendv_pd(i2d(d01), i2d(s), i2d(cmp)));
+ d01 = d2i(_mm256_blend_pd(i2d(max), i2d(min), 0xA));
+ }
+ static INLINE void sort_02v_ascending(__m256i& d01, __m256i& d02) {
+ __m256i tmp, cmp;
+
+ sort_01v_ascending(d01);
+ sort_01v_descending(d02);
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d01, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_descending(__m256i& d01, __m256i& d02) {
+ __m256i tmp, cmp;
+
+ sort_01v_descending(d01);
+ sort_01v_ascending(d02);
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d01, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_02v_merge_ascending(__m256i& d01, __m256i& d02) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d02, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d02), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d02, tmp);
+ d02 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d02), i2d(cmp)));
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_merge_descending(__m256i& d01, __m256i& d02) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d02, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d02), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d02, tmp);
+ d02 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d02), i2d(cmp)));
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_03v_ascending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp, cmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_01v_descending(d03);
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d02, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_descending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp, cmp;
+
+ sort_02v_descending(d01, d02);
+ sort_01v_ascending(d03);
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d02, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_03v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d03, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d03, tmp);
+ d03 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d03), i2d(cmp)));
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d03, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d03, tmp);
+ d03 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d03), i2d(cmp)));
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_04v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp, cmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_02v_descending(d03, d04);
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d02, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ tmp = d04;
+ cmp = _mm256_cmpgt_epi64(d01, d04);
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp, cmp;
+
+ sort_02v_descending(d01, d02);
+ sort_02v_ascending(d03, d04);
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d02, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ tmp = d04;
+ cmp = _mm256_cmpgt_epi64(d01, d04);
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d03, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d03, tmp);
+ d03 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d03), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d04, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d04), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d04, tmp);
+ d04 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d04), i2d(cmp)));
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d03, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d03), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d03, tmp);
+ d03 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d03), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d04, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d04), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d04, tmp);
+ d04 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d04), i2d(cmp)));
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_05v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp, cmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_01v_descending(d05);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp, cmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_01v_ascending(d05);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_05v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_06v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp, cmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_02v_descending(d05, d06);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d06;
+ cmp = _mm256_cmpgt_epi64(d03, d06);
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp, cmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_02v_ascending(d05, d06);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d06;
+ cmp = _mm256_cmpgt_epi64(d03, d06);
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d06, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d06, tmp);
+ d06 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d06), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d06, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d06, tmp);
+ d06 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d06), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_07v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp, cmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_03v_descending(d05, d06, d07);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d06;
+ cmp = _mm256_cmpgt_epi64(d03, d06);
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d07;
+ cmp = _mm256_cmpgt_epi64(d02, d07);
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp, cmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_03v_ascending(d05, d06, d07);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d06;
+ cmp = _mm256_cmpgt_epi64(d03, d06);
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d07;
+ cmp = _mm256_cmpgt_epi64(d02, d07);
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d06, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d06, tmp);
+ d06 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d06), i2d(cmp)));
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d07, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d03), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d07, tmp);
+ d07 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d07), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d06, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d06, tmp);
+ d06 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d06), i2d(cmp)));
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d07, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d03), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d07, tmp);
+ d07 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d07), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_08v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp, cmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_04v_descending(d05, d06, d07, d08);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d06;
+ cmp = _mm256_cmpgt_epi64(d03, d06);
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d07;
+ cmp = _mm256_cmpgt_epi64(d02, d07);
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ tmp = d08;
+ cmp = _mm256_cmpgt_epi64(d01, d08);
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp, cmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_04v_ascending(d05, d06, d07, d08);
+
+ tmp = d05;
+ cmp = _mm256_cmpgt_epi64(d04, d05);
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d06;
+ cmp = _mm256_cmpgt_epi64(d03, d06);
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d07;
+ cmp = _mm256_cmpgt_epi64(d02, d07);
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ tmp = d08;
+ cmp = _mm256_cmpgt_epi64(d01, d08);
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d06, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d06, tmp);
+ d06 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d06), i2d(cmp)));
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d07, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d03), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d07, tmp);
+ d07 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d07), i2d(cmp)));
+
+ tmp = d04;
+ cmp = _mm256_cmpgt_epi64(d08, d04);
+ d04 = d2i(_mm256_blendv_pd(i2d(d08), i2d(d04), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d08, tmp);
+ d08 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d08), i2d(cmp)));
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08) {
+ __m256i tmp, cmp;
+
+ tmp = d01;
+ cmp = _mm256_cmpgt_epi64(d05, d01);
+ d01 = d2i(_mm256_blendv_pd(i2d(d05), i2d(d01), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d05, tmp);
+ d05 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d05), i2d(cmp)));
+
+ tmp = d02;
+ cmp = _mm256_cmpgt_epi64(d06, d02);
+ d02 = d2i(_mm256_blendv_pd(i2d(d06), i2d(d02), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d06, tmp);
+ d06 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d06), i2d(cmp)));
+
+ tmp = d03;
+ cmp = _mm256_cmpgt_epi64(d07, d03);
+ d03 = d2i(_mm256_blendv_pd(i2d(d07), i2d(d03), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d07, tmp);
+ d07 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d07), i2d(cmp)));
+
+ tmp = d04;
+ cmp = _mm256_cmpgt_epi64(d08, d04);
+ d04 = d2i(_mm256_blendv_pd(i2d(d08), i2d(d04), i2d(cmp)));
+ cmp = _mm256_cmpgt_epi64(d08, tmp);
+ d08 = d2i(_mm256_blendv_pd(i2d(tmp), i2d(d08), i2d(cmp)));
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_09v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_descending(d09);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_ascending(d09);
+ }
+ static INLINE void sort_09v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_ascending(d09);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_descending(d09);
+ }
+ static INLINE void sort_10v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_descending(d09, d10);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_ascending(d09, d10);
+ }
+ static INLINE void sort_10v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_ascending(d09, d10);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_descending(d09, d10);
+ }
+ static INLINE void sort_11v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_descending(d09, d10, d11);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_ascending(d09, d10, d11);
+ }
+ static INLINE void sort_11v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_ascending(d09, d10, d11);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_descending(d09, d10, d11);
+ }
+ static INLINE void sort_12v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_descending(d09, d10, d11, d12);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_ascending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_12v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_ascending(d09, d10, d11, d12);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_descending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_13v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_descending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_ascending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_13v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_ascending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_descending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_14v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_descending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d14;
+ cmp = _mm256_cmpgt_epi64(d03, d14);
+ d14 = d2i(_mm256_blendv_pd(i2d(d14), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_ascending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_14v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_ascending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d14;
+ cmp = _mm256_cmpgt_epi64(d03, d14);
+ d14 = d2i(_mm256_blendv_pd(i2d(d14), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_descending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_15v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_descending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d14;
+ cmp = _mm256_cmpgt_epi64(d03, d14);
+ d14 = d2i(_mm256_blendv_pd(i2d(d14), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d15;
+ cmp = _mm256_cmpgt_epi64(d02, d15);
+ d15 = d2i(_mm256_blendv_pd(i2d(d15), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_ascending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_15v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_ascending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d14;
+ cmp = _mm256_cmpgt_epi64(d03, d14);
+ d14 = d2i(_mm256_blendv_pd(i2d(d14), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d15;
+ cmp = _mm256_cmpgt_epi64(d02, d15);
+ d15 = d2i(_mm256_blendv_pd(i2d(d15), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_descending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_16v_ascending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15, __m256i& d16) {
+ __m256i tmp, cmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d14;
+ cmp = _mm256_cmpgt_epi64(d03, d14);
+ d14 = d2i(_mm256_blendv_pd(i2d(d14), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d15;
+ cmp = _mm256_cmpgt_epi64(d02, d15);
+ d15 = d2i(_mm256_blendv_pd(i2d(d15), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ tmp = d16;
+ cmp = _mm256_cmpgt_epi64(d01, d16);
+ d16 = d2i(_mm256_blendv_pd(i2d(d16), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+ static INLINE void sort_16v_descending(__m256i& d01, __m256i& d02, __m256i& d03, __m256i& d04, __m256i& d05, __m256i& d06, __m256i& d07, __m256i& d08, __m256i& d09, __m256i& d10, __m256i& d11, __m256i& d12, __m256i& d13, __m256i& d14, __m256i& d15, __m256i& d16) {
+ __m256i tmp, cmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+ cmp = _mm256_cmpgt_epi64(d08, d09);
+ d09 = d2i(_mm256_blendv_pd(i2d(d09), i2d(d08), i2d(cmp)));
+ d08 = d2i(_mm256_blendv_pd(i2d(d08), i2d(tmp), i2d(cmp)));
+
+ tmp = d10;
+ cmp = _mm256_cmpgt_epi64(d07, d10);
+ d10 = d2i(_mm256_blendv_pd(i2d(d10), i2d(d07), i2d(cmp)));
+ d07 = d2i(_mm256_blendv_pd(i2d(d07), i2d(tmp), i2d(cmp)));
+
+ tmp = d11;
+ cmp = _mm256_cmpgt_epi64(d06, d11);
+ d11 = d2i(_mm256_blendv_pd(i2d(d11), i2d(d06), i2d(cmp)));
+ d06 = d2i(_mm256_blendv_pd(i2d(d06), i2d(tmp), i2d(cmp)));
+
+ tmp = d12;
+ cmp = _mm256_cmpgt_epi64(d05, d12);
+ d12 = d2i(_mm256_blendv_pd(i2d(d12), i2d(d05), i2d(cmp)));
+ d05 = d2i(_mm256_blendv_pd(i2d(d05), i2d(tmp), i2d(cmp)));
+
+ tmp = d13;
+ cmp = _mm256_cmpgt_epi64(d04, d13);
+ d13 = d2i(_mm256_blendv_pd(i2d(d13), i2d(d04), i2d(cmp)));
+ d04 = d2i(_mm256_blendv_pd(i2d(d04), i2d(tmp), i2d(cmp)));
+
+ tmp = d14;
+ cmp = _mm256_cmpgt_epi64(d03, d14);
+ d14 = d2i(_mm256_blendv_pd(i2d(d14), i2d(d03), i2d(cmp)));
+ d03 = d2i(_mm256_blendv_pd(i2d(d03), i2d(tmp), i2d(cmp)));
+
+ tmp = d15;
+ cmp = _mm256_cmpgt_epi64(d02, d15);
+ d15 = d2i(_mm256_blendv_pd(i2d(d15), i2d(d02), i2d(cmp)));
+ d02 = d2i(_mm256_blendv_pd(i2d(d02), i2d(tmp), i2d(cmp)));
+
+ tmp = d16;
+ cmp = _mm256_cmpgt_epi64(d01, d16);
+ d16 = d2i(_mm256_blendv_pd(i2d(d16), i2d(d01), i2d(cmp)));
+ d01 = d2i(_mm256_blendv_pd(i2d(d01), i2d(tmp), i2d(cmp)));
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+
+ static NOINLINE void sort_01v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ sort_01v_ascending(d01);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+}
+
+ static NOINLINE void sort_02v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ sort_02v_ascending(d01, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+}
+
+ static NOINLINE void sort_03v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ sort_03v_ascending(d01, d02, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+}
+
+ static NOINLINE void sort_04v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ sort_04v_ascending(d01, d02, d03, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+}
+
+ static NOINLINE void sort_05v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ sort_05v_ascending(d01, d02, d03, d04, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+}
+
+ static NOINLINE void sort_06v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ sort_06v_ascending(d01, d02, d03, d04, d05, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+}
+
+ static NOINLINE void sort_07v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ sort_07v_ascending(d01, d02, d03, d04, d05, d06, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+}
+
+ static NOINLINE void sort_08v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+}
+
+ static NOINLINE void sort_09v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ sort_09v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+}
+
+ static NOINLINE void sort_10v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ sort_10v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+}
+
+ static NOINLINE void sort_11v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ sort_11v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+}
+
+ static NOINLINE void sort_12v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ sort_12v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+}
+
+ static NOINLINE void sort_13v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ sort_13v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+}
+
+ static NOINLINE void sort_14v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ __m256i d14 = _mm256_lddqu_si256((__m256i const *) ptr + 13);;
+ sort_14v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 13, d14);
+}
+
+ static NOINLINE void sort_15v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ __m256i d14 = _mm256_lddqu_si256((__m256i const *) ptr + 13);;
+ __m256i d15 = _mm256_lddqu_si256((__m256i const *) ptr + 14);;
+ sort_15v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 13, d14);
+ _mm256_storeu_si256((__m256i *) ptr + 14, d15);
+}
+
+ static NOINLINE void sort_16v(int64_t *ptr) {
+ __m256i d01 = _mm256_lddqu_si256((__m256i const *) ptr + 0);;
+ __m256i d02 = _mm256_lddqu_si256((__m256i const *) ptr + 1);;
+ __m256i d03 = _mm256_lddqu_si256((__m256i const *) ptr + 2);;
+ __m256i d04 = _mm256_lddqu_si256((__m256i const *) ptr + 3);;
+ __m256i d05 = _mm256_lddqu_si256((__m256i const *) ptr + 4);;
+ __m256i d06 = _mm256_lddqu_si256((__m256i const *) ptr + 5);;
+ __m256i d07 = _mm256_lddqu_si256((__m256i const *) ptr + 6);;
+ __m256i d08 = _mm256_lddqu_si256((__m256i const *) ptr + 7);;
+ __m256i d09 = _mm256_lddqu_si256((__m256i const *) ptr + 8);;
+ __m256i d10 = _mm256_lddqu_si256((__m256i const *) ptr + 9);;
+ __m256i d11 = _mm256_lddqu_si256((__m256i const *) ptr + 10);;
+ __m256i d12 = _mm256_lddqu_si256((__m256i const *) ptr + 11);;
+ __m256i d13 = _mm256_lddqu_si256((__m256i const *) ptr + 12);;
+ __m256i d14 = _mm256_lddqu_si256((__m256i const *) ptr + 13);;
+ __m256i d15 = _mm256_lddqu_si256((__m256i const *) ptr + 14);;
+ __m256i d16 = _mm256_lddqu_si256((__m256i const *) ptr + 15);;
+ sort_16v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15, d16);
+ _mm256_storeu_si256((__m256i *) ptr + 0, d01);
+ _mm256_storeu_si256((__m256i *) ptr + 1, d02);
+ _mm256_storeu_si256((__m256i *) ptr + 2, d03);
+ _mm256_storeu_si256((__m256i *) ptr + 3, d04);
+ _mm256_storeu_si256((__m256i *) ptr + 4, d05);
+ _mm256_storeu_si256((__m256i *) ptr + 5, d06);
+ _mm256_storeu_si256((__m256i *) ptr + 6, d07);
+ _mm256_storeu_si256((__m256i *) ptr + 7, d08);
+ _mm256_storeu_si256((__m256i *) ptr + 8, d09);
+ _mm256_storeu_si256((__m256i *) ptr + 9, d10);
+ _mm256_storeu_si256((__m256i *) ptr + 10, d11);
+ _mm256_storeu_si256((__m256i *) ptr + 11, d12);
+ _mm256_storeu_si256((__m256i *) ptr + 12, d13);
+ _mm256_storeu_si256((__m256i *) ptr + 13, d14);
+ _mm256_storeu_si256((__m256i *) ptr + 14, d15);
+ _mm256_storeu_si256((__m256i *) ptr + 15, d16);
+}
+ static void sort(int64_t *ptr, size_t length);
+
+};
+}
+}
+
+#undef i2d
+#undef d2i
+#undef i2s
+#undef s2i
+#undef s2d
+#undef d2s
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute pop
+#else
+#pragma GCC pop_options
+#endif
+#endif
+#endif
+
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+#include "bitonic_sort.AVX512.int32_t.generated.h"
+
+using namespace vxsort;
+
+void vxsort::smallsort::bitonic<int32_t, vector_machine::AVX512 >::sort(int32_t *ptr, size_t length) {
+ const int N = 16;
+
+ switch(length / N) {
+ case 1: sort_01v(ptr); break;
+ case 2: sort_02v(ptr); break;
+ case 3: sort_03v(ptr); break;
+ case 4: sort_04v(ptr); break;
+ case 5: sort_05v(ptr); break;
+ case 6: sort_06v(ptr); break;
+ case 7: sort_07v(ptr); break;
+ case 8: sort_08v(ptr); break;
+ case 9: sort_09v(ptr); break;
+ case 10: sort_10v(ptr); break;
+ case 11: sort_11v(ptr); break;
+ case 12: sort_12v(ptr); break;
+ case 13: sort_13v(ptr); break;
+ case 14: sort_14v(ptr); break;
+ case 15: sort_15v(ptr); break;
+ case 16: sort_16v(ptr); break;
+ }
+}
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+/////////////////////////////////////////////////////////////////////////////
+////
+// This file was auto-generated by a tool at 2020-06-22 05:27:48
+//
+// It is recommended you DO NOT directly edit this file but instead edit
+// the code-generator that generated this source file instead.
+/////////////////////////////////////////////////////////////////////////////
+
+#ifndef BITONIC_SORT_AVX512_INT32_T_H
+#define BITONIC_SORT_AVX512_INT32_T_H
+
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx512f"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx512f")
+#endif
+#endif
+
+#include <immintrin.h>
+#include "bitonic_sort.h"
+
+#define i2d _mm512_castsi512_pd
+#define d2i _mm512_castpd_si512
+#define i2s _mm512_castsi512_ps
+#define s2i _mm512_castps_si512
+#define s2d _mm512_castps_pd
+#define d2s _mm521_castpd_ps
+
+namespace vxsort {
+namespace smallsort {
+template<> struct bitonic<int32_t, AVX512> {
+public:
+
+ static INLINE void sort_01v_ascending(__m512i& d01) {
+ __m512i min, s;
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xAAAA, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_ABCD);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xCCCC, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xAAAA, s, d01);
+
+ s = _mm512_permutex_epi64(_mm512_shuffle_epi32(d01, _MM_PERM_ABCD), _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xF0F0, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xCCCC, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xAAAA, s, d01);
+
+ s = _mm512_shuffle_i64x2(_mm512_shuffle_epi32(d01, _MM_PERM_ABCD), _mm512_shuffle_epi32(d01, _MM_PERM_ABCD), _MM_PERM_ABCD);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xFF00, s, d01);
+
+ s = _mm512_permutex_epi64(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xF0F0, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xCCCC, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xAAAA, s, d01);
+ }
+ static INLINE void sort_01v_merge_ascending(__m512i& d01) {
+ __m512i min, s;
+
+ s = _mm512_shuffle_i64x2(d01, d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xFF00, s, d01);
+
+ s = _mm512_permutex_epi64(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xF0F0, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xCCCC, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0xAAAA, s, d01);
+ }
+ static INLINE void sort_01v_descending(__m512i& d01) {
+ __m512i min, s;
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x5555, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_ABCD);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x3333, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x5555, s, d01);
+
+ s = _mm512_permutex_epi64(_mm512_shuffle_epi32(d01, _MM_PERM_ABCD), _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x0F0F, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x3333, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x5555, s, d01);
+
+ s = _mm512_shuffle_i64x2(_mm512_shuffle_epi32(d01, _MM_PERM_ABCD), _mm512_shuffle_epi32(d01, _MM_PERM_ABCD), _MM_PERM_ABCD);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x00FF, s, d01);
+
+ s = _mm512_permutex_epi64(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x0F0F, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x3333, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x5555, s, d01);
+ }
+ static INLINE void sort_01v_merge_descending(__m512i& d01) {
+ __m512i min, s;
+
+ s = _mm512_shuffle_i64x2(d01, d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x00FF, s, d01);
+
+ s = _mm512_permutex_epi64(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x0F0F, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_BADC);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x3333, s, d01);
+
+ s = _mm512_shuffle_epi32(d01, _MM_PERM_CDAB);
+ min = _mm512_min_epi32(s, d01);
+ d01 = _mm512_mask_max_epi32(min, 0x5555, s, d01);
+ }
+ static INLINE void sort_02v_ascending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ sort_01v_ascending(d01);
+ sort_01v_descending(d02);
+
+ tmp = d02;
+ d02 = _mm512_max_epi32(d01, d02);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_descending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ sort_01v_descending(d01);
+ sort_01v_ascending(d02);
+
+ tmp = d02;
+ d02 = _mm512_max_epi32(d01, d02);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_02v_merge_ascending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d02, d01);
+ d02 = _mm512_max_epi32(d02, tmp);
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_merge_descending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d02, d01);
+ d02 = _mm512_max_epi32(d02, tmp);
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_03v_ascending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_01v_descending(d03);
+
+ tmp = d03;
+ d03 = _mm512_max_epi32(d02, d03);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_descending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ sort_02v_descending(d01, d02);
+ sort_01v_ascending(d03);
+
+ tmp = d03;
+ d03 = _mm512_max_epi32(d02, d03);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_03v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d03, d01);
+ d03 = _mm512_max_epi32(d03, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d03, d01);
+ d03 = _mm512_max_epi32(d03, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_04v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_02v_descending(d03, d04);
+
+ tmp = d03;
+ d03 = _mm512_max_epi32(d02, d03);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ tmp = d04;
+ d04 = _mm512_max_epi32(d01, d04);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ sort_02v_descending(d01, d02);
+ sort_02v_ascending(d03, d04);
+
+ tmp = d03;
+ d03 = _mm512_max_epi32(d02, d03);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ tmp = d04;
+ d04 = _mm512_max_epi32(d01, d04);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d03, d01);
+ d03 = _mm512_max_epi32(d03, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d04, d02);
+ d04 = _mm512_max_epi32(d04, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d03, d01);
+ d03 = _mm512_max_epi32(d03, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d04, d02);
+ d04 = _mm512_max_epi32(d04, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_05v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_01v_descending(d05);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_01v_ascending(d05);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_05v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_06v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_02v_descending(d05, d06);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi32(d03, d06);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_02v_ascending(d05, d06);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi32(d03, d06);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d06, d02);
+ d06 = _mm512_max_epi32(d06, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d06, d02);
+ d06 = _mm512_max_epi32(d06, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_07v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_03v_descending(d05, d06, d07);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi32(d03, d06);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi32(d02, d07);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_03v_ascending(d05, d06, d07);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi32(d03, d06);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi32(d02, d07);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d06, d02);
+ d06 = _mm512_max_epi32(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi32(d07, d03);
+ d07 = _mm512_max_epi32(d07, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d06, d02);
+ d06 = _mm512_max_epi32(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi32(d07, d03);
+ d07 = _mm512_max_epi32(d07, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_08v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_04v_descending(d05, d06, d07, d08);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi32(d03, d06);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi32(d02, d07);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ tmp = d08;
+ d08 = _mm512_max_epi32(d01, d08);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_04v_ascending(d05, d06, d07, d08);
+
+ tmp = d05;
+ d05 = _mm512_max_epi32(d04, d05);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi32(d03, d06);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi32(d02, d07);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ tmp = d08;
+ d08 = _mm512_max_epi32(d01, d08);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d06, d02);
+ d06 = _mm512_max_epi32(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi32(d07, d03);
+ d07 = _mm512_max_epi32(d07, tmp);
+
+ tmp = d04;
+ d04 = _mm512_min_epi32(d08, d04);
+ d08 = _mm512_max_epi32(d08, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi32(d05, d01);
+ d05 = _mm512_max_epi32(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi32(d06, d02);
+ d06 = _mm512_max_epi32(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi32(d07, d03);
+ d07 = _mm512_max_epi32(d07, tmp);
+
+ tmp = d04;
+ d04 = _mm512_min_epi32(d08, d04);
+ d08 = _mm512_max_epi32(d08, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_09v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_descending(d09);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_ascending(d09);
+ }
+ static INLINE void sort_09v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_ascending(d09);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_descending(d09);
+ }
+ static INLINE void sort_10v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_descending(d09, d10);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_ascending(d09, d10);
+ }
+ static INLINE void sort_10v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_ascending(d09, d10);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_descending(d09, d10);
+ }
+ static INLINE void sort_11v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_descending(d09, d10, d11);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_ascending(d09, d10, d11);
+ }
+ static INLINE void sort_11v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_ascending(d09, d10, d11);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_descending(d09, d10, d11);
+ }
+ static INLINE void sort_12v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_descending(d09, d10, d11, d12);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_ascending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_12v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_ascending(d09, d10, d11, d12);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_descending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_13v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_descending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_ascending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_13v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_ascending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_descending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_14v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_descending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi32(d03, d14);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_ascending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_14v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_ascending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi32(d03, d14);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_descending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_15v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_descending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi32(d03, d14);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi32(d02, d15);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_ascending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_15v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_ascending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi32(d03, d14);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi32(d02, d15);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_descending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_16v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15, __m512i& d16) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi32(d03, d14);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi32(d02, d15);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ tmp = d16;
+ d16 = _mm512_max_epi32(d01, d16);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+ static INLINE void sort_16v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15, __m512i& d16) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+ d09 = _mm512_max_epi32(d08, d09);
+ d08 = _mm512_min_epi32(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi32(d07, d10);
+ d07 = _mm512_min_epi32(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi32(d06, d11);
+ d06 = _mm512_min_epi32(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi32(d05, d12);
+ d05 = _mm512_min_epi32(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi32(d04, d13);
+ d04 = _mm512_min_epi32(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi32(d03, d14);
+ d03 = _mm512_min_epi32(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi32(d02, d15);
+ d02 = _mm512_min_epi32(d02, tmp);
+
+ tmp = d16;
+ d16 = _mm512_max_epi32(d01, d16);
+ d01 = _mm512_min_epi32(d01, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+
+ static NOINLINE void sort_01v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ sort_01v_ascending(d01);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+}
+
+ static NOINLINE void sort_02v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ sort_02v_ascending(d01, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+}
+
+ static NOINLINE void sort_03v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ sort_03v_ascending(d01, d02, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+}
+
+ static NOINLINE void sort_04v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ sort_04v_ascending(d01, d02, d03, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+}
+
+ static NOINLINE void sort_05v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ sort_05v_ascending(d01, d02, d03, d04, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+}
+
+ static NOINLINE void sort_06v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ sort_06v_ascending(d01, d02, d03, d04, d05, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+}
+
+ static NOINLINE void sort_07v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ sort_07v_ascending(d01, d02, d03, d04, d05, d06, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+}
+
+ static NOINLINE void sort_08v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+}
+
+ static NOINLINE void sort_09v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ sort_09v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+}
+
+ static NOINLINE void sort_10v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ sort_10v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+}
+
+ static NOINLINE void sort_11v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ sort_11v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+}
+
+ static NOINLINE void sort_12v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ sort_12v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+}
+
+ static NOINLINE void sort_13v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ sort_13v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+}
+
+ static NOINLINE void sort_14v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ __m512i d14 = _mm512_loadu_si512((__m512i const *) ptr + 13);;
+ sort_14v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 13, d14);
+}
+
+ static NOINLINE void sort_15v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ __m512i d14 = _mm512_loadu_si512((__m512i const *) ptr + 13);;
+ __m512i d15 = _mm512_loadu_si512((__m512i const *) ptr + 14);;
+ sort_15v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 13, d14);
+ _mm512_storeu_si512((__m512i *) ptr + 14, d15);
+}
+
+ static NOINLINE void sort_16v(int32_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ __m512i d14 = _mm512_loadu_si512((__m512i const *) ptr + 13);;
+ __m512i d15 = _mm512_loadu_si512((__m512i const *) ptr + 14);;
+ __m512i d16 = _mm512_loadu_si512((__m512i const *) ptr + 15);;
+ sort_16v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15, d16);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 13, d14);
+ _mm512_storeu_si512((__m512i *) ptr + 14, d15);
+ _mm512_storeu_si512((__m512i *) ptr + 15, d16);
+}
+ static void sort(int32_t *ptr, size_t length);
+
+};
+}
+}
+
+#undef i2d
+#undef d2i
+#undef i2s
+#undef s2i
+#undef s2d
+#undef d2s
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute pop
+#else
+#pragma GCC pop_options
+#endif
+#endif
+#endif
+
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#include "common.h"
+#include "bitonic_sort.AVX512.int64_t.generated.h"
+
+using namespace vxsort;
+
+void vxsort::smallsort::bitonic<int64_t, vector_machine::AVX512 >::sort(int64_t *ptr, size_t length) {
+ const int N = 8;
+
+ switch(length / N) {
+ case 1: sort_01v(ptr); break;
+ case 2: sort_02v(ptr); break;
+ case 3: sort_03v(ptr); break;
+ case 4: sort_04v(ptr); break;
+ case 5: sort_05v(ptr); break;
+ case 6: sort_06v(ptr); break;
+ case 7: sort_07v(ptr); break;
+ case 8: sort_08v(ptr); break;
+ case 9: sort_09v(ptr); break;
+ case 10: sort_10v(ptr); break;
+ case 11: sort_11v(ptr); break;
+ case 12: sort_12v(ptr); break;
+ case 13: sort_13v(ptr); break;
+ case 14: sort_14v(ptr); break;
+ case 15: sort_15v(ptr); break;
+ case 16: sort_16v(ptr); break;
+ }
+}
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+/////////////////////////////////////////////////////////////////////////////
+////
+// This file was auto-generated by a tool at 2020-06-22 05:27:48
+//
+// It is recommended you DO NOT directly edit this file but instead edit
+// the code-generator that generated this source file instead.
+/////////////////////////////////////////////////////////////////////////////
+
+#ifndef BITONIC_SORT_AVX512_INT64_T_H
+#define BITONIC_SORT_AVX512_INT64_T_H
+
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx512f"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx512f")
+#endif
+#endif
+
+#include <immintrin.h>
+#include "bitonic_sort.h"
+
+#define i2d _mm512_castsi512_pd
+#define d2i _mm512_castpd_si512
+#define i2s _mm512_castsi512_ps
+#define s2i _mm512_castps_si512
+#define s2d _mm512_castps_pd
+#define d2s _mm521_castpd_ps
+
+namespace vxsort {
+namespace smallsort {
+template<> struct bitonic<int64_t, AVX512> {
+public:
+
+ static INLINE void sort_01v_ascending(__m512i& d01) {
+ __m512i min, s;
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00AA, s, d01);
+
+ s = d2i(_mm512_permutex_pd(i2d(d01), _MM_PERM_ABCD));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00CC, s, d01);
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00AA, s, d01);
+
+ s = d2i(_mm512_shuffle_f64x2(_mm512_permutex_pd(i2d(d01), _MM_PERM_ABCD), _mm512_permutex_pd(i2d(d01), _MM_PERM_ABCD), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00F0, s, d01);
+
+ s = d2i(_mm512_permutex_pd(i2d(d01), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00CC, s, d01);
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00AA, s, d01);
+ }
+ static INLINE void sort_01v_merge_ascending(__m512i& d01) {
+ __m512i min, s;
+
+ s = d2i(_mm512_shuffle_f64x2(i2d(d01), i2d(d01), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00F0, s, d01);
+
+ s = d2i(_mm512_permutex_pd(i2d(d01), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00CC, s, d01);
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x00AA, s, d01);
+ }
+ static INLINE void sort_01v_descending(__m512i& d01) {
+ __m512i min, s;
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x0055, s, d01);
+
+ s = d2i(_mm512_permutex_pd(i2d(d01), _MM_PERM_ABCD));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x0033, s, d01);
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x0055, s, d01);
+
+ s = d2i(_mm512_shuffle_f64x2(_mm512_permutex_pd(i2d(d01), _MM_PERM_ABCD), _mm512_permutex_pd(i2d(d01), _MM_PERM_ABCD), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x000F, s, d01);
+
+ s = d2i(_mm512_permutex_pd(i2d(d01), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x0033, s, d01);
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x0055, s, d01);
+ }
+ static INLINE void sort_01v_merge_descending(__m512i& d01) {
+ __m512i min, s;
+
+ s = d2i(_mm512_shuffle_f64x2(i2d(d01), i2d(d01), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x000F, s, d01);
+
+ s = d2i(_mm512_permutex_pd(i2d(d01), _MM_PERM_BADC));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x0033, s, d01);
+
+ s = d2i(_mm512_permute_pd(i2d(d01), _MM_PERM_BBBB));
+ min = _mm512_min_epi64(s, d01);
+ d01 = _mm512_mask_max_epi64(min, 0x0055, s, d01);
+ }
+ static INLINE void sort_02v_ascending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ sort_01v_ascending(d01);
+ sort_01v_descending(d02);
+
+ tmp = d02;
+ d02 = _mm512_max_epi64(d01, d02);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_descending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ sort_01v_descending(d01);
+ sort_01v_ascending(d02);
+
+ tmp = d02;
+ d02 = _mm512_max_epi64(d01, d02);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_02v_merge_ascending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d02, d01);
+ d02 = _mm512_max_epi64(d02, tmp);
+
+ sort_01v_merge_ascending(d01);
+ sort_01v_merge_ascending(d02);
+ }
+ static INLINE void sort_02v_merge_descending(__m512i& d01, __m512i& d02) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d02, d01);
+ d02 = _mm512_max_epi64(d02, tmp);
+
+ sort_01v_merge_descending(d01);
+ sort_01v_merge_descending(d02);
+ }
+ static INLINE void sort_03v_ascending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_01v_descending(d03);
+
+ tmp = d03;
+ d03 = _mm512_max_epi64(d02, d03);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_descending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ sort_02v_descending(d01, d02);
+ sort_01v_ascending(d03);
+
+ tmp = d03;
+ d03 = _mm512_max_epi64(d02, d03);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_03v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d03, d01);
+ d03 = _mm512_max_epi64(d03, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_01v_merge_ascending(d03);
+ }
+ static INLINE void sort_03v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d03, d01);
+ d03 = _mm512_max_epi64(d03, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_01v_merge_descending(d03);
+ }
+ static INLINE void sort_04v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ sort_02v_ascending(d01, d02);
+ sort_02v_descending(d03, d04);
+
+ tmp = d03;
+ d03 = _mm512_max_epi64(d02, d03);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ tmp = d04;
+ d04 = _mm512_max_epi64(d01, d04);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ sort_02v_descending(d01, d02);
+ sort_02v_ascending(d03, d04);
+
+ tmp = d03;
+ d03 = _mm512_max_epi64(d02, d03);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ tmp = d04;
+ d04 = _mm512_max_epi64(d01, d04);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d03, d01);
+ d03 = _mm512_max_epi64(d03, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d04, d02);
+ d04 = _mm512_max_epi64(d04, tmp);
+
+ sort_02v_merge_ascending(d01, d02);
+ sort_02v_merge_ascending(d03, d04);
+ }
+ static INLINE void sort_04v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d03, d01);
+ d03 = _mm512_max_epi64(d03, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d04, d02);
+ d04 = _mm512_max_epi64(d04, tmp);
+
+ sort_02v_merge_descending(d01, d02);
+ sort_02v_merge_descending(d03, d04);
+ }
+ static INLINE void sort_05v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_01v_descending(d05);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_01v_ascending(d05);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_05v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_01v_merge_ascending(d05);
+ }
+ static INLINE void sort_05v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_01v_merge_descending(d05);
+ }
+ static INLINE void sort_06v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_02v_descending(d05, d06);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi64(d03, d06);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_02v_ascending(d05, d06);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi64(d03, d06);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d06, d02);
+ d06 = _mm512_max_epi64(d06, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_02v_merge_ascending(d05, d06);
+ }
+ static INLINE void sort_06v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d06, d02);
+ d06 = _mm512_max_epi64(d06, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_02v_merge_descending(d05, d06);
+ }
+ static INLINE void sort_07v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_03v_descending(d05, d06, d07);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi64(d03, d06);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi64(d02, d07);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_03v_ascending(d05, d06, d07);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi64(d03, d06);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi64(d02, d07);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d06, d02);
+ d06 = _mm512_max_epi64(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi64(d07, d03);
+ d07 = _mm512_max_epi64(d07, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_03v_merge_ascending(d05, d06, d07);
+ }
+ static INLINE void sort_07v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d06, d02);
+ d06 = _mm512_max_epi64(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi64(d07, d03);
+ d07 = _mm512_max_epi64(d07, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_03v_merge_descending(d05, d06, d07);
+ }
+ static INLINE void sort_08v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ sort_04v_ascending(d01, d02, d03, d04);
+ sort_04v_descending(d05, d06, d07, d08);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi64(d03, d06);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi64(d02, d07);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ tmp = d08;
+ d08 = _mm512_max_epi64(d01, d08);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ sort_04v_descending(d01, d02, d03, d04);
+ sort_04v_ascending(d05, d06, d07, d08);
+
+ tmp = d05;
+ d05 = _mm512_max_epi64(d04, d05);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d06;
+ d06 = _mm512_max_epi64(d03, d06);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d07;
+ d07 = _mm512_max_epi64(d02, d07);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ tmp = d08;
+ d08 = _mm512_max_epi64(d01, d08);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d06, d02);
+ d06 = _mm512_max_epi64(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi64(d07, d03);
+ d07 = _mm512_max_epi64(d07, tmp);
+
+ tmp = d04;
+ d04 = _mm512_min_epi64(d08, d04);
+ d08 = _mm512_max_epi64(d08, tmp);
+
+ sort_04v_merge_ascending(d01, d02, d03, d04);
+ sort_04v_merge_ascending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_08v_merge_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08) {
+ __m512i tmp;
+
+ tmp = d01;
+ d01 = _mm512_min_epi64(d05, d01);
+ d05 = _mm512_max_epi64(d05, tmp);
+
+ tmp = d02;
+ d02 = _mm512_min_epi64(d06, d02);
+ d06 = _mm512_max_epi64(d06, tmp);
+
+ tmp = d03;
+ d03 = _mm512_min_epi64(d07, d03);
+ d07 = _mm512_max_epi64(d07, tmp);
+
+ tmp = d04;
+ d04 = _mm512_min_epi64(d08, d04);
+ d08 = _mm512_max_epi64(d08, tmp);
+
+ sort_04v_merge_descending(d01, d02, d03, d04);
+ sort_04v_merge_descending(d05, d06, d07, d08);
+ }
+ static INLINE void sort_09v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_descending(d09);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_ascending(d09);
+ }
+ static INLINE void sort_09v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_ascending(d09);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_01v_merge_descending(d09);
+ }
+ static INLINE void sort_10v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_descending(d09, d10);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_ascending(d09, d10);
+ }
+ static INLINE void sort_10v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_ascending(d09, d10);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_02v_merge_descending(d09, d10);
+ }
+ static INLINE void sort_11v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_descending(d09, d10, d11);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_ascending(d09, d10, d11);
+ }
+ static INLINE void sort_11v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_ascending(d09, d10, d11);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_03v_merge_descending(d09, d10, d11);
+ }
+ static INLINE void sort_12v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_descending(d09, d10, d11, d12);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_ascending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_12v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_ascending(d09, d10, d11, d12);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_04v_merge_descending(d09, d10, d11, d12);
+ }
+ static INLINE void sort_13v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_descending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_ascending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_13v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_ascending(d09, d10, d11, d12, d13);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_05v_merge_descending(d09, d10, d11, d12, d13);
+ }
+ static INLINE void sort_14v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_descending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi64(d03, d14);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_ascending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_14v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_ascending(d09, d10, d11, d12, d13, d14);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi64(d03, d14);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_06v_merge_descending(d09, d10, d11, d12, d13, d14);
+ }
+ static INLINE void sort_15v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_descending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi64(d03, d14);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi64(d02, d15);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_ascending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_15v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_ascending(d09, d10, d11, d12, d13, d14, d15);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi64(d03, d14);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi64(d02, d15);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_07v_merge_descending(d09, d10, d11, d12, d13, d14, d15);
+ }
+ static INLINE void sort_16v_ascending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15, __m512i& d16) {
+ __m512i tmp;
+
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi64(d03, d14);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi64(d02, d15);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ tmp = d16;
+ d16 = _mm512_max_epi64(d01, d16);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_08v_merge_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+ static INLINE void sort_16v_descending(__m512i& d01, __m512i& d02, __m512i& d03, __m512i& d04, __m512i& d05, __m512i& d06, __m512i& d07, __m512i& d08, __m512i& d09, __m512i& d10, __m512i& d11, __m512i& d12, __m512i& d13, __m512i& d14, __m512i& d15, __m512i& d16) {
+ __m512i tmp;
+
+ sort_08v_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_ascending(d09, d10, d11, d12, d13, d14, d15, d16);
+
+ tmp = d09;
+ d09 = _mm512_max_epi64(d08, d09);
+ d08 = _mm512_min_epi64(d08, tmp);
+
+ tmp = d10;
+ d10 = _mm512_max_epi64(d07, d10);
+ d07 = _mm512_min_epi64(d07, tmp);
+
+ tmp = d11;
+ d11 = _mm512_max_epi64(d06, d11);
+ d06 = _mm512_min_epi64(d06, tmp);
+
+ tmp = d12;
+ d12 = _mm512_max_epi64(d05, d12);
+ d05 = _mm512_min_epi64(d05, tmp);
+
+ tmp = d13;
+ d13 = _mm512_max_epi64(d04, d13);
+ d04 = _mm512_min_epi64(d04, tmp);
+
+ tmp = d14;
+ d14 = _mm512_max_epi64(d03, d14);
+ d03 = _mm512_min_epi64(d03, tmp);
+
+ tmp = d15;
+ d15 = _mm512_max_epi64(d02, d15);
+ d02 = _mm512_min_epi64(d02, tmp);
+
+ tmp = d16;
+ d16 = _mm512_max_epi64(d01, d16);
+ d01 = _mm512_min_epi64(d01, tmp);
+
+ sort_08v_merge_descending(d01, d02, d03, d04, d05, d06, d07, d08);
+ sort_08v_merge_descending(d09, d10, d11, d12, d13, d14, d15, d16);
+ }
+
+ static NOINLINE void sort_01v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ sort_01v_ascending(d01);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+}
+
+ static NOINLINE void sort_02v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ sort_02v_ascending(d01, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+}
+
+ static NOINLINE void sort_03v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ sort_03v_ascending(d01, d02, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+}
+
+ static NOINLINE void sort_04v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ sort_04v_ascending(d01, d02, d03, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+}
+
+ static NOINLINE void sort_05v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ sort_05v_ascending(d01, d02, d03, d04, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+}
+
+ static NOINLINE void sort_06v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ sort_06v_ascending(d01, d02, d03, d04, d05, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+}
+
+ static NOINLINE void sort_07v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ sort_07v_ascending(d01, d02, d03, d04, d05, d06, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+}
+
+ static NOINLINE void sort_08v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ sort_08v_ascending(d01, d02, d03, d04, d05, d06, d07, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+}
+
+ static NOINLINE void sort_09v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ sort_09v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+}
+
+ static NOINLINE void sort_10v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ sort_10v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+}
+
+ static NOINLINE void sort_11v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ sort_11v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+}
+
+ static NOINLINE void sort_12v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ sort_12v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+}
+
+ static NOINLINE void sort_13v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ sort_13v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+}
+
+ static NOINLINE void sort_14v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ __m512i d14 = _mm512_loadu_si512((__m512i const *) ptr + 13);;
+ sort_14v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 13, d14);
+}
+
+ static NOINLINE void sort_15v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ __m512i d14 = _mm512_loadu_si512((__m512i const *) ptr + 13);;
+ __m512i d15 = _mm512_loadu_si512((__m512i const *) ptr + 14);;
+ sort_15v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 13, d14);
+ _mm512_storeu_si512((__m512i *) ptr + 14, d15);
+}
+
+ static NOINLINE void sort_16v(int64_t *ptr) {
+ __m512i d01 = _mm512_loadu_si512((__m512i const *) ptr + 0);;
+ __m512i d02 = _mm512_loadu_si512((__m512i const *) ptr + 1);;
+ __m512i d03 = _mm512_loadu_si512((__m512i const *) ptr + 2);;
+ __m512i d04 = _mm512_loadu_si512((__m512i const *) ptr + 3);;
+ __m512i d05 = _mm512_loadu_si512((__m512i const *) ptr + 4);;
+ __m512i d06 = _mm512_loadu_si512((__m512i const *) ptr + 5);;
+ __m512i d07 = _mm512_loadu_si512((__m512i const *) ptr + 6);;
+ __m512i d08 = _mm512_loadu_si512((__m512i const *) ptr + 7);;
+ __m512i d09 = _mm512_loadu_si512((__m512i const *) ptr + 8);;
+ __m512i d10 = _mm512_loadu_si512((__m512i const *) ptr + 9);;
+ __m512i d11 = _mm512_loadu_si512((__m512i const *) ptr + 10);;
+ __m512i d12 = _mm512_loadu_si512((__m512i const *) ptr + 11);;
+ __m512i d13 = _mm512_loadu_si512((__m512i const *) ptr + 12);;
+ __m512i d14 = _mm512_loadu_si512((__m512i const *) ptr + 13);;
+ __m512i d15 = _mm512_loadu_si512((__m512i const *) ptr + 14);;
+ __m512i d16 = _mm512_loadu_si512((__m512i const *) ptr + 15);;
+ sort_16v_ascending(d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12, d13, d14, d15, d16);
+ _mm512_storeu_si512((__m512i *) ptr + 0, d01);
+ _mm512_storeu_si512((__m512i *) ptr + 1, d02);
+ _mm512_storeu_si512((__m512i *) ptr + 2, d03);
+ _mm512_storeu_si512((__m512i *) ptr + 3, d04);
+ _mm512_storeu_si512((__m512i *) ptr + 4, d05);
+ _mm512_storeu_si512((__m512i *) ptr + 5, d06);
+ _mm512_storeu_si512((__m512i *) ptr + 6, d07);
+ _mm512_storeu_si512((__m512i *) ptr + 7, d08);
+ _mm512_storeu_si512((__m512i *) ptr + 8, d09);
+ _mm512_storeu_si512((__m512i *) ptr + 9, d10);
+ _mm512_storeu_si512((__m512i *) ptr + 10, d11);
+ _mm512_storeu_si512((__m512i *) ptr + 11, d12);
+ _mm512_storeu_si512((__m512i *) ptr + 12, d13);
+ _mm512_storeu_si512((__m512i *) ptr + 13, d14);
+ _mm512_storeu_si512((__m512i *) ptr + 14, d15);
+ _mm512_storeu_si512((__m512i *) ptr + 15, d16);
+}
+ static void sort(int64_t *ptr, size_t length);
+
+};
+}
+}
+
+#undef i2d
+#undef d2i
+#undef i2s
+#undef s2i
+#undef s2d
+#undef d2s
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute pop
+#else
+#pragma GCC pop_options
+#endif
+#endif
+#endif
+
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#ifndef BITONIC_SORT_H
+#define BITONIC_SORT_H
+
+#include <stdint.h>
+#include "../defs.h"
+#include "../machine_traits.h"
+
+namespace vxsort {
+namespace smallsort {
+template <typename T, vector_machine M>
+struct bitonic {
+ public:
+ static void sort(T* ptr, size_t length);
+};
+} // namespace smallsort
+} // namespace gcsort
+#endif
--- /dev/null
+##
+## Licensed to the .NET Foundation under one or more agreements.
+## The .NET Foundation licenses this file to you under the MIT license.
+##
+
+import os
+from datetime import datetime
+
+from utils import native_size_map, next_power_of_2
+from bitonic_isa import BitonicISA
+
+
+class AVX2BitonicISA(BitonicISA):
+ def __init__(self, type):
+ self.vector_size_in_bytes = 32
+
+ self.type = type
+
+ self.bitonic_size_map = {}
+
+ for t, s in native_size_map.items():
+ self.bitonic_size_map[t] = int(self.vector_size_in_bytes / s)
+
+ self.bitonic_type_map = {
+ "int32_t": "__m256i",
+ "uint32_t": "__m256i",
+ "float": "__m256",
+ "int64_t": "__m256i",
+ "uint64_t": "__m256i",
+ "double": "__m256d",
+ }
+
+ def max_bitonic_sort_vectors(self):
+ return 16
+
+ def vector_size(self):
+ return self.bitonic_size_map[self.type]
+
+ def vector_type(self):
+ return self.bitonic_type_map[self.type]
+
+ @classmethod
+ def supported_types(cls):
+ return native_size_map.keys()
+
+ def i2d(self, v):
+ t = self.type
+ if t == "double":
+ return v
+ elif t == "float":
+ return f"s2d({v})"
+ return f"i2d({v})"
+
+ def i2s(self, v):
+ t = self.type
+ if t == "double":
+ raise Exception("WTF")
+ elif t == "float":
+ return f"i2s({v})"
+ return v
+
+ def d2i(self, v):
+ t = self.type
+ if t == "double":
+ return v
+ elif t == "float":
+ return f"d2s({v})"
+ return f"d2i({v})"
+
+ def s2i(self, v):
+ t = self.type
+ if t == "double":
+ raise Exception("WTF")
+ elif t == "float":
+ return f"s2i({v})"
+ return v
+
+ def generate_param_list(self, start, numParams):
+ return str.join(", ", list(map(lambda p: f"d{p:02d}", range(start, start + numParams))))
+
+ def generate_param_def_list(self, numParams):
+ t = self.type
+ return str.join(", ", list(map(lambda p: f"{self.vector_type()}& d{p:02d}", range(1, numParams + 1))))
+
+ def generate_shuffle_X1(self, v):
+ size = self.vector_size()
+ if size == 8:
+ return self.i2s(f"_mm256_shuffle_epi32({self.s2i(v)}, 0xB1)")
+ elif size == 4:
+ return self.d2i(f"_mm256_shuffle_pd({self.i2d(v)}, {self.i2d(v)}, 0x5)")
+
+ def generate_shuffle_X2(self, v):
+ size = self.vector_size()
+ if size == 8:
+ return self.i2s(f"_mm256_shuffle_epi32({self.s2i(v)}, 0x4E)")
+ elif size == 4:
+ return self.d2i(f"_mm256_permute4x64_pd({self.i2d(v)}, 0x4E)")
+
+ def generate_shuffle_XR(self, v):
+ size = self.vector_size()
+ if size == 8:
+ return self.i2s(f"_mm256_shuffle_epi32({self.s2i(v)}, 0x1B)")
+ elif size == 4:
+ return self.d2i(f"_mm256_permute4x64_pd({self.i2d(v)}, 0x1B)")
+
+ def generate_blend_B1(self, v1, v2, ascending):
+ size = self.vector_size()
+ if size == 8:
+ if ascending:
+ return self.i2s(f"_mm256_blend_epi32({self.s2i(v1)}, {self.s2i(v2)}, 0xAA)")
+ else:
+ return self.i2s(f"_mm256_blend_epi32({self.s2i(v2)}, {self.s2i(v1)}, 0xAA)")
+ elif size == 4:
+ if ascending:
+ return self.d2i(f"_mm256_blend_pd({self.i2d(v1)}, {self.i2d(v2)}, 0xA)")
+ else:
+ return self.d2i(f"_mm256_blend_pd({self.i2d(v2)}, {self.i2d(v1)}, 0xA)")
+
+ def generate_blend_B2(self, v1, v2, ascending):
+ size = self.vector_size()
+ if size == 8:
+ if ascending:
+ return self.i2s(f"_mm256_blend_epi32({self.s2i(v1)}, {self.s2i(v2)}, 0xCC)")
+ else:
+ return self.i2s(f"_mm256_blend_epi32({self.s2i(v2)}, {self.s2i(v1)}, 0xCC)")
+ elif size == 4:
+ if ascending:
+ return self.d2i(f"_mm256_blend_pd({self.i2d(v1)}, {self.i2d(v2)}, 0xC)")
+ else:
+ return self.d2i(f"_mm256_blend_pd({self.i2d(v2)}, {self.i2d(v1)}, 0xC)")
+
+ def generate_blend_B4(self, v1, v2, ascending):
+ size = self.vector_size()
+ if size == 8:
+ if ascending:
+ return self.i2s(f"_mm256_blend_epi32({self.s2i(v1)}, {self.s2i(v2)}, 0xF0)")
+ else:
+ return self.i2s(f"_mm256_blend_epi32({self.s2i(v2)}, {self.s2i(v1)}, 0xF0)")
+ elif size == 4:
+ raise Exception("WTF")
+
+ def generate_cross(self, v):
+ size = self.vector_size()
+ if size == 8:
+ return self.d2i(f"_mm256_permute4x64_pd({self.i2d(v)}, 0x4E)")
+ elif size == 4:
+ raise Exception("WTF")
+
+ def generate_reverse(self, v):
+ size = self.vector_size()
+ if size == 8:
+ v = f"_mm256_shuffle_epi32({self.s2i(v)}, 0x1B)"
+ return self.d2i(f"_mm256_permute4x64_pd(i2d({v}), 0x4E)")
+ elif size == 4:
+ return self.d2i(f"_mm256_permute4x64_pd({self.i2d(v)}, 0x1B)")
+
+ def crappity_crap_crap(self, v1, v2):
+ t = self.type
+ if t == "int64_t":
+ return f"cmp = _mm256_cmpgt_epi64({v1}, {v2});"
+ elif t == "uint64_t":
+ return f"cmp = _mm256_cmpgt_epi64(_mm256_xor_si256(topBit, {v1}), _mm256_xor_si256(topBit, {v2}));"
+
+ return ""
+
+ def generate_min(self, v1, v2):
+ t = self.type
+ if t == "int32_t":
+ return f"_mm256_min_epi32({v1}, {v2})"
+ elif t == "uint32_t":
+ return f"_mm256_min_epu32({v1}, {v2})"
+ elif t == "float":
+ return f"_mm256_min_ps({v1}, {v2})"
+ elif t == "int64_t":
+ return self.d2i(f"_mm256_blendv_pd({self.i2d(v1)}, {self.i2d(v2)}, i2d(cmp))")
+ elif t == "uint64_t":
+ return self.d2i(f"_mm256_blendv_pd({self.i2d(v1)}, {self.i2d(v2)}, i2d(cmp))")
+ elif t == "double":
+ return f"_mm256_min_pd({v1}, {v2})"
+
+ def generate_max(self, v1, v2):
+ t = self.type
+ if t == "int32_t":
+ return f"_mm256_max_epi32({v1}, {v2})"
+ elif t == "uint32_t":
+ return f"_mm256_max_epu32({v1}, {v2})"
+ elif t == "float":
+ return f"_mm256_max_ps({v1}, {v2})"
+ elif t == "int64_t":
+ return self.d2i(f"_mm256_blendv_pd({self.i2d(v2)}, {self.i2d(v1)}, i2d(cmp))")
+ elif t == "uint64_t":
+ return self.d2i(f"_mm256_blendv_pd({self.i2d(v2)}, {self.i2d(v1)}, i2d(cmp))")
+ elif t == "double":
+ return f"_mm256_max_pd({v1}, {v2})"
+
+ def get_load_intrinsic(self, v, offset):
+ t = self.type
+ if t == "double":
+ return f"_mm256_loadu_pd(({t} const *) ((__m256d const *) {v} + {offset}))"
+ if t == "float":
+ return f"_mm256_loadu_ps(({t} const *) ((__m256 const *) {v} + {offset}))"
+ return f"_mm256_lddqu_si256((__m256i const *) {v} + {offset});"
+
+
+ def get_store_intrinsic(self, ptr, offset, value):
+ t = self.type
+ if t == "double":
+ return f"_mm256_storeu_pd(({t} *) ((__m256d *) {ptr} + {offset}), {value})"
+ if t == "float":
+ return f"_mm256_storeu_ps(({t} *) ((__m256 *) {ptr} + {offset}), {value})"
+ return f"_mm256_storeu_si256((__m256i *) {ptr} + {offset}, {value})"
+
+ def autogenerated_blabber(self):
+ return f"""/////////////////////////////////////////////////////////////////////////////
+////
+// This file was auto-generated by a tool at {datetime.now().strftime("%F %H:%M:%S")}
+//
+// It is recommended you DO NOT directly edit this file but instead edit
+// the code-generator that generated this source file instead.
+/////////////////////////////////////////////////////////////////////////////"""
+
+ def generate_prologue(self, f):
+ t = self.type
+ s = f"""{self.autogenerated_blabber()}
+
+#ifndef BITONIC_SORT_AVX2_{t.upper()}_H
+#define BITONIC_SORT_AVX2_{t.upper()}_H
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx2"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx2")
+#endif
+#endif
+
+#include <immintrin.h>
+#include "bitonic_sort.h"
+
+#define i2d _mm256_castsi256_pd
+#define d2i _mm256_castpd_si256
+#define i2s _mm256_castsi256_ps
+#define s2i _mm256_castps_si256
+#define s2d _mm256_castps_pd
+#define d2s _mm256_castpd_ps
+
+namespace vxsort {{
+namespace smallsort {{
+template<> struct bitonic<{t}, AVX2> {{
+public:
+"""
+ print(s, file=f)
+
+ def generate_epilogue(self, f):
+ s = f"""
+}};
+}}
+}}
+
+#undef i2d
+#undef d2i
+#undef i2s
+#undef s2i
+#undef s2d
+#undef d2s
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute pop
+#else
+#pragma GCC pop_options
+#endif
+#endif
+#endif
+ """
+ print(s, file=f)
+
+ def generate_1v_basic_sorters(self, f, ascending):
+ g = self
+ type = self.type
+ maybe_cmp = lambda: ", cmp" if (type == "int64_t" or type == "uint64_t") else ""
+ maybe_topbit = lambda: f"\n {g.vector_type()} topBit = _mm256_set1_epi64x(1LLU << 63);" if (type == "uint64_t") else ""
+ suffix = "ascending" if ascending else "descending"
+
+ s = f""" static INLINE void sort_01v_{suffix}({g.generate_param_def_list(1)}) {{
+ {g.vector_type()} min, max, s{maybe_cmp()};{maybe_topbit()}
+
+ s = {g.generate_shuffle_X1("d01")};
+ {g.crappity_crap_crap("s", "d01")}
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B1("min", "max", ascending)};
+
+ s = {g.generate_shuffle_XR("d01")};
+ {g.crappity_crap_crap("s", "d01")}
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B2("min", "max", ascending)};
+
+ s = {g.generate_shuffle_X1("d01")};
+ {g.crappity_crap_crap("s", "d01")}
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B1("min", "max", ascending)};"""
+
+ print(s, file=f)
+
+ if g.vector_size() == 8:
+ s = f"""
+ s = {g.generate_reverse("d01")};
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B4("min", "max", ascending)};
+
+ s = {g.generate_shuffle_X2("d01")};
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B2("min", "max", ascending)};
+
+ s = {g.generate_shuffle_X1("d01")};
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B1("min", "max", ascending)};"""
+ print(s, file=f)
+ print("}", file=f)
+
+
+
+ def generate_1v_merge_sorters(self, f, ascending: bool):
+ g = self
+ type = self.type
+ maybe_cmp = lambda: ", cmp" if (type == "int64_t" or type == "uint64_t") else ""
+ maybe_topbit = lambda: f"\n {g.vector_type()} topBit = _mm256_set1_epi64x(1LLU << 63);" if (
+ type == "uint64_t") else ""
+
+ suffix = "ascending" if ascending else "descending"
+
+ s = f""" static INLINE void sort_01v_merge_{suffix}({g.generate_param_def_list(1)}) {{
+ {g.vector_type()} min, max, s{maybe_cmp()};{maybe_topbit()}"""
+ print(s, file=f)
+
+ if g.vector_size() == 8:
+ s = f"""
+ s = {g.generate_cross("d01")};
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B4("min", "max", ascending)};"""
+ print(s, file=f)
+
+ s = f"""
+ s = {g.generate_shuffle_X2("d01")};
+ {g.crappity_crap_crap("s", "d01")}
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B2("min", "max", ascending)};
+
+ s = {g.generate_shuffle_X1("d01")};
+ {g.crappity_crap_crap("s", "d01")}
+ min = {g.generate_min("s", "d01")};
+ max = {g.generate_max("s", "d01")};
+ d01 = {g.generate_blend_B1("min", "max", ascending)};"""
+
+ print(s, file=f)
+ print(" }", file=f)
+
+ def generate_compounded_sorter(self, f, width, ascending, inline):
+ type = self.type
+ g = self
+ maybe_cmp = lambda: ", cmp" if (type == "int64_t" or type == "uint64_t") else ""
+ maybe_topbit = lambda: f"\n {g.vector_type()} topBit = _mm256_set1_epi64x(1LLU << 63);" if (
+ type == "uint64_t") else ""
+
+ w1 = int(next_power_of_2(width) / 2)
+ w2 = int(width - w1)
+
+ suffix = "ascending" if ascending else "descending"
+ rev_suffix = "descending" if ascending else "ascending"
+
+ inl = "INLINE" if inline else "NOINLINE"
+
+ s = f""" static {inl} void sort_{width:02d}v_{suffix}({g.generate_param_def_list(width)}) {{
+ {g.vector_type()} tmp{maybe_cmp()};{maybe_topbit()}
+
+ sort_{w1:02d}v_{suffix}({g.generate_param_list(1, w1)});
+ sort_{w2:02d}v_{rev_suffix}({g.generate_param_list(w1 + 1, w2)});"""
+
+ print(s, file=f)
+
+ for r in range(w1 + 1, width + 1):
+ x = w1 + 1 - (r - w1)
+ s = f"""
+ tmp = d{r:02d};
+ {g.crappity_crap_crap(f"d{x:02d}", f"d{r:02d}")}
+ d{r:02d} = {g.generate_max(f"d{x:02d}", f"d{r:02d}")};
+ d{x:02d} = {g.generate_min(f"d{x:02d}", "tmp")};"""
+
+ print(s, file=f)
+
+ s = f"""
+ sort_{w1:02d}v_merge_{suffix}({g.generate_param_list(1, w1)});
+ sort_{w2:02d}v_merge_{suffix}({g.generate_param_list(w1 + 1, w2)});"""
+ print(s, file=f)
+ print(" }", file=f)
+
+
+ def generate_compounded_merger(self, f, width, ascending, inline):
+ type = self.type
+ g = self
+ maybe_cmp = lambda: ", cmp" if (type == "int64_t" or type == "uint64_t") else ""
+ maybe_topbit = lambda: f"\n {g.vector_type()} topBit = _mm256_set1_epi64x(1LLU << 63);" if (
+ type == "uint64_t") else ""
+
+ w1 = int(next_power_of_2(width) / 2)
+ w2 = int(width - w1)
+
+ suffix = "ascending" if ascending else "descending"
+ rev_suffix = "descending" if ascending else "ascending"
+
+ inl = "INLINE" if inline else "NOINLINE"
+
+ s = f""" static {inl} void sort_{width:02d}v_merge_{suffix}({g.generate_param_def_list(width)}) {{
+ {g.vector_type()} tmp{maybe_cmp()};{maybe_topbit()}"""
+ print(s, file=f)
+
+ for r in range(w1 + 1, width + 1):
+ x = r - w1
+ s = f"""
+ tmp = d{x:02d};
+ {g.crappity_crap_crap(f"d{r:02d}", f"d{x:02d}")}
+ d{x:02d} = {g.generate_min(f"d{r:02d}", f"d{x:02d}")};
+ {g.crappity_crap_crap(f"d{r:02d}", "tmp")}
+ d{r:02d} = {g.generate_max(f"d{r:02d}", "tmp")};"""
+ print(s, file=f)
+
+ s = f"""
+ sort_{w1:02d}v_merge_{suffix}({g.generate_param_list(1, w1)});
+ sort_{w2:02d}v_merge_{suffix}({g.generate_param_list(w1 + 1, w2)});"""
+ print(s, file=f)
+ print(" }", file=f)
+
+
+ def generate_entry_points(self, f):
+ type = self.type
+ g = self
+ for m in range(1, g.max_bitonic_sort_vectors() + 1):
+ s = f"""
+ static NOINLINE void sort_{m:02d}v({type} *ptr) {{"""
+ print(s, file=f)
+
+ for l in range(0, m):
+ s = f" {g.vector_type()} d{l + 1:02d} = {g.get_load_intrinsic('ptr', l)};"
+ print(s, file=f)
+
+ s = f" sort_{m:02d}v_ascending({g.generate_param_list(1, m)});"
+ print(s, file=f)
+
+ for l in range(0, m):
+ s = f" {g.get_store_intrinsic('ptr', l, f'd{l + 1:02d}')};"
+ print(s, file=f)
+
+ print("}", file=f)
+
+
+ def generate_master_entry_point(self, f_header, f_src):
+ basename = os.path.basename(f_header.name)
+ s = f"""#include "{basename}"
+
+using namespace vxsort;
+"""
+ print(s, file=f_src)
+
+ t = self.type
+ g = self
+
+ s = f""" static void sort({t} *ptr, size_t length);"""
+ print(s, file=f_header)
+
+ s = f"""void vxsort::smallsort::bitonic<{t}, vector_machine::AVX2 >::sort({t} *ptr, size_t length) {{
+ const int N = {g.vector_size()};
+
+ switch(length / N) {{"""
+ print(s, file=f_src)
+
+ for m in range(1, self.max_bitonic_sort_vectors() + 1):
+ s = f" case {m}: sort_{m:02d}v(ptr); break;"
+ print(s, file=f_src)
+ print(" }", file=f_src)
+ print("}", file=f_src)
+ pass
--- /dev/null
+##
+## Licensed to the .NET Foundation under one or more agreements.
+## The .NET Foundation licenses this file to you under the MIT license.
+##
+
+from datetime import datetime
+
+from utils import native_size_map, next_power_of_2
+from bitonic_isa import BitonicISA
+import os
+
+
+class AVX512BitonicISA(BitonicISA):
+ def __init__(self, type):
+ self.vector_size_in_bytes = 64
+
+ self.type = type
+
+ self.bitonic_size_map = {}
+
+ for t, s in native_size_map.items():
+ self.bitonic_size_map[t] = int(self.vector_size_in_bytes / s)
+
+ self.bitonic_type_map = {
+ "int32_t": "__m512i",
+ "uint32_t": "__m512i",
+ "float": "__m512",
+ "int64_t": "__m512i",
+ "uint64_t": "__m512i",
+ "double": "__m512d",
+ }
+
+ def max_bitonic_sort_vectors(self):
+ return 16
+
+ def vector_size(self):
+ return self.bitonic_size_map[self.type]
+
+ def vector_type(self):
+ return self.bitonic_type_map[self.type]
+
+ @classmethod
+ def supported_types(cls):
+ return native_size_map.keys()
+
+ def i2d(self, v):
+ t = self.type
+ if t == "double":
+ return v
+ elif t == "float":
+ raise Exception("WTF")
+ return f"i2d({v})"
+
+ def i2s(self, v):
+ t = self.type
+ if t == "double":
+ raise Exception("WTF")
+ elif t == "float":
+ return f"i2s({v})"
+ return v
+
+ def d2i(self, v):
+ t = self.type
+ if t == "double":
+ return v
+ elif t == "float":
+ raise Exception("WTF")
+ return f"d2i({v})"
+
+ def s2i(self, v):
+ t = self.type
+ if t == "double":
+ raise Exception("WTF")
+ elif t == "float":
+ return f"s2i({v})"
+ return v
+
+ def generate_param_list(self, start, numParams):
+ return str.join(", ", list(map(lambda p: f"d{p:02d}", range(start, start + numParams))))
+
+ def generate_param_def_list(self, numParams):
+ t = self.type
+ return str.join(", ", list(map(lambda p: f"{self.vector_type()}& d{p:02d}", range(1, numParams + 1))))
+
+ def generate_shuffle_S1(self, v):
+ t = self.type
+ size = self.bitonic_size_map[t]
+ if size == 16:
+ return self.i2s(f"_mm512_shuffle_epi32({self.s2i(v)}, _MM_PERM_CDAB)")
+ elif size == 8:
+ return self.d2i(f"_mm512_permute_pd({self.i2d(v)}, _MM_PERM_BBBB)")
+
+ def generate_shuffle_X4(self, v):
+ t = self.type
+ size = self.bitonic_size_map[t]
+ if size == 16:
+ return self.i2s(f"_mm512_shuffle_epi32({self.s2i(v)}, _MM_PERM_ABCD)")
+ elif size == 8:
+ return self.d2i(f"_mm512_permutex_pd({self.i2d(v)}, _MM_PERM_ABCD)")
+
+ def generate_shuffle_X8(self, v):
+ t = self.type
+ size = self.bitonic_size_map[t]
+ if size == 16:
+ s1 = f"_mm512_shuffle_epi32({self.s2i(v)}, _MM_PERM_ABCD)"
+ return self.i2s(f"_mm512_permutex_epi64({s1}, _MM_PERM_BADC)")
+ elif size == 8:
+ s1 = f"_mm512_permutex_pd({self.i2d(v)}, _MM_PERM_ABCD)"
+ return self.d2i(f"_mm512_shuffle_f64x2({s1}, {s1}, _MM_PERM_BADC)")
+
+ def generate_shuffle_S2(self, v):
+ t = self.type
+ size = self.bitonic_size_map[t]
+ if size == 16:
+ return self.i2s(f"_mm512_shuffle_epi32({self.s2i(v)}, _MM_PERM_BADC)")
+ elif size == 8:
+ return self.d2i(f"_mm512_permutex_pd({self.i2d(v)}, _MM_PERM_BADC)")
+
+ def generate_shuffle_X16(self, v):
+ t = self.type
+ size = self.bitonic_size_map[t]
+ if size == 16:
+ s1 = f"_mm512_shuffle_epi32({self.s2i(v)}, _MM_PERM_ABCD)"
+ return self.i2s(f"_mm512_shuffle_i64x2({s1}, {s1}, _MM_PERM_ABCD)")
+ elif size == 8:
+ return self.d2i(f"_mm512_shuffle_pd({self.i2d(v)}, {self.i2d(v)}, 0xB1)")
+
+ def generate_shuffle_S4(self, v):
+ t = self.type
+ size = self.bitonic_size_map[t]
+ if size == 16:
+ return self.i2s(f"_mm512_permutex_epi64({self.s2i(v)}, _MM_PERM_BADC)")
+ elif size == 8:
+ return self.d2i(f"_mm512_shuffle_f64x2({self.i2d(v)}, {self.i2d(v)}, _MM_PERM_BADC)")
+
+ def generate_shuffle_S8(self, v):
+ t = self.type
+ size = self.bitonic_size_map[t]
+ if size == 16:
+ return self.i2s(f"_mm512_shuffle_i64x2({self.s2i(v)}, {self.s2i(v)}, _MM_PERM_BADC)")
+ elif size == 8:
+ return self.d2i(f"_mm512_shuffle_pd({self.i2d(v)}, {self.i2d(v)}, 0xB1)")
+
+ def generate_min(self, v1, v2):
+ t = self.type
+ if t == "int32_t":
+ return f"_mm512_min_epi32({v1}, {v2})"
+ elif t == "uint32_t":
+ return f"_mm512_min_epu32({v1}, {v2})"
+ elif t == "float":
+ return f"_mm512_min_ps({v1}, {v2})"
+ elif t == "int64_t":
+ return f"_mm512_min_epi64({v1}, {v2})"
+ elif t == "uint64_t":
+ return f"_mm512_min_epu64({v1}, {v2})"
+ elif t == "double":
+ return f"_mm512_min_pd({v1}, {v2})"
+
+ def generate_max(self, v1, v2):
+ t = self.type
+ if t == "int32_t":
+ return f"_mm512_max_epi32({v1}, {v2})"
+ elif t == "uint32_t":
+ return f"_mm512_max_epu32({v1}, {v2})"
+ elif t == "float":
+ return f"_mm512_max_ps({v1}, {v2})"
+ elif t == "int64_t":
+ return f"_mm512_max_epi64({v1}, {v2})"
+ elif t == "uint64_t":
+ return f"_mm512_max_epu64({v1}, {v2})"
+ elif t == "double":
+ return f"_mm512_max_pd({v1}, {v2})"
+
+ def generate_mask(self, stride, ascending):
+ b = 1 << stride
+ b = b - 1
+ if ascending:
+ b = b << stride
+
+ mask = 0
+ size = self.vector_size()
+ while size > 0:
+ mask = mask << (stride * 2) | b
+ size = size - (stride * 2)
+ return mask
+
+
+ def generate_max_with_blend(self, src, v1, v2, stride, ascending):
+ mask = self.generate_mask(stride, ascending)
+ t = self.type
+ if t == "int32_t":
+ return f"_mm512_mask_max_epi32({src}, 0x{mask:04X}, {v1}, {v2})"
+ elif t == "uint32_t":
+ return f"_mm512_mask_max_epu32({src}, 0x{mask:04X}, {v1}, {v2})"
+ elif t == "float":
+ return f"_mm512_mask_max_ps({src}, 0x{mask:04X}, {v1}, {v2})"
+ elif t == "int64_t":
+ return f"_mm512_mask_max_epi64({src}, 0x{mask:04X}, {v1}, {v2})"
+ elif t == "uint64_t":
+ return f"_mm512_mask_max_epu64({src}, 0x{mask:04X}, {v1}, {v2})"
+ elif t == "double":
+ return f"_mm512_mask_max_pd({src}, 0x{mask:04X}, {v1}, {v2})"
+
+
+ def get_load_intrinsic(self, v, offset):
+ t = self.type
+ if t == "double":
+ return f"_mm512_loadu_pd(({t} const *) ((__m512d const *) {v} + {offset}))"
+ if t == "float":
+ return f"_mm512_loadu_ps(({t} const *) ((__m512 const *) {v} + {offset}))"
+ return f"_mm512_loadu_si512((__m512i const *) {v} + {offset});"
+
+
+ def get_store_intrinsic(self, ptr, offset, value):
+ t = self.type
+ if t == "double":
+ return f"_mm512_storeu_pd(({t} *) ((__m512d *) {ptr} + {offset}), {value})"
+ if t == "float":
+ return f"_mm512_storeu_ps(({t} *) ((__m512 *) {ptr} + {offset}), {value})"
+ return f"_mm512_storeu_si512((__m512i *) {ptr} + {offset}, {value})"
+
+ def autogenerated_blabber(self):
+ return f"""/////////////////////////////////////////////////////////////////////////////
+////
+// This file was auto-generated by a tool at {datetime.now().strftime("%F %H:%M:%S")}
+//
+// It is recommended you DO NOT directly edit this file but instead edit
+// the code-generator that generated this source file instead.
+/////////////////////////////////////////////////////////////////////////////"""
+
+ def generate_prologue(self, f):
+ t = self.type
+ s = f"""{self.autogenerated_blabber()}
+
+#ifndef BITONIC_SORT_AVX512_{t.upper()}_H
+#define BITONIC_SORT_AVX512_{t.upper()}_H
+
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx512f"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx512f")
+#endif
+#endif
+
+#include <immintrin.h>
+#include "bitonic_sort.h"
+
+#define i2d _mm512_castsi512_pd
+#define d2i _mm512_castpd_si512
+#define i2s _mm512_castsi512_ps
+#define s2i _mm512_castps_si512
+#define s2d _mm512_castps_pd
+#define d2s _mm521_castpd_ps
+
+namespace vxsort {{
+namespace smallsort {{
+template<> struct bitonic<{t}, AVX512> {{
+public:
+"""
+ print(s, file=f)
+
+ def generate_epilogue(self, f):
+ s = f"""
+}};
+}}
+}}
+
+#undef i2d
+#undef d2i
+#undef i2s
+#undef s2i
+#undef s2d
+#undef d2s
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute pop
+#else
+#pragma GCC pop_options
+#endif
+#endif
+#endif
+"""
+ print(s, file=f)
+
+ def generate_1v_basic_sorters(self, f, ascending):
+ g = self
+ type = self.type
+ suffix = "ascending" if ascending else "descending"
+
+ s = f""" static INLINE void sort_01v_{suffix}({g.generate_param_def_list(1)}) {{
+ {g.vector_type()} min, s;
+
+ s = {g.generate_shuffle_S1("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 1, ascending)};
+
+ s = {g.generate_shuffle_X4("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 2, ascending)};
+
+ s = {g.generate_shuffle_S1("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 1, ascending)};
+
+ s = {g.generate_shuffle_X8("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 4, ascending)};
+
+ s = {g.generate_shuffle_S2("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 2, ascending)};
+
+ s = {g.generate_shuffle_S1("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 1, ascending)};"""
+
+ print(s, file=f)
+
+ if g.vector_size() == 16:
+ s = f"""
+ s = {g.generate_shuffle_X16("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 8, ascending)};
+
+ s = {g.generate_shuffle_S4("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 4, ascending)};
+
+ s = {g.generate_shuffle_S2("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 2, ascending)};
+
+ s = {g.generate_shuffle_S1("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 1, ascending)};"""
+ print(s, file=f)
+ print(" }", file=f)
+
+ def generate_1v_merge_sorters(self, f, ascending: bool):
+ g = self
+ type = self.type
+ suffix = "ascending" if ascending else "descending"
+
+ s = f""" static INLINE void sort_01v_merge_{suffix}({g.generate_param_def_list(1)}) {{
+ {g.vector_type()} min, s;"""
+ print(s, file=f)
+
+ if g.vector_size() == 16:
+ s = f"""
+ s = {g.generate_shuffle_S8("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 8, ascending)};"""
+ print(s, file=f)
+
+ s = f"""
+ s = {g.generate_shuffle_S4("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 4, ascending)};
+
+ s = {g.generate_shuffle_S2("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 2, ascending)};
+
+ s = {g.generate_shuffle_S1("d01")};
+ min = {g.generate_min("s", "d01")};
+ d01 = {g.generate_max_with_blend("min", "s", "d01", 1, ascending)};"""
+
+ print(s, file=f)
+ print(" }", file=f)
+
+
+ def generate_compounded_sorter(self, f, width, ascending, inline):
+ type = self.type
+ g = self
+
+ w1 = int(next_power_of_2(width) / 2)
+ w2 = int(width - w1)
+
+ suffix = "ascending" if ascending else "descending"
+ rev_suffix = "descending" if ascending else "ascending"
+
+ inl = "INLINE" if inline else "NOINLINE"
+
+ s = f""" static {inl} void sort_{width:02d}v_{suffix}({g.generate_param_def_list(width)}) {{
+ {g.vector_type()} tmp;
+
+ sort_{w1:02d}v_{suffix}({g.generate_param_list(1, w1)});
+ sort_{w2:02d}v_{rev_suffix}({g.generate_param_list(w1 + 1, w2)});"""
+
+ print(s, file=f)
+
+ for r in range(w1 + 1, width + 1):
+ x = w1 + 1 - (r - w1)
+ s = f"""
+ tmp = d{r:02d};
+ d{r:02d} = {g.generate_max(f"d{x:02d}", f"d{r:02d}")};
+ d{x:02d} = {g.generate_min(f"d{x:02d}", "tmp")};"""
+
+ print(s, file=f)
+
+ s = f"""
+ sort_{w1:02d}v_merge_{suffix}({g.generate_param_list(1, w1)});
+ sort_{w2:02d}v_merge_{suffix}({g.generate_param_list(w1 + 1, w2)});"""
+ print(s, file=f)
+ print(" }", file=f)
+
+
+ def generate_compounded_merger(self, f, width, ascending, inline):
+ type = self.type
+ g = self
+
+ w1 = int(next_power_of_2(width) / 2)
+ w2 = int(width - w1)
+
+ suffix = "ascending" if ascending else "descending"
+ rev_suffix = "descending" if ascending else "ascending"
+
+ inl = "INLINE" if inline else "NOINLINE"
+
+ s = f""" static {inl} void sort_{width:02d}v_merge_{suffix}({g.generate_param_def_list(width)}) {{
+ {g.vector_type()} tmp;"""
+ print(s, file=f)
+
+ for r in range(w1 + 1, width + 1):
+ x = r - w1
+ s = f"""
+ tmp = d{x:02d};
+ d{x:02d} = {g.generate_min(f"d{r:02d}", f"d{x:02d}")};
+ d{r:02d} = {g.generate_max(f"d{r:02d}", "tmp")};"""
+ print(s, file=f)
+
+ s = f"""
+ sort_{w1:02d}v_merge_{suffix}({g.generate_param_list(1, w1)});
+ sort_{w2:02d}v_merge_{suffix}({g.generate_param_list(w1 + 1, w2)});"""
+ print(s, file=f)
+ print(" }", file=f)
+
+
+ def generate_entry_points(self, f):
+ type = self.type
+ g = self
+ for m in range(1, g.max_bitonic_sort_vectors() + 1):
+ s = f"""
+ static NOINLINE void sort_{m:02d}v({type} *ptr) {{"""
+ print(s, file=f)
+
+ for l in range(0, m):
+ s = f" {g.vector_type()} d{l + 1:02d} = {g.get_load_intrinsic('ptr', l)};"
+ print(s, file=f)
+
+ s = f" sort_{m:02d}v_ascending({g.generate_param_list(1, m)});"
+ print(s, file=f)
+
+ for l in range(0, m):
+ s = f" {g.get_store_intrinsic('ptr', l, f'd{l + 1:02d}')};"
+ print(s, file=f)
+
+ print("}", file=f)
+
+
+ def generate_master_entry_point(self, f_header, f_src):
+ basename = os.path.basename(f_header.name)
+ s = f"""#include "{basename}"
+
+using namespace vxsort;
+"""
+ print(s, file=f_src)
+
+ t = self.type
+ g = self
+
+ s = f""" static void sort({t} *ptr, size_t length);"""
+ print(s, file=f_header)
+
+ s = f"""void vxsort::smallsort::bitonic<{t}, vector_machine::AVX512 >::sort({t} *ptr, size_t length) {{
+ const int N = {g.vector_size()};
+
+ switch(length / N) {{"""
+ print(s, file=f_src)
+
+ for m in range(1, self.max_bitonic_sort_vectors() + 1):
+ s = f" case {m}: sort_{m:02d}v(ptr); break;"
+ print(s, file=f_src)
+ print(" }", file=f_src)
+ print("}", file=f_src)
+ pass
--- /dev/null
+##
+## Licensed to the .NET Foundation under one or more agreements.
+## The .NET Foundation licenses this file to you under the MIT license.
+##
+
+#!/usr/bin/env python3
+#
+# This is a tool to generate the bitonic sorter code that is used for small arrays.
+#
+# usage: bitonic_gen.py [-h] [--vector-isa VECTOR_ISA [VECTOR_ISA ...]]
+# [--break-inline BREAK_INLINE] [--output-dir OUTPUT_DIR]
+#
+# the files in src/coreclr/src/gc/vxsort/smallsort checked in can be generated with:
+# python bitonic_gen.py --output-dir c:\temp --vector-isa AVX2 AVX512
+#
+import argparse
+import os
+from enum import Enum
+
+from avx2 import AVX2BitonicISA
+from avx512 import AVX512BitonicISA
+from bitonic_isa import BitonicISA
+
+BitonicISA.register(AVX2BitonicISA)
+BitonicISA.register(AVX512BitonicISA)
+
+
+def get_generator_supported_types(vector_isa):
+ if isinstance(vector_isa, str):
+ vector_isa = VectorISA[vector_isa]
+ if vector_isa == VectorISA.AVX2:
+ return AVX2BitonicISA.supported_types()
+ elif vector_isa == VectorISA.AVX512:
+ return AVX512BitonicISA.supported_types()
+ else:
+ raise Exception(f"Non-supported vector machine-type: {vector_isa}")
+
+
+def get_generator(vector_isa, type):
+ if isinstance(vector_isa, str):
+ vector_isa = VectorISA[vector_isa]
+ if vector_isa == VectorISA.AVX2:
+ return AVX2BitonicISA(type)
+ elif vector_isa == VectorISA.AVX512:
+ return AVX512BitonicISA(type)
+ else:
+ raise Exception(f"Non-supported vector machine-type: {vector_isa}")
+
+
+def generate_per_type(f_header, f_src, type, vector_isa, break_inline):
+ g = get_generator(vector_isa, type)
+ g.generate_prologue(f_header)
+ g.generate_1v_sorters(f_header, ascending=True)
+ g.generate_1v_sorters(f_header, ascending=False)
+ for width in range(2, g.max_bitonic_sort_vectors() + 1):
+
+ # Allow breaking the inline chain once in a while (configurable)
+ if break_inline == 0 or width & break_inline != 0:
+ inline = True
+ else:
+ inline = False
+ g.generate_compounded_sorter(f_header, width, ascending=True, inline=inline)
+ g.generate_compounded_sorter(f_header, width, ascending=False, inline=inline)
+ if width <= g.largest_merge_variant_needed():
+ g.generate_compounded_merger(f_header, width, ascending=True, inline=inline)
+ g.generate_compounded_merger(f_header, width, ascending=False, inline=inline)
+
+ g.generate_entry_points(f_header)
+ g.generate_master_entry_point(f_header, f_src)
+ g.generate_epilogue(f_header)
+
+
+class Language(Enum):
+ csharp = 'csharp'
+ cpp = 'cpp'
+ rust = 'rust'
+
+ def __str__(self):
+ return self.value
+
+
+class VectorISA(Enum):
+ AVX2 = 'AVX2'
+ AVX512 = 'AVX512'
+ SVE = 'SVE'
+
+ def __str__(self):
+ return self.value
+
+def generate_all_types():
+ parser = argparse.ArgumentParser()
+ #parser.add_argument("--language", type=Language, choices=list(Language),
+ # help="select output language: csharp/cpp/rust")
+ parser.add_argument("--vector-isa",
+ nargs='+',
+ default='all',
+ help='list of vector ISA to generate',
+ choices=list(VectorISA).append("all"))
+ parser.add_argument("--break-inline", type=int, default=0, help="break inlining every N levels")
+
+ parser.add_argument("--output-dir", type=str,
+ help="output directory")
+
+ opts = parser.parse_args()
+
+ if 'all' in opts.vector_isa:
+ opts.vector_isa = list(VectorISA)
+
+ for isa in opts.vector_isa:
+ for t in get_generator_supported_types(isa):
+ filename = f"bitonic_sort.{isa}.{t}.generated"
+ print(f"Generating {filename}.{{h,.cpp}}")
+ h_filename = os.path.join(opts.output_dir, filename + ".h")
+ h_src = os.path.join(opts.output_dir, filename + ".cpp")
+ with open(h_filename, "w") as f_header, open(h_src, "w") as f_source:
+ generate_per_type(f_header, f_source, t, isa, opts.break_inline)
+
+if __name__ == '__main__':
+ generate_all_types()
--- /dev/null
+##
+## Licensed to the .NET Foundation under one or more agreements.
+## The .NET Foundation licenses this file to you under the MIT license.
+##
+
+from abc import ABC, ABCMeta, abstractmethod
+
+from utils import next_power_of_2
+
+
+class BitonicISA(ABC, metaclass=ABCMeta):
+
+ @abstractmethod
+ def vector_size(self):
+ pass
+
+ @abstractmethod
+ def max_bitonic_sort_vectors(self):
+ pass
+
+ def largest_merge_variant_needed(self):
+ return next_power_of_2(self.max_bitonic_sort_vectors()) / 2;
+
+ @abstractmethod
+ def vector_size(self):
+ pass
+
+ @abstractmethod
+ def vector_type(self):
+ pass
+
+ @classmethod
+ @abstractmethod
+ def supported_types(cls):
+ pass
+
+ @abstractmethod
+ def generate_prologue(self, f):
+ pass
+
+ @abstractmethod
+ def generate_epilogue(self, f):
+ pass
+
+
+ @abstractmethod
+ def generate_1v_basic_sorters(self, f, ascending):
+ pass
+
+ @abstractmethod
+ def generate_1v_merge_sorters(self, f, ascending):
+ pass
+
+ def generate_1v_sorters(self, f, ascending):
+ self.generate_1v_basic_sorters(f, ascending)
+ self.generate_1v_merge_sorters(f, ascending)
+
+ @abstractmethod
+ def generate_compounded_sorter(self, f, width, ascending, inline):
+ pass
+
+ @abstractmethod
+ def generate_compounded_merger(self, f, width, ascending, inline):
+ pass
+
+ @abstractmethod
+ def generate_entry_points(self, f):
+ pass
+
+ @abstractmethod
+ def generate_master_entry_point(self, f):
+ pass
--- /dev/null
+##
+## Licensed to the .NET Foundation under one or more agreements.
+## The .NET Foundation licenses this file to you under the MIT license.
+##
+
+native_size_map = {
+ "int32_t": 4,
+ "uint32_t": 4,
+ "float": 4,
+ "int64_t": 8,
+ "uint64_t": 8,
+ "double": 8,
+}
+
+
+def next_power_of_2(v):
+ v = v - 1
+ v |= v >> 1
+ v |= v >> 2
+ v |= v >> 4
+ v |= v >> 8
+ v |= v >> 16
+ v = v + 1
+ return int(v)
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#ifndef VXSORT_VXSORT_H
+#define VXSORT_VXSORT_H
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("popcnt"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("popcnt")
+#endif
+#endif
+
+
+#include <assert.h>
+#include <immintrin.h>
+
+
+#include "defs.h"
+//#include "isa_detection.h"
+#include "alignment.h"
+#include "machine_traits.h"
+#include "smallsort/bitonic_sort.h"
+
+//#include <algorithm>
+//#include <cstring>
+//#include <cstdint>
+
+namespace vxsort {
+using vxsort::smallsort::bitonic;
+
+
+template <typename T, vector_machine M, int Unroll=1>
+class vxsort {
+ static_assert(Unroll >= 1, "Unroll can be in the range 1..12");
+ static_assert(Unroll <= 12, "Unroll can be in the range 1..12");
+
+private:
+ using MT = vxsort_machine_traits<T, M>;
+ typedef typename MT::TV TV;
+ typedef alignment_hint<sizeof(TV)> AH;
+
+ static const int ELEMENT_ALIGN = sizeof(T) - 1;
+ static const int N = sizeof(TV) / sizeof(T);
+ static const int32_t MAX_BITONIC_SORT_VECTORS = 16;
+ static const int32_t SMALL_SORT_THRESHOLD_ELEMENTS = MAX_BITONIC_SORT_VECTORS * N;
+ static const int32_t MaxInnerUnroll = (MAX_BITONIC_SORT_VECTORS - 3) / 2;
+ static const int32_t SafeInnerUnroll = MaxInnerUnroll > Unroll ? Unroll : MaxInnerUnroll;
+ static const int32_t SLACK_PER_SIDE_IN_VECTORS = Unroll;
+ static const size_t ALIGN = AH::ALIGN;
+ static const size_t ALIGN_MASK = ALIGN - 1;
+
+ static const int SLACK_PER_SIDE_IN_ELEMENTS = SLACK_PER_SIDE_IN_VECTORS * N;
+ // The formula for figuring out how much temporary space we need for partitioning:
+ // 2 x the number of slack elements on each side for the purpose of partitioning in unrolled manner +
+ // 2 x amount of maximal bytes needed for alignment (32)
+ // one more vector's worth of elements since we write with N-way stores from both ends of the temporary area
+ // and we must make sure we do not accidentally over-write from left -> right or vice-versa right on that edge...
+ // In other words, while we allocated this much temp memory, the actual amount of elements inside said memory
+ // is smaller by 8 elements + 1 for each alignment (max alignment is actually N-1, I just round up to N...)
+ // This long sense just means that we over-allocate N+2 elements...
+ static const int PARTITION_TMP_SIZE_IN_ELEMENTS =
+ (2 * SLACK_PER_SIDE_IN_ELEMENTS + N + 4*N);
+
+ static int floor_log2_plus_one(size_t n) {
+ auto result = 0;
+ while (n >= 1) {
+ result++;
+ n /= 2;
+ }
+ return result;
+ }
+ static void swap(T* left, T* right) {
+ auto tmp = *left;
+ *left = *right;
+ *right = tmp;
+ }
+ static void swap_if_greater(T* left, T* right) {
+ if (*left <= *right)
+ return;
+ swap(left, right);
+ }
+
+ static void insertion_sort(T* lo, T* hi) {
+ for (auto i = lo + 1; i <= hi; i++) {
+ auto j = i;
+ auto t = *i;
+ while ((j > lo) && (t < *(j - 1))) {
+ *j = *(j - 1);
+ j--;
+ }
+ *j = t;
+ }
+ }
+
+ static void heap_sort(T* lo, T* hi) {
+ size_t n = hi - lo + 1;
+ for (size_t i = n / 2; i >= 1; i--) {
+ down_heap(i, n, lo);
+ }
+ for (size_t i = n; i > 1; i--) {
+ swap(lo, lo + i - 1);
+ down_heap(1, i - 1, lo);
+ }
+ }
+ static void down_heap(size_t i, size_t n, T* lo) {
+ auto d = *(lo + i - 1);
+ size_t child;
+ while (i <= n / 2) {
+ child = 2 * i;
+ if (child < n && *(lo + child - 1) < (*(lo + child))) {
+ child++;
+ }
+ if (!(d < *(lo + child - 1))) {
+ break;
+ }
+ *(lo + i - 1) = *(lo + child - 1);
+ i = child;
+ }
+ *(lo + i - 1) = d;
+ }
+
+ void reset(T* start, T* end) {
+ _depth = 0;
+ _startPtr = start;
+ _endPtr = end;
+ }
+
+ T* _startPtr = nullptr;
+ T* _endPtr = nullptr;
+
+ T _temp[PARTITION_TMP_SIZE_IN_ELEMENTS];
+ int _depth = 0;
+
+ NOINLINE
+ T* align_left_scalar_uncommon(T* read_left, T pivot,
+ T*& tmp_left, T*& tmp_right) {
+ if (((size_t)read_left & ALIGN_MASK) == 0)
+ return read_left;
+
+ auto next_align = (T*)(((size_t)read_left + ALIGN) & ~ALIGN_MASK);
+ while (read_left < next_align) {
+ auto v = *(read_left++);
+ if (v <= pivot) {
+ *(tmp_left++) = v;
+ } else {
+ *(--tmp_right) = v;
+ }
+ }
+
+ return read_left;
+ }
+
+ NOINLINE
+ T* align_right_scalar_uncommon(T* readRight, T pivot,
+ T*& tmpLeft, T*& tmpRight) {
+ if (((size_t) readRight & ALIGN_MASK) == 0)
+ return readRight;
+
+ auto nextAlign = (T *) ((size_t) readRight & ~ALIGN_MASK);
+ while (readRight > nextAlign) {
+ auto v = *(--readRight);
+ if (v <= pivot) {
+ *(tmpLeft++) = v;
+ } else {
+ *(--tmpRight) = v;
+ }
+ }
+
+ return readRight;
+ }
+
+ void sort(T* left, T* right, AH realignHint,
+ int depthLimit) {
+ auto length = (size_t)(right - left + 1);
+
+ T* mid;
+ switch (length) {
+ case 0:
+ case 1:
+ return;
+ case 2:
+ swap_if_greater(left, right);
+ return;
+ case 3:
+ mid = right - 1;
+ swap_if_greater(left, mid);
+ swap_if_greater(left, right);
+ swap_if_greater(mid, right);
+ return;
+ }
+
+ // Go to insertion sort below this threshold
+ if (length <= SMALL_SORT_THRESHOLD_ELEMENTS) {
+
+ auto nextLength = (length & (N-1)) > 0 ? (length + N) & ~(N-1) : length;
+
+ auto extraSpaceNeeded = nextLength - length;
+ auto fakeLeft = left - extraSpaceNeeded;
+ if (fakeLeft >= _startPtr) {
+ bitonic<T, M>::sort(fakeLeft, nextLength);
+ } else {
+ insertion_sort(left, right);
+ }
+ return;
+ }
+
+ // Detect a whole bunch of bad cases where partitioning
+ // will not do well:
+ // 1. Reverse sorted array
+ // 2. High degree of repeated values (dutch flag problem, one value)
+ if (depthLimit == 0) {
+ heap_sort(left, right);
+ _depth--;
+ return;
+ }
+ depthLimit--;
+
+ // This is going to be a bit weird:
+ // Pre/Post alignment calculations happen here: we prepare hints to the
+ // partition function of how much to align and in which direction
+ // (pre/post). The motivation to do these calculations here and the actual
+ // alignment inside the partitioning code is that here, we can cache those
+ // calculations. As we recurse to the left we can reuse the left cached
+ // calculation, And when we recurse to the right we reuse the right
+ // calculation, so we can avoid re-calculating the same aligned addresses
+ // throughout the recursion, at the cost of a minor code complexity
+ // Since we branch on the magi values REALIGN_LEFT & REALIGN_RIGHT its safe
+ // to assume the we are not torturing the branch predictor.'
+
+ // We use a long as a "struct" to pass on alignment hints to the
+ // partitioning By packing 2 32 bit elements into it, as the JIT seem to not
+ // do this. In reality we need more like 2x 4bits for each side, but I
+ // don't think there is a real difference'
+
+ if (realignHint.left_align == AH::REALIGN) {
+ // Alignment flow:
+ // * Calculate pre-alignment on the left
+ // * See it would cause us an out-of bounds read
+ // * Since we'd like to avoid that, we adjust for post-alignment
+ // * No branches since we do branch->arithmetic
+ auto preAlignedLeft = reinterpret_cast<T*>(reinterpret_cast<size_t>(left) & ~ALIGN_MASK);
+ auto cannotPreAlignLeft = (preAlignedLeft - _startPtr) >> 63;
+ realignHint.left_align = (preAlignedLeft - left) + (N & cannotPreAlignLeft);
+ assert(realignHint.left_align >= -N && realignHint.left_align <= N);
+ assert(AH::is_aligned(left + realignHint.left_align));
+ }
+
+ if (realignHint.right_align == AH::REALIGN) {
+ // Same as above, but in addition:
+ // right is pointing just PAST the last element we intend to partition
+ // (it's pointing to where we will store the pivot!) So we calculate alignment based on
+ // right - 1
+ auto preAlignedRight = reinterpret_cast<T*>(((reinterpret_cast<size_t>(right) - 1) & ~ALIGN_MASK) + ALIGN);
+ auto cannotPreAlignRight = (_endPtr - preAlignedRight) >> 63;
+ realignHint.right_align = (preAlignedRight - right - (N & cannotPreAlignRight));
+ assert(realignHint.right_align >= -N && realignHint.right_align <= N);
+ assert(AH::is_aligned(right + realignHint.right_align));
+ }
+
+ // Compute median-of-three, of:
+ // the first, mid and one before last elements
+ mid = left + ((right - left) / 2);
+ swap_if_greater(left, mid);
+ swap_if_greater(left, right - 1);
+ swap_if_greater(mid, right - 1);
+
+ // Pivot is mid, place it in the right hand side
+ swap(mid, right);
+
+ auto sep = (length < PARTITION_TMP_SIZE_IN_ELEMENTS) ?
+ vectorized_partition<SafeInnerUnroll>(left, right, realignHint) :
+ vectorized_partition<Unroll>(left, right, realignHint);
+
+
+
+ _depth++;
+ sort(left, sep - 2, realignHint.realign_right(), depthLimit);
+ sort(sep, right, realignHint.realign_left(), depthLimit);
+ _depth--;
+ }
+
+
+ static INLINE void partition_block(TV& dataVec,
+ const TV& P,
+ T*& left,
+ T*& right) {
+ if (MT::supports_compress_writes()) {
+ partition_block_with_compress(dataVec, P, left, right);
+ } else {
+ partition_block_without_compress(dataVec, P, left, right);
+ }
+ }
+
+ static INLINE void partition_block_without_compress(TV& dataVec,
+ const TV& P,
+ T*& left,
+ T*& right) {
+ auto mask = MT::get_cmpgt_mask(dataVec, P);
+ dataVec = MT::partition_vector(dataVec, mask);
+ MT::store_vec(reinterpret_cast<TV*>(left), dataVec);
+ MT::store_vec(reinterpret_cast<TV*>(right), dataVec);
+ auto popCount = -_mm_popcnt_u64(mask);
+ right += popCount;
+ left += popCount + N;
+ }
+
+ static INLINE void partition_block_with_compress(TV& dataVec,
+ const TV& P,
+ T*& left,
+ T*& right) {
+ auto mask = MT::get_cmpgt_mask(dataVec, P);
+ auto popCount = -_mm_popcnt_u64(mask);
+ MT::store_compress_vec(reinterpret_cast<TV*>(left), dataVec, ~mask);
+ MT::store_compress_vec(reinterpret_cast<TV*>(right + N + popCount), dataVec, mask);
+ right += popCount;
+ left += popCount + N;
+ }
+
+ template<int InnerUnroll>
+ T* vectorized_partition(T* const left, T* const right, const AH hint) {
+ assert(right - left >= SMALL_SORT_THRESHOLD_ELEMENTS);
+ assert((reinterpret_cast<size_t>(left) & ELEMENT_ALIGN) == 0);
+ assert((reinterpret_cast<size_t>(right) & ELEMENT_ALIGN) == 0);
+
+ // Vectorized double-pumped (dual-sided) partitioning:
+ // We start with picking a pivot using the media-of-3 "method"
+ // Once we have sensible pivot stored as the last element of the array
+ // We process the array from both ends.
+ //
+ // To get this rolling, we first read 2 Vector256 elements from the left and
+ // another 2 from the right, and store them in some temporary space in order
+ // to leave enough "space" inside the vector for storing partitioned values.
+ // Why 2 from each side? Because we need n+1 from each side where n is the
+ // number of Vector256 elements we process in each iteration... The
+ // reasoning behind the +1 is because of the way we decide from *which* side
+ // to read, we may end up reading up to one more vector from any given side
+ // and writing it in its entirety to the opposite side (this becomes
+ // slightly clearer when reading the code below...) Conceptually, the bulk
+ // of the processing looks like this after clearing out some initial space
+ // as described above:
+
+ // [.............................................................................]
+ // ^wl ^rl rr^ wr^
+ // Where:
+ // wl = writeLeft
+ // rl = readLeft
+ // rr = readRight
+ // wr = writeRight
+
+ // In every iteration, we select what side to read from based on how much
+ // space is left between head read/write pointer on each side...
+ // We read from where there is a smaller gap, e.g. that side
+ // that is closer to the unfortunate possibility of its write head
+ // overwriting its read head... By reading from THAT side, we're ensuring
+ // this does not happen
+
+ // An additional unfortunate complexity we need to deal with is that the
+ // right pointer must be decremented by another Vector256<T>.Count elements
+ // Since the Load/Store primitives obviously accept start addresses
+ auto pivot = *right;
+ // We do this here just in case we need to pre-align to the right
+ // We end up
+ *right = std::numeric_limits<T>::Max();
+
+ // Broadcast the selected pivot
+ const TV P = MT::broadcast(pivot);
+
+ auto readLeft = left;
+ auto readRight = right;
+
+ auto tmpStartLeft = _temp;
+ auto tmpLeft = tmpStartLeft;
+ auto tmpStartRight = _temp + PARTITION_TMP_SIZE_IN_ELEMENTS;
+ auto tmpRight = tmpStartRight;
+
+ tmpRight -= N;
+
+ // the read heads always advance by 8 elements, or 32 bytes,
+ // We can spend some extra time here to align the pointers
+ // so they start at a cache-line boundary
+ // Once that happens, we can read with Avx.LoadAlignedVector256
+ // And also know for sure that our reads will never cross cache-lines
+ // Otherwise, 50% of our AVX2 Loads will need to read from two cache-lines
+ align_vectorized(left, right, hint, P, readLeft, readRight,
+ tmpStartLeft, tmpLeft, tmpStartRight, tmpRight);
+
+ const auto leftAlign = hint.left_align;
+ const auto rightAlign = hint.right_align;
+ if (leftAlign > 0) {
+ tmpRight += N;
+ readLeft = align_left_scalar_uncommon(readLeft, pivot, tmpLeft, tmpRight);
+ tmpRight -= N;
+ }
+
+ if (rightAlign < 0) {
+ tmpRight += N;
+ readRight =
+ align_right_scalar_uncommon(readRight, pivot, tmpLeft, tmpRight);
+ tmpRight -= N;
+ }
+
+ assert(((size_t)readLeft & ALIGN_MASK) == 0);
+ assert(((size_t)readRight & ALIGN_MASK) == 0);
+
+ assert((((size_t)readRight - (size_t)readLeft) % ALIGN) == 0);
+ assert((readRight - readLeft) >= InnerUnroll * 2);
+
+ // From now on, we are fully aligned
+ // and all reading is done in full vector units
+ auto readLeftV = (TV*) readLeft;
+ auto readRightV = (TV*) readRight;
+ #ifndef NDEBUG
+ readLeft = nullptr;
+ readRight = nullptr;
+ #endif
+
+ for (auto u = 0; u < InnerUnroll; u++) {
+ auto dl = MT::load_vec(readLeftV + u);
+ auto dr = MT::load_vec(readRightV - (u + 1));
+ partition_block(dl, P, tmpLeft, tmpRight);
+ partition_block(dr, P, tmpLeft, tmpRight);
+ }
+
+ tmpRight += N;
+ // Adjust for the reading that was made above
+ readLeftV += InnerUnroll;
+ readRightV -= InnerUnroll*2;
+ TV* nextPtr;
+
+ auto writeLeft = left;
+ auto writeRight = right - N;
+
+ while (readLeftV < readRightV) {
+ if (writeRight - ((T *) readRightV) < (2 * (InnerUnroll * N) - N)) {
+ nextPtr = readRightV;
+ readRightV -= InnerUnroll;
+ } else {
+ mess_up_cmov();
+ nextPtr = readLeftV;
+ readLeftV += InnerUnroll;
+ }
+
+ TV d01, d02, d03, d04, d05, d06, d07, d08, d09, d10, d11, d12;
+
+ switch (InnerUnroll) {
+ case 12: d12 = MT::load_vec(nextPtr + InnerUnroll - 12);
+ case 11: d11 = MT::load_vec(nextPtr + InnerUnroll - 11);
+ case 10: d10 = MT::load_vec(nextPtr + InnerUnroll - 10);
+ case 9: d09 = MT::load_vec(nextPtr + InnerUnroll - 9);
+ case 8: d08 = MT::load_vec(nextPtr + InnerUnroll - 8);
+ case 7: d07 = MT::load_vec(nextPtr + InnerUnroll - 7);
+ case 6: d06 = MT::load_vec(nextPtr + InnerUnroll - 6);
+ case 5: d05 = MT::load_vec(nextPtr + InnerUnroll - 5);
+ case 4: d04 = MT::load_vec(nextPtr + InnerUnroll - 4);
+ case 3: d03 = MT::load_vec(nextPtr + InnerUnroll - 3);
+ case 2: d02 = MT::load_vec(nextPtr + InnerUnroll - 2);
+ case 1: d01 = MT::load_vec(nextPtr + InnerUnroll - 1);
+ }
+
+ switch (InnerUnroll) {
+ case 12: partition_block(d12, P, writeLeft, writeRight);
+ case 11: partition_block(d11, P, writeLeft, writeRight);
+ case 10: partition_block(d10, P, writeLeft, writeRight);
+ case 9: partition_block(d09, P, writeLeft, writeRight);
+ case 8: partition_block(d08, P, writeLeft, writeRight);
+ case 7: partition_block(d07, P, writeLeft, writeRight);
+ case 6: partition_block(d06, P, writeLeft, writeRight);
+ case 5: partition_block(d05, P, writeLeft, writeRight);
+ case 4: partition_block(d04, P, writeLeft, writeRight);
+ case 3: partition_block(d03, P, writeLeft, writeRight);
+ case 2: partition_block(d02, P, writeLeft, writeRight);
+ case 1: partition_block(d01, P, writeLeft, writeRight);
+ }
+ }
+
+ readRightV += (InnerUnroll - 1);
+
+ while (readLeftV <= readRightV) {
+ if (writeRight - (T *) readRightV < N) {
+ nextPtr = readRightV;
+ readRightV -= 1;
+ } else {
+ mess_up_cmov();
+ nextPtr = readLeftV;
+ readLeftV += 1;
+ }
+
+ auto d = MT::load_vec(nextPtr);
+ partition_block(d, P, writeLeft, writeRight);
+ //partition_block_without_compress(d, P, writeLeft, writeRight);
+ }
+
+ // 3. Copy-back the 4 registers + remainder we partitioned in the beginning
+ auto leftTmpSize = tmpLeft - tmpStartLeft;
+ memcpy(writeLeft, tmpStartLeft, leftTmpSize * sizeof(T));
+ writeLeft += leftTmpSize;
+ auto rightTmpSize = tmpStartRight - tmpRight;
+ memcpy(writeLeft, tmpRight, rightTmpSize * sizeof(T));
+
+ // Shove to pivot back to the boundary
+ *right = *writeLeft;
+ *writeLeft++ = pivot;
+
+ assert(writeLeft > left);
+ assert(writeLeft <= right);
+
+ return writeLeft;
+ }
+ void align_vectorized(const T* left,
+ const T* right,
+ const AH& hint,
+ const TV P,
+ T*& readLeft,
+ T*& readRight,
+ T*& tmpStartLeft,
+ T*& tmpLeft,
+ T*& tmpStartRight,
+ T*& tmpRight) const {
+ const auto leftAlign = hint.left_align;
+ const auto rightAlign = hint.right_align;
+ const auto rai = ~((rightAlign - 1) >> 31);
+ const auto lai = leftAlign >> 31;
+ const auto preAlignedLeft = (TV*) (left + leftAlign);
+ const auto preAlignedRight = (TV*) (right + rightAlign - N);
+
+ // Alignment with vectorization is tricky, so read carefully before changing code:
+ // 1. We load data, which we might need to align, if the alignment hints
+ // mean pre-alignment (or overlapping alignment)
+ // 2. We partition and store in the following order:
+ // a) right-portion of right vector to the right-side
+ // b) left-portion of left vector to the right side
+ // c) at this point one-half of each partitioned vector has been committed
+ // back to memory.
+ // d) we advance the right write (tmpRight) pointer by how many elements
+ // were actually needed to be written to the right hand side
+ // e) We write the right portion of the left vector to the right side
+ // now that its write position has been updated
+ auto RT0 = MT::load_vec(preAlignedRight);
+ auto LT0 = MT::load_vec(preAlignedLeft);
+ auto rtMask = MT::get_cmpgt_mask(RT0, P);
+ auto ltMask = MT::get_cmpgt_mask(LT0, P);
+ const auto rtPopCountRightPart = max(_mm_popcnt_u32(rtMask), rightAlign);
+ const auto ltPopCountRightPart = _mm_popcnt_u32(ltMask);
+ const auto rtPopCountLeftPart = N - rtPopCountRightPart;
+ const auto ltPopCountLeftPart = N - ltPopCountRightPart;
+
+ if (MT::supports_compress_writes()) {
+ MT::store_compress_vec((TV *) (tmpRight + N - rtPopCountRightPart), RT0, rtMask);
+ MT::store_compress_vec((TV *) tmpLeft, LT0, ~ltMask);
+
+ tmpRight -= rtPopCountRightPart & rai;
+ readRight += (rightAlign - N) & rai;
+
+ MT::store_compress_vec((TV *) (tmpRight + N - ltPopCountRightPart), LT0, ltMask);
+ tmpRight -= ltPopCountRightPart & lai;
+ tmpLeft += ltPopCountLeftPart & lai;
+ tmpStartLeft += -leftAlign & lai;
+ readLeft += (leftAlign + N) & lai;
+
+ MT::store_compress_vec((TV*) tmpLeft, RT0, ~rtMask);
+ tmpLeft += rtPopCountLeftPart & rai;
+ tmpStartRight -= rightAlign & rai;
+ }
+ else {
+ RT0 = MT::partition_vector(RT0, rtMask);
+ LT0 = MT::partition_vector(LT0, ltMask);
+ MT::store_vec((TV*) tmpRight, RT0);
+ MT::store_vec((TV*) tmpLeft, LT0);
+
+
+ tmpRight -= rtPopCountRightPart & rai;
+ readRight += (rightAlign - N) & rai;
+
+ MT::store_vec((TV*) tmpRight, LT0);
+ tmpRight -= ltPopCountRightPart & lai;
+
+ tmpLeft += ltPopCountLeftPart & lai;
+ tmpStartLeft += -leftAlign & lai;
+ readLeft += (leftAlign + N) & lai;
+
+ MT::store_vec((TV*) tmpLeft, RT0);
+ tmpLeft += rtPopCountLeftPart & rai;
+ tmpStartRight -= rightAlign & rai;
+ }
+ }
+
+ public:
+ NOINLINE void sort(T* left, T* right) {
+ reset(left, right);
+ auto depthLimit = 2 * floor_log2_plus_one(right + 1 - left);
+ sort(left, right, AH(), depthLimit);
+ }
+};
+
+} // namespace gcsort
+
+#include "vxsort_targets_disable.h"
+
+#endif
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute pop
+#else
+#pragma GCC pop_options
+#endif
+#endif
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx2"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx512f")
+#endif
+#endif
--- /dev/null
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+
+#ifdef __GNUC__
+#ifdef __clang__
+#pragma clang attribute push (__attribute__((target("avx512f"))), apply_to = any(function))
+#else
+#pragma GCC push_options
+#pragma GCC target("avx512f")
+#endif
+#endif
../gc/softwarewritewatch.cpp
../gc/handletablecache.cpp)
+if (CLR_CMAKE_TARGET_ARCH_AMD64 AND CLR_CMAKE_TARGET_WIN32)
+ set ( GC_SOURCES_WKS
+ ${GC_SOURCES_WKS}
+ ../gc/vxsort/isa_detection.cpp
+ ../gc/vxsort/do_vxsort_avx2.cpp
+ ../gc/vxsort/do_vxsort_avx512.cpp
+ ../gc/vxsort/machine_traits.avx2.cpp
+ ../gc/vxsort/smallsort/bitonic_sort.AVX2.int64_t.generated.cpp
+ ../gc/vxsort/smallsort/bitonic_sort.AVX2.int32_t.generated.cpp
+ ../gc/vxsort/smallsort/bitonic_sort.AVX512.int64_t.generated.cpp
+ ../gc/vxsort/smallsort/bitonic_sort.AVX512.int32_t.generated.cpp
+)
+endif (CLR_CMAKE_TARGET_ARCH_AMD64 AND CLR_CMAKE_TARGET_WIN32)
+
set(GC_HEADERS_WKS
${GC_HEADERS_DAC_AND_WKS_COMMON}
../gc/gceventstatus.h
projects / platform / upstream / dotnet / runtime.git / commitdiff