platform/upstream/llvm.git
2 years ago[ARM][clang] Option b-key must not affect __ARM_FEATURE_PAC_DEFAULT
Ties Stuij [Thu, 9 Dec 2021 11:51:31 +0000 (11:51 +0000)]
[ARM][clang] Option b-key must not affect __ARM_FEATURE_PAC_DEFAULT

When using -mbranch-protection=pac-ret+b-key, macro __ARM_FEATURE_PAC_DEFAULT
should still have the value corresponding to a-key, because b-key is only valid
for AArch64.

This patch is part of a series that adds support for the PACBTI-M extension of
the Armv8.1-M architecture, as detailed here:

https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/armv8-1-m-pointer-authentication-and-branch-target-identification-extension

The PACBTI-M specification can be found in the Armv8-M Architecture Reference
Manual:

https://developer.arm.com/documentation/ddi0553/latest

The following people contributed to this patch:

- Victor Campos

Reviewed By: danielkiss

Differential Revision: https://reviews.llvm.org/D115140

2 years ago[mlir][linalg][bufferize] LinalgOps can bufferize inplace with input args
Matthias Springer [Thu, 9 Dec 2021 10:50:47 +0000 (19:50 +0900)]
[mlir][linalg][bufferize] LinalgOps can bufferize inplace with input args

LinalgOp results usually bufferize inplace with output args. With this change, they may buffer inplace with input args if the value of the output arg is not used in the computation.

Differential Revision: https://reviews.llvm.org/D115022

2 years ago[clang][clangd] Desugar array type.
lh123 [Thu, 9 Dec 2021 12:12:48 +0000 (20:12 +0800)]
[clang][clangd] Desugar array type.

Desugar array type.

Reviewed By: sammccall

Differential Revision: https://reviews.llvm.org/D115107

2 years ago[MetaRenamer] Add command line options to disable renaming name with specified prefixes
Dmitry Makogon [Thu, 9 Dec 2021 11:41:48 +0000 (18:41 +0700)]
[MetaRenamer] Add command line options to disable renaming name with specified prefixes

This patch adds 4 options for specifying functions, aliases, globals and
structs name prefixes hat don't need to be renamed by MetaRenamer pass.
This is useful if one has some downstream logic that depends directly
on an entity name. MetaRenamer can break this logic, but with the patch
you can tell it not to rename certain names.

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D115323

2 years ago[lldb][NFC] clang-format some files as preparation for https://reviews.llvm.org/D114627
Lasse Folger [Thu, 9 Dec 2021 11:09:56 +0000 (12:09 +0100)]
[lldb][NFC] clang-format some files as preparation for https://reviews.llvm.org/D114627

Reviewed By: werat

Differential Revision: https://reviews.llvm.org/D115110

2 years ago[MLIR] Move Presburger Math from FlatAffineConstraints to Presburger/IntegerPolyhedron
Groverkss [Thu, 9 Dec 2021 10:41:00 +0000 (16:11 +0530)]
[MLIR] Move Presburger Math from FlatAffineConstraints to Presburger/IntegerPolyhedron

This patch factors out math functionality that is a subset of Presburger arithmetic and moves it from FlatAffineConstraints to Presburger/IntegerPolyhedron. This patch only moves some parts of the functionality planned to be moved, with subsequent patches moving more functionality. There are three main reasons for this:

1. This split makes the Presburger Library easier and more flexible to use
    across MLIR, by not depending on IR.
2. This split allows the Presburger library to be developed independently from
    Affine Analysis, with Affine Analysis using this library.
3. With more functionality being upstreamed to the Presburger Library, the
    mlir/Analysis directory will be cluttered with Presburger library components
    since they depend on math functionality from FlatAffineConstraints. Moving this
    functionality to the Presburger directory allows keeping the new functionality
    in the Presburger directory.

This patch is part of an ongoing effort to make the Presburger Library easier to use. The motivation for this effort is the feedback received at the LLVM conference from Mehdi and others.

Reviewed By: bondhugula

Differential Revision: https://reviews.llvm.org/D114674

2 years agoRevert "Reapply [runtimes] Fix building initial libunwind+libcxxabi+libcxx with compi...
Martin Storsjö [Thu, 9 Dec 2021 10:54:37 +0000 (12:54 +0200)]
Revert "Reapply [runtimes] Fix building initial libunwind+libcxxabi+libcxx with compiler implied -lunwind"

This reverts commit 317dc31e53b83c1d2a468d7a541925f0cc7d9dce.

After that change, OpenMP doesn't find dependencies in the host
system (it fails do find e.g. /usr/lib/x86_64-linux-gnu/libelf.so
which it found before), which causes some OpenMP target offloading
plugins to not be found. This doesn't break the build, but just
causes the AMDGPU OpenMP target plugin to be omitted. See
https://reviews.llvm.org/D113253#3181934 for the report of this
issue.

2 years ago[LV] Mark various functions as const (NFC).
Florian Hahn [Thu, 9 Dec 2021 10:51:29 +0000 (10:51 +0000)]
[LV] Mark various functions as const (NFC).

Make sure various accessors do not modify any state, in preparation for
D115111.

2 years ago[ARM][clang] Define feature test macro for the PACBTI-M extension
Ties Stuij [Thu, 9 Dec 2021 09:40:46 +0000 (09:40 +0000)]
[ARM][clang] Define feature test macro for the PACBTI-M extension

If the extension string "+pacbti" was given in -march=... or -mcpu=... options the compiler shall define the following preprocessor macros:

    __ARM_FEATURE_PAUTH with value 1.
    __ARM_FEATURE_BTI with value 1.

This patch is part of a series that adds support for the PACBTI-M extension of
the Armv8.1-M architecture, as detailed here:

https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/armv8-1-m-pointer-authentication-and-branch-target-identification-extension

The PACBTI-M specification can be found in the Armv8-M Architecture Reference
Manual:

https://developer.arm.com/documentation/ddi0553/latest

The following people contributed to this patch:

- Momchil Velikov
- Ties Stuij

Reviewed By: miyuki

Differential Revision: https://reviews.llvm.org/D112431

2 years ago[clang-format] PR48916 PointerAlignment not working when using C++20 init-statement...
mydeveloperday [Thu, 9 Dec 2021 10:34:55 +0000 (10:34 +0000)]
[clang-format] PR48916 PointerAlignment not working when using C++20 init-statement in for loop

https://bugs.llvm.org/show_bug.cgi?id=48916

Left and Right Alignment inside a loop is misaligned.

Reviewed By: HazardyKnusperkeks, curdeius

Differential Revision: https://reviews.llvm.org/D115050

2 years ago[clang][deps] Use MemoryBuffer in minimizing FS
Jan Svoboda [Thu, 9 Dec 2021 09:47:54 +0000 (10:47 +0100)]
[clang][deps] Use MemoryBuffer in minimizing FS

This patch avoids unnecessarily copying contents of `mmap`-ed files into `CachedFileSystemEntry` by storing `MemoryBuffer` instead. The change leads to ~50% reduction of peak memory footprint when scanning LLVM+Clang via `clang-scan-deps`.

Depends on D115331.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D115043

2 years ago[llvm] Add null-termination capability to SmallVectorMemoryBuffer
Jan Svoboda [Wed, 8 Dec 2021 12:24:45 +0000 (13:24 +0100)]
[llvm] Add null-termination capability to SmallVectorMemoryBuffer

Most of `MemoryBuffer` interfaces expose a `RequiresNullTerminator` parameter that's being used to:
* determine how to open a file (`mmap` vs `open`),
* assert newly initialized buffer indeed has an implicit null terminator.

This patch adds the paramater to the `SmallVectorMemoryBuffer` constructors, meaning:
* null terminator can now be added to `SmallVector`s that didn't have one before,
* `SmallVectors` that had a null terminator before keep it even after the move.

In line with existing code, the new parameter is defaulted to `true`. This patch makes sure all calls to the `SmallVectorMemoryBuffer` constructor set it to `false` to preserve the current semantics.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D115331

2 years ago[llvm][lldb] Remove unused SmallVectorMemoryBuffer.h includes
Jan Svoboda [Thu, 9 Dec 2021 09:45:10 +0000 (10:45 +0100)]
[llvm][lldb] Remove unused SmallVectorMemoryBuffer.h includes

2 years ago[MLIR] Introduce coalesce for PresburgerSet
Michel Weber [Thu, 9 Dec 2021 10:16:31 +0000 (15:46 +0530)]
[MLIR] Introduce coalesce for PresburgerSet

This patch provides functionality for simplifying `PresburgerSet`s by checking if any `FlatAffineConstraints` in the set is contained in another, and removing such redundant FACs.

This is part of a series of patches to provide functionality for [integer set coalescing](http://impact.gforge.inria.fr/impact2015/papers/impact2015-verdoolaege.pdf) in MLIR.

Reviewed By: arjunp

Differential Revision: https://reviews.llvm.org/D110617

2 years ago[MLIR][OpenMP] Added omp.atomic.update
Shraiysh Vaishay [Thu, 9 Dec 2021 05:35:55 +0000 (11:05 +0530)]
[MLIR][OpenMP] Added omp.atomic.update

This patch supports the atomic construct (update) following section 2.17.7 of OpenMP 5.0 standard. Also added tests and verifier for the same.

Reviewed By: kiranchandramohan, peixin

Differential Revision: https://reviews.llvm.org/D112982

2 years ago[clang] Fix a misadjusted path style comparison in a unittest
Martin Storsjö [Fri, 5 Nov 2021 09:28:21 +0000 (11:28 +0200)]
[clang] Fix a misadjusted path style comparison in a unittest

This was changed incorrectly by accident in
99023627010bbfefb71e25a2b4d056de1cbd354e.

Differential Revision: https://reviews.llvm.org/D113254

2 years ago[clang][deps] Use lock_guard instead of unique_lock
Jan Svoboda [Thu, 9 Dec 2021 09:42:47 +0000 (10:42 +0100)]
[clang][deps] Use lock_guard instead of unique_lock

This patch changes uses of `std::unique_lock` to `std::lock_guard`.

The `std::unique_lock` template provides some advanced capabilities (deferred locking, time-constrained locking attempts, etc.) we don't use in the caching filesystem. Plain `std::lock_guard` will do here.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D115332

2 years ago[Test] [GVN] Add test showing equivalent PHIs generation by GVN
Dmitry Makogon [Thu, 9 Dec 2021 09:22:39 +0000 (16:22 +0700)]
[Test] [GVN] Add test showing equivalent PHIs generation by GVN

2 years ago[ARM] Fix gcc warning about mix of enumeral and non-enumeral types
Mikael Holmen [Thu, 9 Dec 2021 09:20:27 +0000 (10:20 +0100)]
[ARM] Fix gcc warning about mix of enumeral and non-enumeral types

gcc warned with
../lib/Target/ARM/ARMFrameLowering.cpp:797:31: warning: enumeral and non-enumeral type in conditional expression [-Wextra]
  797 |               Reg == ARM::R12 ? ARM::RA_AUTH_CODE : Reg, true);
      |               ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~

2 years ago[PowerPC] Fix gcc warning about unused variable [NFC]
Mikael Holmen [Thu, 9 Dec 2021 09:18:29 +0000 (10:18 +0100)]
[PowerPC] Fix gcc warning about unused variable [NFC]

gcc warned about
../lib/Target/PowerPC/PPCTargetTransformInfo.cpp:1401:13: warning: unused variable 'VecTy' [-Wunused-variable]
 1401 |   if (auto *VecTy = dyn_cast<FixedVectorType>(DataType)) {
      |             ^~~~~

2 years agotsan: new runtime (v3)
Dmitry Vyukov [Thu, 2 Dec 2021 07:36:44 +0000 (08:36 +0100)]
tsan: new runtime (v3)

This change switches tsan to the new runtime which features:
 - 2x smaller shadow memory (2x of app memory)
 - faster fully vectorized race detection
 - small fixed-size vector clocks (512b)
 - fast vectorized vector clock operations
 - unlimited number of alive threads/goroutimes

Depends on D112602.

Reviewed By: melver

Differential Revision: https://reviews.llvm.org/D112603

2 years ago[NewPM] Port FlattenCFGPass to NPM
Fraser Cormack [Wed, 8 Dec 2021 17:54:30 +0000 (17:54 +0000)]
[NewPM] Port FlattenCFGPass to NPM

Reviewed By: aeubanks

Differential Revision: https://reviews.llvm.org/D115361

2 years ago[mlir][memref] Fix subview offset verification.
Nicolas Vasilache [Thu, 9 Dec 2021 07:40:19 +0000 (07:40 +0000)]
[mlir][memref] Fix subview offset verification.

Offset-specific verification seems to have been lost in one of the recent refactorings.
Also add proper tests that would have caught this omission.

This addresses the immediate issues discussed in:
https://llvm.discourse.group/t/memref-subview-affine-map-and-symbols/4851

Differential Revision: https://reviews.llvm.org/D115427

2 years ago[NFC][Verifier] Remove checks for atomic loads/stores that alignment is non-zero
Arthur Eubanks [Wed, 8 Dec 2021 21:03:43 +0000 (13:03 -0800)]
[NFC][Verifier] Remove checks for atomic loads/stores that alignment is non-zero

The alignment is never 0 since getAlign() returns 1 << bits.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D115388

2 years ago[C++20] [Coroutines] Mark coroutine done if unhandled_exception throws
Chuanqi Xu [Thu, 9 Dec 2021 06:55:18 +0000 (14:55 +0800)]
[C++20] [Coroutines] Mark coroutine done if unhandled_exception throws

According to [dcl.fct.def.coroutine]/p14:
> If the evaluation of the expression promise.unhandled_­exception()
> exits via an exception, the coroutine is considered suspended at the
> final suspend point.

But this is not implemented in clang before. This patch would implement
this feature by marking the coroutine as done at the place of
coro.end(frame, /*InUnwindPath=*/true ).

Reviewed By: rjmccall

Differential Revision: https://reviews.llvm.org/D115219

2 years ago[RISCV] Fix vm operand constraint to fit GCC's behavior
Kito Cheng [Thu, 7 Oct 2021 12:38:37 +0000 (20:38 +0800)]
[RISCV] Fix vm operand constraint to fit GCC's behavior

- `vm` constraint is used for masking operand, which always v0.

- Update testcase, only masking operand should use `vm`, vector mask operations
  should just use `vr` for any vector register.

 - Revise the description of `vm` constraint.

- This patch also fix issue on RISCVRegisterInfo.td and RISCVISelLowering.cpp.

  RISCVRegisterInfo.td:
  - The first VT in the list must be the largest total size since the
    SelectionDAGBuilder uses the first register in the list as the canonical
    type for the register.

  RISCVISelLowering.cpp:
  - Fix RISCVTargetLowering::splitValueIntoRegisterParts and
    RISCVTargetLowering::joinRegisterPartsIntoValue for handling vectors
    with different total size, that will happened on fractional LMUL since
    fractional LMUL is always occupy one vector register.

Reviewed By: frasercrmck

Differential Revision: https://reviews.llvm.org/D112599

2 years ago[Coroutines] Remove unused coroutine builtin/intrinsics llvm.coro.param (NFC-ish)
Chuanqi Xu [Thu, 9 Dec 2021 06:37:05 +0000 (14:37 +0800)]
[Coroutines] Remove unused coroutine builtin/intrinsics llvm.coro.param (NFC-ish)

I found that the coroutine intrinsic llvm.coro.param in documentation
(https://llvm.org/docs/Coroutines.html#id101) didn't get used actually
since there isn't lowering codes in LLVM. I also checked the
implementation of libstdc++ and libc++. Both of them didn't use
llvm.coro.param. So I am pretty sure that the llvm.coro.param intrinsic
is unused. I think it would be better t to remove it to avoid possible
misleading understandings.

Note: according to [class.copy.elision]/p1.3, this optimization is
allowed by the C++ language specification. Let's make it someday.

Reviewed By: rjmccall

Differential Revision: https://reviews.llvm.org/D115222

2 years ago[mlir][Linalg] Bufferize the region of LinalgOps as well.
MaheshRavishankar [Thu, 9 Dec 2021 06:10:12 +0000 (22:10 -0800)]
[mlir][Linalg] Bufferize the region of LinalgOps as well.

The region of `linalg.generic` might contain `tensor` operations. For
example, current lowering of `gather` uses a `tensor.extract` in the
body of the `LinalgOp`. Bufferize the ops within a `LinalgOp` region
as well to catch such cases.

Differential Revision: https://reviews.llvm.org/D115322

2 years agotsan: fork runtime
Dmitry Vyukov [Thu, 9 Dec 2021 06:18:20 +0000 (07:18 +0100)]
tsan: fork runtime

Fork the current version of tsan runtime before commiting
rewrite of the runtime (D112603). The old runtime can be
enabled with TSAN_USE_OLD_RUNTIME option.
This is a temporal measure for emergencies and is required
for Chromium rollout (for context see http://crbug.com/1275581).
The old runtime is supposed to be deleted soon.

Reviewed By: thakis

Differential Revision: https://reviews.llvm.org/D115223

2 years ago[C++20 Modules] Don't create global module fragment for extern linkage declaration...
Chuanqi Xu [Thu, 9 Dec 2021 05:52:28 +0000 (13:52 +0800)]
[C++20 Modules] Don't create global module fragment for extern linkage declaration in GMF already

Previously we would create global module fragment for extern linkage
declaration which is alreday in global module fragment. However, it is
clearly redundant to do so. This patch would check if the extern linkage
declaration are already in GMF before we create a GMF for it.

2 years ago[NFC] Rename MachineFunction::cloneMachineInstrBundle (coding style)
Mircea Trofin [Thu, 9 Dec 2021 05:12:54 +0000 (21:12 -0800)]
[NFC] Rename MachineFunction::cloneMachineInstrBundle (coding style)

2 years ago[NFC] Rename MachineFunction::deleteMachineInstr (coding style)
Mircea Trofin [Thu, 9 Dec 2021 04:35:33 +0000 (20:35 -0800)]
[NFC] Rename MachineFunction::deleteMachineInstr (coding style)

2 years ago[llvm] Use range-based for loops (NFC)
Kazu Hirata [Thu, 9 Dec 2021 04:35:39 +0000 (20:35 -0800)]
[llvm] Use range-based for loops (NFC)

2 years ago[gn build] Port 059e03476cbb
LLVM GN Syncbot [Thu, 9 Dec 2021 04:11:22 +0000 (04:11 +0000)]
[gn build] Port 059e03476cbb

2 years ago[NFC][mlgo] Generalize model runner interface
Mircea Trofin [Tue, 7 Dec 2021 23:07:39 +0000 (15:07 -0800)]
[NFC][mlgo] Generalize model runner interface

This prepares it for the regalloc work. Part of it is making model
evaluation accross 'development' and 'release' scenarios more reusable.
This patch:
- extends support to tensors of any shape (not just scalars, like we had
in the inliner -Oz case). While the tensor shape can be anything, we
assume row-major layout and expose the tensor as a buffer.
- exposes the NoInferenceModelRunner, which we use in the 'development'
mode to keep the evaluation code path consistent and simplify logging,
as we'll want to reuse it in the regalloc case.

Differential Revision: https://reviews.llvm.org/D115306

2 years ago[CSKY] Complete codegen of basic arithmetic and load/store operations
Zi Xuan Wu [Thu, 9 Dec 2021 03:35:34 +0000 (11:35 +0800)]
[CSKY] Complete codegen of basic arithmetic and load/store operations

Complete basic arithmetic operations such as add/sub/mul/div, and it also includes converions
and some specific operations such as bswap.Add load/store patterns to generate different addressing mode instructions.

Also enable some infra such as copy physical register and eliminate frame index.

2 years ago[libc++][NFC] Remove test/support/tracked_value.h
Joe Loser [Thu, 9 Dec 2021 01:01:39 +0000 (20:01 -0500)]
[libc++][NFC] Remove test/support/tracked_value.h

No tests are using `libcxx/test/support/tracked_value.h`. So, remove it.

Differential Revision: https://reviews.llvm.org/D115411

2 years ago[NFC] Rename MachineFunction::DeleteMachineBasicBlock
Mircea Trofin [Thu, 9 Dec 2021 01:36:37 +0000 (17:36 -0800)]
[NFC] Rename MachineFunction::DeleteMachineBasicBlock

Renamed to conform to coding style

2 years ago[Support] [Debuginfod] Use libcurl imported library.
Noah Shutty [Thu, 9 Dec 2021 00:51:33 +0000 (00:51 +0000)]
[Support] [Debuginfod] Use libcurl imported library.

A [[ https://lists.llvm.org/pipermail/llvm-dev/2021-December/154203.html | report on llvm-dev ]] indicated that `curl.h` could be found during build configuration but not during compilation. This diff uses the libcurl imported library created by findcurl, so that includes and libs are automatically managed correctly by cmake.

Reviewed By: phosek

Differential Revision: https://reviews.llvm.org/D115189

2 years ago[PowerPC] copy byval parameter to caller's stack when needed
Chen Zheng [Sat, 9 Oct 2021 09:06:53 +0000 (09:06 +0000)]
[PowerPC] copy byval parameter to caller's stack when needed

Now we won't copy the byval parameter (bigger than 8 bytes) to
caller's parameter save area. Instead, we will only copy the
byval parameter when it can not be passed entirely in registers
which means we have to use parameter save area according to the
64 bit SVR4 ABI.

Reviewed By: jsji

Differential Revision: https://reviews.llvm.org/D111485

2 years ago[gn build] (manually) port ccf1469a4cdb (lldbVersion)
Nico Weber [Thu, 9 Dec 2021 00:45:17 +0000 (19:45 -0500)]
[gn build] (manually) port ccf1469a4cdb (lldbVersion)

2 years agoRevert "[asan] Run background thread for asan only on THUMB"
Andrew Browne [Thu, 9 Dec 2021 00:41:36 +0000 (16:41 -0800)]
Revert "[asan] Run background thread for asan only on THUMB"

This reverts commit 5c27740238007d22f2d0cd0ebe2aaffa90a9c92b.

Reverting due to Windows build issue:

sanitizer_stackdepot.cpp.obj : error LNK2005: "void __cdecl __sanitizer::StackDepotStopBackgroundThread(void)" (?StackDepotStopBackgroundThread@__sanitizer@@YAXXZ) already defined in sanitizer_common_libcdep.cpp.obj
LINK : fatal error LNK1181: cannot open input file 'projects\compiler-rt\lib\asan\CMakeFiles\RTAsan_dynamic.x86_64.dir\asan_rtl_x86_64.S.obj'

2 years agoSupport: Avoid using SmallVector::set_size() in zlib
Duncan P. N. Exon Smith [Thu, 9 Dec 2021 00:18:00 +0000 (16:18 -0800)]
Support: Avoid using SmallVector::set_size() in zlib

Stop using `SmallVector::set_size()` in zlib. Replace pairs of
`reserve()` / `set_size()` with `resize_for_overwrite()` and
`truncate()`.

Differential Revision: https://reviews.llvm.org/D115391

2 years agoSupport: Avoid using SmallVector::set_size() in sys::path
Duncan P. N. Exon Smith [Wed, 8 Dec 2021 01:22:44 +0000 (17:22 -0800)]
Support: Avoid using SmallVector::set_size() in sys::path

Stop using `SmallVector::set_size()` in sys::path APIs. In both cases,
use `truncate()` instead.

Differential Revision: https://reviews.llvm.org/D115391

2 years ago[gn build] (manually) port f75cce0be861
Nico Weber [Thu, 9 Dec 2021 00:20:52 +0000 (19:20 -0500)]
[gn build] (manually) port f75cce0be861

2 years ago[libc] Add a .clang-tidy file for the toplevel libc directory.
Siva Chandra Reddy [Wed, 8 Dec 2021 07:02:48 +0000 (07:02 +0000)]
[libc] Add a .clang-tidy file for the toplevel libc directory.

Generation of the .yaml has been removed to prevent lint from
running with every ninja invocation. The new .clang-tidy file is copied
to the libc build directory so that generated files also get checked.

Reviewed By: michaelrj

Differential Revision: https://reviews.llvm.org/D115405

2 years agoRevert "[ASan] Shared optimized callbacks implementation."
Kirill Stoimenov [Thu, 9 Dec 2021 00:16:14 +0000 (00:16 +0000)]
Revert "[ASan] Shared optimized callbacks implementation."

This reverts commit f71c553a30cc52c0b4a6abbaa82ce97c30c13979.

Reviewed By: kstoimenov

Differential Revision: https://reviews.llvm.org/D115407

2 years agoADT: Make StringRef::size() and StringRef::empty() constexpr
Duncan P. N. Exon Smith [Wed, 8 Dec 2021 21:41:31 +0000 (13:41 -0800)]
ADT: Make StringRef::size() and StringRef::empty() constexpr

This unblocks using `StringLiteral::size()` for a SmallVector size in
another patch.

Differential Revision: https://reviews.llvm.org/D115395

2 years ago[gn build] (manually) port f71c553a30cc
Nico Weber [Thu, 9 Dec 2021 00:10:53 +0000 (19:10 -0500)]
[gn build] (manually) port f71c553a30cc

2 years ago[ASan] Shared optimized callbacks implementation.
Kirill Stoimenov [Wed, 17 Nov 2021 22:39:48 +0000 (22:39 +0000)]
[ASan] Shared optimized callbacks implementation.

This change moves optimized callbacks from each .o file to compiler-rt. Instead of using code generation it uses direct assembly implementation. Please note that the 'or' version is not implemented and it will produce unresolved external if somehow 'or' version is requested.

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D114558

2 years agoAMDGPU: Mark scc defs dead in SGPR to VMEM path for no free SGPRs
Matt Arsenault [Wed, 8 Dec 2021 17:46:27 +0000 (12:46 -0500)]
AMDGPU: Mark scc defs dead in SGPR to VMEM path for no free SGPRs

This introduces verifier errors into this broken situation which we do
not handle correctly, which is better than being silently
miscompiled. For the emergency stack slot, the scavenger likes to move
the restore instruction as late as possible, which ends up separating
the SCC def from the conditional branch.

2 years agoAMDGPU: Simplify test for SGPR spilling bug
Matt Arsenault [Wed, 8 Dec 2021 17:44:40 +0000 (12:44 -0500)]
AMDGPU: Simplify test for SGPR spilling bug

Remove the control flow from the test from
25eb7fa01d7ebbe67648ea03841cda55b4239ab2. It's not necessary to
reproduce the original assert with the patch reverted. The control
flow happens to expose a different issue that calls for a separate
test in a future change.

2 years agoAMDGPU: Mark SCC def as dead when expanding frame indexes
Matt Arsenault [Tue, 7 Dec 2021 23:24:29 +0000 (18:24 -0500)]
AMDGPU: Mark SCC def as dead when expanding frame indexes

This improves liveness queries in a future change.

2 years ago[RISCV] Improve tracking of EndLoc in the assembly parser.
Craig Topper [Wed, 8 Dec 2021 23:13:40 +0000 (15:13 -0800)]
[RISCV] Improve tracking of EndLoc in the assembly parser.

The SMLoc::getFromPointer(S.getPointer() - 1) pattern used in
several place didn't make sense since that puts the End before the
Start location.

This patch corrects this to properly calculate the end location as
we parse. Unsure how much this matters, a lot of these are for custom
operand parsing. If the custom parsing succeeds, the instruction
matching probably won't fail, so the end loc won't be used to build
a range for a diagnostic.

I've also fixed the creation functions for the CSR and VType operands
to assign the EndLoc of the RISCVOperand class using the StartLoc
instead of an empty SMLoc. I don't think these will ever be accessed,
but it makes the code look less like we forgot to assign it. This is
consistent with some operands on the ARM target.

Reviewed By: luismarques

Differential Revision: https://reviews.llvm.org/D115192

2 years ago[lldb/Target] Slide source display for artificial locations at function start
Med Ismail Bennani [Wed, 8 Dec 2021 01:49:44 +0000 (17:49 -0800)]
[lldb/Target] Slide source display for artificial locations at function start

It can happen that a line entry reports that some source code is located
at line 0. In DWARF, line 0 is a special location which indicates that
code has no 1-1 mapping with source.

When stopping in one of those artificial locations, lldb doesn't know which
line to display and shows the beginning of the file instead.

This patch mitigates this behaviour by checking if the current symbol context
of the line entry has a matching function, in which case, it slides the
source listing to the start of that function.

This patch also shows the user a warning explaining why lldb couldn't
show sources at that location.

rdar://83118425

Differential Revision: https://reviews.llvm.org/D115313

Signed-off-by: Med Ismail Bennani <medismail.bennani@gmail.com>
2 years agoOpenMP: Avoid using SmallVector::set_size()
Duncan P. N. Exon Smith [Wed, 8 Dec 2021 01:22:37 +0000 (17:22 -0800)]
OpenMP: Avoid using SmallVector::set_size()

Update `OpenMPIRBuilder::collapseLoops()` to call `resize()` instead of
`set_size()`. The latter asserts on capacity limits and cannot grow,
which seems likely to be unintentional here (if it is, I think a local
assertion would be good for clarity).

Also update `CodeGenFunction::EmitOMPCollapsedCanonicalLoopNest()` to
use `pop_back_n()` instead of `set_size()`.

Differential Revision: https://reviews.llvm.org/D115378

2 years ago[lldb] Make lldbVersion a full fledged library
Jonas Devlieghere [Wed, 8 Dec 2021 20:23:57 +0000 (12:23 -0800)]
[lldb] Make lldbVersion a full fledged library

Because of its dependency on clang (and potentially other compilers
downstream, such as swift) lldb_private::GetVersion already lives in its
own library called lldbBase. Despite that, its implementation was spread
across unrelated files. This patch improves things by introducing a
Version library with its own directory, header and implementation file.

The benefits of this patch include:

 - We can get rid of the ugly quoting macros.
 - Other parts of LLDB can read the version number from
   lldb/Version/Version.inc.
 - The implementation can be swapped out for tools like lldb-server than
   don't need to depend on clang at all.

Differential revision: https://reviews.llvm.org/D115211

2 years ago[OpenMP][AMDGPU] Switch host-device memory copy to asynchronous version
Carlo Bertolli [Wed, 8 Dec 2021 23:02:19 +0000 (23:02 +0000)]
[OpenMP][AMDGPU] Switch host-device memory copy to asynchronous version

Prepare amdgpu plugin for asynchronous implementation. This patch switches to using HSA API for asynchronous memory copy.
Moving away from hsa_memory_copy means that plugin is responsible for locking/unlocking host memory pointers.

Reviewed By: JonChesterfield

Differential Revision: https://reviews.llvm.org/D115279

2 years ago[dfsan] Add a flag to ignore personality routines.
Taewook Oh [Wed, 8 Dec 2021 04:56:28 +0000 (20:56 -0800)]
[dfsan] Add a flag to ignore personality routines.

This diff adds "dfsan-ignore-personality-routine" flag, which makes
the dfsan pass to not to generate wrappers for the personality functions if the
function is marked uninstrumented.

This flag is to support dfsan with the cases where the exception handling
routines cannot be instrumented (e.g. use the prebuilt version of c++ standard
library). When the personality function cannot be instrumented it is supposed
to be marked "uninstrumented" from the abi list file. While DFSan generates a
wrapper function for uninstrumented functions, it cannot cannot generate a
valid wrapper for vararg functions, and indirect invocation of vararg function
wrapper terminates the execution of dfsan-instrumented programs. This makes
invocation of personality routine to crash the program, because 1) clang adds a
declaration of personality functions as a vararg function with no fixed
argument, and 2) personality routines are always called indirectly.

To address this issue, the flag introduced in this diff makes dfsan to not to
instrument the personality function. This is not the "correct" solution in the
sense that return value label from the personality function will be undefined.
However, in practice, if the exception handling routines are uninstrumented we
wouldn't expect precise label propagation around them, and it would be more
beneficial to make the rest of the program run without termination.

Reviewed By: browneee

Differential Revision: https://reviews.llvm.org/D115317

2 years ago[mlir] Added ctlz and cttz to math dialect and LLVM dialect
Rob Suderman [Wed, 8 Dec 2021 22:32:09 +0000 (14:32 -0800)]
[mlir] Added ctlz and cttz to math dialect and LLVM dialect

Count leading/trailing zeros are an existing LLVM intrinsic. Added LLVM
support for the intrinsics with lowerings from the math dialect to LLVM
dialect.

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D115206

2 years ago[llvm-profgen] remove check Attributes to fix build failure
wlei [Wed, 8 Dec 2021 22:12:53 +0000 (14:12 -0800)]
[llvm-profgen] remove check Attributes to fix build failure

2 years agoRevert "A new hidden option test-changed=exe that calls exe after each time IR changes"
Arthur Eubanks [Wed, 8 Dec 2021 21:57:25 +0000 (13:57 -0800)]
Revert "A new hidden option test-changed=exe that calls exe after each time IR changes"

This reverts commit f9235e45fd1f5ca21f95105427184a6afd0f9d95.

Causes breakages on Windows: http://45.33.8.238/win/50453/step_11.txt.

2 years agoADT: Add SmallVectorImpl::truncate() to replace uses of set_size()
Duncan P. N. Exon Smith [Wed, 8 Dec 2021 00:20:18 +0000 (16:20 -0800)]
ADT: Add SmallVectorImpl::truncate() to replace uses of set_size()

Add `SmallVectorImpl::truncate()`, a variant of `resize()` that cannot
increase the size.

- Compared to `resize()`, this has no code path for growing the
  allocation and can be better optimized.
- Compared to `set_size()`, this formally calls destructors, and does
  not skip any constructors.
- Compared to `pop_back_n()`, this takes the new desired size, which in
  many contexts is more intuitive than the number of elements to remove.

The immediate motivation is to pair this with `resize_for_overwrite()`
to remove uses of `set_size()`, which can then be made private.

Differential Revision: https://reviews.llvm.org/D115383

2 years agoUpdate sink instruction testcase
Anna Thomas [Wed, 8 Dec 2021 21:53:09 +0000 (16:53 -0500)]
Update sink instruction testcase

Add inferable writeonly attribute. Simplify testcase.

2 years ago[libc++] [test] Refactor range.prim/empty.pass.cpp.
Arthur O'Dwyer [Thu, 7 Oct 2021 00:12:27 +0000 (20:12 -0400)]
[libc++] [test] Refactor range.prim/empty.pass.cpp.

No decrease in test coverage intended. The original goal here
was just to get rid of the global name `sentinel` so that we can
rename the `sentinel_wrapper` in "test_iterators.h" to `sentinel`;
but then I took a closer look at the offending tests and saw
that some of them probably weren't testing what they intended.

Also, add one `/*explicit*/` and one #if'ed out test indicating
bugs in the current ranges::empty (to be fixed by D115312 or
some equivalent patch).

Reviewed as part of D115272.

2 years ago[libc++] [test] Use sized_sentinel<int*> in range.prim/ssize.pass.cpp.
Arthur O'Dwyer [Thu, 7 Oct 2021 00:23:10 +0000 (20:23 -0400)]
[libc++] [test] Use sized_sentinel<int*> in range.prim/ssize.pass.cpp.

Reviewed as part of D115272.

2 years ago[libc++] [test_iterators] Make all ADL base() functions into hidden friends. NFCI.
Arthur O'Dwyer [Tue, 7 Dec 2021 19:08:23 +0000 (14:08 -0500)]
[libc++] [test_iterators] Make all ADL base() functions into hidden friends. NFCI.

This follows up on my addition of base(cpp20_input_iterator) in D115177,
making all the ADL base() functions consistent.
Also align cpp20_input_iterator with the other test iterators' style.

Reviewed as part of D115272.

2 years ago[lit] Pass the COMSPEC variable through to test processes on Windows
Martin Storsjö [Wed, 8 Dec 2021 18:42:40 +0000 (20:42 +0200)]
[lit] Pass the COMSPEC variable through to test processes on Windows

This variable is necessary for the system() function for running
external processes.

This is needes for some libcxx tests. With the current libcxx test
infrastructure, all OS environment variables are passed through, but
with the new "from scratch" libcxx test setup, we only pass through
the variables listed here.

Differential Revision: https://reviews.llvm.org/D115363

2 years ago[llvm-profgen] fix build failure in cs-extbinary.test
wlei [Wed, 8 Dec 2021 21:18:16 +0000 (13:18 -0800)]
[llvm-profgen] fix build failure in cs-extbinary.test

2 years ago[ASan] Added __cplusplus guard around the C++ code so it is safe to include asan_mapp...
Kirill Stoimenov [Wed, 8 Dec 2021 20:16:14 +0000 (20:16 +0000)]
[ASan] Added __cplusplus guard around the C++ code so it is safe to include asan_mapping.h in assembly files.

This will avoid addind asan_shadow_defines.h in D114558.

Reviewed By: vitalybuka

Differential Revision: https://reviews.llvm.org/D115348

2 years agoUpdate with additional tests for sinking calls to uses
Anna Thomas [Mon, 6 Dec 2021 21:31:02 +0000 (16:31 -0500)]
Update with additional tests for sinking calls to uses

Precommitted from D109917.
Note that the log call is already handled by instCombine.

2 years ago[sanitizer] Run Stack compression in background thread
Vitaly Buka [Tue, 23 Nov 2021 05:24:10 +0000 (21:24 -0800)]
[sanitizer] Run Stack compression in background thread

Depends on D114495.

Reviewed By: dvyukov

Differential Revision: https://reviews.llvm.org/D114498

2 years ago[llvm-profgen] fix to use profile-summary-hot-count instead of profile-summary-cold...
wlei [Wed, 8 Dec 2021 20:53:19 +0000 (12:53 -0800)]
[llvm-profgen] fix to use profile-summary-hot-count instead of profile-summary-cold-count for CS profile

2 years ago[compiler-rt] Use the runtimes build for custom libc++
Petr Hosek [Wed, 1 Dec 2021 09:07:14 +0000 (01:07 -0800)]
[compiler-rt] Use the runtimes build for custom libc++

Some of the compiler-rt runtimes use custom instrumented libc++ build.
Use the runtimes build for building this custom libc++.

Differential Revision: https://reviews.llvm.org/D114922

2 years agoADT: Reduce nesting in resize_for_overwrite(), NFC
Duncan P. N. Exon Smith [Wed, 8 Dec 2021 00:16:45 +0000 (16:16 -0800)]
ADT: Reduce nesting in resize_for_overwrite(), NFC

Use an early return in SmallVectorImpl::resize_for_overwite() to reduce
nesting. This also makes it easier to visually compare against the
two-argument version of resize().

2 years ago[asan] Run background thread for asan only on THUMB
Vitaly Buka [Wed, 8 Dec 2021 20:28:12 +0000 (12:28 -0800)]
[asan] Run background thread for asan only on THUMB

As in D114934, or lsan crashes on the same bot.

2 years ago[NFC][sanitizer] Move/rename macro into sanitizer_platform.h
Vitaly Buka [Wed, 8 Dec 2021 19:53:14 +0000 (11:53 -0800)]
[NFC][sanitizer] Move/rename macro into sanitizer_platform.h

Folloup for D114934

2 years ago[compiler-rt] Disabled tests and benchmarks for libc++
Vitaly Buka [Wed, 8 Dec 2021 20:22:47 +0000 (12:22 -0800)]
[compiler-rt] Disabled tests and benchmarks for libc++

We don't run tests or benchmarks from this build anyway.
Benchmarks in custom libc++ break my local build.

Reviewed By: ldionne

Differential Revision: https://reviews.llvm.org/D115375

2 years ago[llvm-profgen] Fix total samples related issues
wlei [Fri, 3 Dec 2021 00:51:42 +0000 (16:51 -0800)]
[llvm-profgen] Fix total samples related issues

Since total sample and body sample are used to compute hotness threshold in compiler, we found in some services changing the total samples computation will cause noticeable regression. Hence, here we will revert the changes and just keep all total samples number identical to the old tool.

Three changes in this diff:

1. Revert previous diff(https://reviews.llvm.org/D112672: [llvm-profgen] Update total samples by accumulating all its body samples) and put it under a switch.

2. Keep the negative line number. Although compiler doesn't consume the count but it will be used to compute hot threshold.

3. Change to accumulate total samples per byte instead of per instruction.

Reviewed By: hoy, wenlei

Differential Revision: https://reviews.llvm.org/D115013

2 years ago[InstCombine] Add tests for D114272
Fabian Wolff [Wed, 8 Dec 2021 20:28:06 +0000 (21:28 +0100)]
[InstCombine] Add tests for D114272

2 years ago[instcombine] A couple style tweaks to visitExtractElementInst [nfc]
Philip Reames [Wed, 8 Dec 2021 20:21:10 +0000 (12:21 -0800)]
[instcombine] A couple style tweaks to visitExtractElementInst [nfc]

2 years ago[llvm-profgen] Trim cold function profiles for non-CS AutoFDO
wlei [Mon, 29 Nov 2021 07:43:11 +0000 (23:43 -0800)]
[llvm-profgen] Trim cold function profiles for non-CS AutoFDO

This change allows to trim the profile if it's considered to be cold for baseline AutoFDO. We reuse the cold threshold from `ProfileSummaryBuilder::getColdCountThreshold(..)` which can be set by percent(--profile-summary-cutoff-cold) or by value(--profile-summary-cold-count).

Reviewed By: hoy, wenlei

Differential Revision: https://reviews.llvm.org/D113785

2 years ago[ASan] Ran clang-format on asan_mapping.h.
Kirill Stoimenov [Wed, 8 Dec 2021 19:47:12 +0000 (19:47 +0000)]
[ASan] Ran clang-format on asan_mapping.h.

Reviewed By: kstoimenov

Differential Revision: https://reviews.llvm.org/D115371

2 years ago[libc][Obvious] Fix variable naming in the generic sqrt implementations.
Siva Chandra Reddy [Wed, 8 Dec 2021 07:03:35 +0000 (07:03 +0000)]
[libc][Obvious] Fix variable naming in the generic sqrt implementations.

2 years ago[MemoryLocation] Support memset_pattern{4,8} in getForArgument.
Florian Hahn [Wed, 8 Dec 2021 19:39:45 +0000 (19:39 +0000)]
[MemoryLocation] Support memset_pattern{4,8} in getForArgument.

memset_pattern{4,8} behave as memset_pattern16, with the only difference
being the size of the pattern location.

Reviewed By: ab

Differential Revision: https://reviews.llvm.org/D114905

2 years agoA new hidden option test-changed=exe that calls exe after each time IR changes
Jamie Schmeiser [Wed, 8 Dec 2021 19:22:54 +0000 (14:22 -0500)]
A new hidden option test-changed=exe that calls exe after each time IR changes

Summary:
A new option test-changed is defined that allows one to specify an
exe that is called after each pass in the opt pipeline that changes the IR.
The test-changed=exe option saves the IR in a temporary file and calls exe
with the name of the file and the name of the pass that just changed it after
each pass alters the IR. exe is also called with the initial IR. This
can be used, for example, to determine which pass corrupts the IR by having
exe as a script that calls llc and runs a test to see after which pass the
results change. The print-changed filtering options are respected.

Author: Jamie Schmeiser <schmeise@ca.ibm.com>
Reviewed By: aeubanks (Arthur Eubanks)
Differential Revision: https://reviews.llvm.org/D110776

2 years ago[mlir][spirv] math.erf OpenCL lowering
Butygin [Thu, 28 Oct 2021 16:04:35 +0000 (19:04 +0300)]
[mlir][spirv] math.erf OpenCL lowering

Differential Revision: https://reviews.llvm.org/D115335

2 years agoRevert "[gn build] (manually) port 02cc8d698c49"
Nico Weber [Wed, 8 Dec 2021 18:55:57 +0000 (13:55 -0500)]
Revert "[gn build] (manually) port 02cc8d698c49"

This reverts commit 0f865dc6941b5bd2b2d891adda514a165b99db41.
02cc8d698c49 was reverted in aaec63d2a7dbab5.

2 years agoRevert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."
Noah Shutty [Wed, 8 Dec 2021 18:49:12 +0000 (18:49 +0000)]
Revert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."

This reverts commit 02cc8d698c4941f8f0120ea1a5d7205fb33a312d because it
caused buildbot failures. The issue appears to be simply that we need to
only enable debuginfod when the HTTPClient has been initialized by the
running tool, since InitLLVM does not do the initialization step anymore.

2 years ago[CostModel][X86] Split MUL/SDIV+SREM/UDIV+UREM PowerOf2 handling. NFC.
Simon Pilgrim [Wed, 8 Dec 2021 18:44:45 +0000 (18:44 +0000)]
[CostModel][X86] Split MUL/SDIV+SREM/UDIV+UREM PowerOf2 handling. NFC.

This is a NFC cleanup to simplify some upcoming refactoring required to address the regressions in D111968.

2 years ago[gn build] (manually) port 02cc8d698c49
Nico Weber [Wed, 8 Dec 2021 18:45:54 +0000 (13:45 -0500)]
[gn build] (manually) port 02cc8d698c49

2 years agoSilence format string warning harder.
Benjamin Kramer [Wed, 8 Dec 2021 18:36:33 +0000 (19:36 +0100)]
Silence format string warning harder.

This can be unsigned long or unsigned long long depending on where it's
compiled. Use the ugly portable way.
PlatformWindows.cpp:397:63: warning: format specifies type 'unsigned long long' but the argument has type 'uint64_t' (aka 'unsigned long')

2 years agoDon't offer partial fix-its for `modernize-pass-by-value`
Adrian Vogelsgesang [Wed, 8 Dec 2021 18:31:30 +0000 (13:31 -0500)]
Don't offer partial fix-its for `modernize-pass-by-value`

This commit improves the fix-its of modernize-pass-by-value by
no longer proposing partial fixes. In the presence of using/typedef,
we failed to rewrite the function signature but still adjusted the
function body. This led to incorrect, partial fix-its. Instead, the
check now simply doesn't offer any fixes at all in such a situation.

2 years ago[libc++] `= delete` member functions with // = delete;
Nikolas Klauser [Wed, 8 Dec 2021 09:57:12 +0000 (10:57 +0100)]
[libc++] `= delete` member functions with // = delete;

Use `= delete` for member functions that are marked with `// = delete;`

Reviewed By: ldionne, Quuxplusone, #libc

Spies: jloser, libcxx-commits

Differential Revision: https://reviews.llvm.org/D115291

2 years ago[unroll] Add test coverage for loops with small estimated trip counts and multiple...
Philip Reames [Wed, 8 Dec 2021 18:02:17 +0000 (10:02 -0800)]
[unroll] Add test coverage for loops with small estimated trip counts and multiple exits

2 years ago[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.
Noah Shutty [Wed, 8 Dec 2021 17:52:40 +0000 (17:52 +0000)]
[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D113717

2 years ago[mlir][nvvm] Add async copy ops to nvvm dialect
Thomas Raoux [Wed, 8 Dec 2021 03:28:14 +0000 (19:28 -0800)]
[mlir][nvvm] Add async copy ops to nvvm dialect

Differential Revision: https://reviews.llvm.org/D115314

2 years agoRevert "[compiler-rt] Use the runtimes build for custom libc++"
Petr Hosek [Wed, 8 Dec 2021 17:13:03 +0000 (09:13 -0800)]
Revert "[compiler-rt] Use the runtimes build for custom libc++"

This reverts commit bda3f2dd763b06427dd2713aa062e356c9d7bdc6 since
it broke MSan tests.

2 years ago[clangd] Suppress IncludeCleaner warnings for headers behind pragma keep
Kirill Bobyrev [Wed, 8 Dec 2021 17:07:39 +0000 (18:07 +0100)]
[clangd] Suppress IncludeCleaner warnings for headers behind pragma keep

D114072 allows filtering out the warnings for headers behind `// IWYU pragma:
keep`. This is the first step towards more useful IWYU pragmas support and
fine-grained control over the IncludeCleaner warnings.

Reviewed By: kadircet

Differential Revision: https://reviews.llvm.org/D115345

2 years ago[NFC][CodeGen] Remove rarely used DL variable from SelectionDAGBuilder
David Sherwood [Tue, 23 Nov 2021 17:15:53 +0000 (17:15 +0000)]
[NFC][CodeGen] Remove rarely used DL variable from SelectionDAGBuilder

There is a pointer to the DataLayout in SelectionDAGBuilder called
'DL' that is hardly ever used. In most cases the code seems to just
use `DAG.getDataLayout()` instead. Given that DL is also often used
as a shadowed variable for the debug location it seems sensible to
just kill off the few remaining uses and be consistent with the rest
of the code.

Differential Revision: https://reviews.llvm.org/D114451