[utils][filecheck-lint] Add filecheck-lint, a linter for FileCheck directives.
authorBenjamin Chetioui <bchetioui@google.com>
Thu, 12 Jan 2023 13:19:42 +0000 (13:19 +0000)
committerBenjamin Chetioui <bchetioui@google.com>
Thu, 12 Jan 2023 13:20:02 +0000 (13:20 +0000)
commitf702c75995db9ea3978d39e3f3885b2113356f68
treeae0534099e5e4a41e6fbf06aaaa3ebcca7e4812c
parenta74bc436703057ab0f53ad78ab3f84227a45e37f
[utils][filecheck-lint] Add filecheck-lint, a linter for FileCheck directives.

FileCheck is a versatile tool for testing compiler output regardless of syntax.
FileCheck achieves this versatility by processing only input lines in which it
detects a valid CHECK directive, and ignoring everything else.

This comes at a price: if a directive is not typed properly, it is not
processed by FileCheck, and the test code it precedes is effectively disabled.
This results in the illusion that the code is tested, while the test may have
actually never run.

This scenario is not hypothetical, see the fixes introduced in, e.g.
https://github.com/tensorflow/tensorflow/commit/48cacf049f3d6ed3f289ccc48ec50491b6d8d9a8,
https://reviews.llvm.org/D139698, https://reviews.llvm.org/D139636,
https://github.com/iree-org/iree/pull/11693.

The findings corrected in the above changes originate in filecheck-lint.
In a given test file, filecheck-lint uses the edit distance between anything
that looks like a FileCheck directive and the set of locally valid directives
to detect likely misspelled directives.

The tool is not yet feature complete---e.g. it does not parse custom comment
prefixes to exclude them from reporting---but it already yields useful results,
as demonstrated above.

Differential Revision: https://reviews.llvm.org/D141508
llvm/utils/filecheck_lint/filecheck_lint.py [new file with mode: 0644]
llvm/utils/filecheck_lint/filecheck_lint_test.py [new file with mode: 0644]