From: Endre Fülöp Date: Wed, 10 Jun 2020 06:59:04 +0000 (+0200) Subject: [analyzer] On-demand parsing capability for CTU X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=5cc18516c4839fccc64b54eaa5aa447a8e1ed8fa;p=platform%2Fupstream%2Fllvm.git [analyzer] On-demand parsing capability for CTU Summary: Introduce on-demand parsing of needed ASTs during CTU analysis. The index-file format is extended, and analyzer-option CTUInvocationList is added to specify the exact invocations needed to parse the needed source-files. Reviewers: martong, balazske, Szelethus, xazax.hun, whisperity Reviewed By: martong, xazax.hun Subscribers: gribozavr2, thakis, ASDenysPetrov, ormris, mgorny, whisperity, xazax.hun, baloghadamsoftware, szepet, rnkovacs, a.sidorin, mikhail.ramalho, Szelethus, donat.nagy, dkrupp, Charusso, steakhal, cfe-commits Tags: #clang Differential Revision: https://reviews.llvm.org/D75665 --- diff --git a/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst b/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst index 86f972b..36be82f 100644 --- a/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst +++ b/clang/docs/analyzer/user-docs/CrossTranslationUnit.rst @@ -3,14 +3,35 @@ Cross Translation Unit (CTU) Analysis ===================================== Normally, static analysis works in the boundary of one translation unit (TU). -However, with additional steps and configuration we can enable the analysis to inline the definition of a function from another TU. +However, with additional steps and configuration we can enable the analysis to inline the definition of a function from +another TU. .. contents:: :local: -Manual CTU Analysis -------------------- +Overview +________ +CTU analysis can be used in a variety of ways. The importing of external TU definitions can work with pre-dumped PCH +files or generating the necessary AST structure on-demand, during the analysis of the main TU. Driving the static +analysis can also be implemented in multiple ways. The most direct way is to specify the necessary commandline options +of the Clang frontend manually (and generate the prerequisite dependencies of the specific import method by hand). This +process can be automated by other tools, like `CodeChecker `_ and scan-build-py +(preference for the former). + +PCH-based analysis +__________________ +The analysis needs the PCH dumps of all the translations units used in the project. +These can be generated by the Clang Frontend itself, and must be arranged in a specific way in the filesystem. +The index, which maps symbols' USR names to PCH dumps containing them must also be generated by the +`clang-extdef-mapping`. Entries in the index *must* have an `.ast` suffix if the goal +is to use PCH-based analysis, as the lack of that extension signals that the entry is to be used as a source-file, and parsed on-demand. +This tool uses a :doc:`compilation database <../../JSONCompilationDatabase>` to +determine the compilation flags used. +The analysis invocation must be provided with the directory which contains the dumps and the mapping files. + +Manual CTU Analysis +################### Let's consider these source files in our minimal example: .. code-block:: cpp @@ -47,7 +68,8 @@ And a compilation database: ] We'd like to analyze `main.cpp` and discover the division by zero bug. -In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file of `foo.cpp`: +In order to be able to inline the definition of `foo` from `foo.cpp` first we have to generate the `AST` (or `PCH`) file +of `foo.cpp`: .. code-block:: bash @@ -58,7 +80,8 @@ In order to be able to inline the definition of `foo` from `foo.cpp` first we ha compile_commands.json foo.cpp.ast foo.cpp main.cpp $ -The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the source files: +The next step is to create a CTU index file which holds the `USR` name and location of external definitions in the +source files: .. code-block:: bash @@ -85,47 +108,33 @@ We have to feed Clang with CTU specific extra arguments: $ pwd /path/to/your/project - $ clang++ --analyze -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true -Xclang -analyzer-config -Xclang ctu-dir=. -Xclang -analyzer-output=plist-multi-file main.cpp + $ clang++ --analyze \ + -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \ + -Xclang -analyzer-config -Xclang ctu-dir=. \ + -Xclang -analyzer-output=plist-multi-file \ + main.cpp main.cpp:5:12: warning: Division by zero return 3 / foo(); ~~^~~~~~~ 1 warning generated. $ # The plist file with the result is generated. - $ ls + $ ls -F compile_commands.json externalDefMap.txt foo.ast foo.cpp foo.cpp.ast main.cpp main.plist $ -This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use `CodeChecker` or `scan-build-py`. +This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use +`CodeChecker` or `scan-build-py`. Automated CTU Analysis with CodeChecker ---------------------------------------- +####################################### The `CodeChecker `_ project fully supports automated CTU analysis with Clang. Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes: .. code-block:: bash $ CodeChecker analyze --ctu compile_commands.json -o reports - [INFO 2019-07-16 17:21] - Pre-analysis started. - [INFO 2019-07-16 17:21] - Collecting data for ctu analysis. - [INFO 2019-07-16 17:21] - [1/2] foo.cpp - [INFO 2019-07-16 17:21] - [2/2] main.cpp - [INFO 2019-07-16 17:21] - Pre-analysis finished. - [INFO 2019-07-16 17:21] - Starting static analysis ... - [INFO 2019-07-16 17:21] - [1/2] clangsa analyzed foo.cpp successfully. - [INFO 2019-07-16 17:21] - [2/2] clangsa analyzed main.cpp successfully. - [INFO 2019-07-16 17:21] - ----==== Summary ====---- - [INFO 2019-07-16 17:21] - Successfully analyzed - [INFO 2019-07-16 17:21] - clangsa: 2 - [INFO 2019-07-16 17:21] - Total analyzed compilation commands: 2 - [INFO 2019-07-16 17:21] - ----=================---- - [INFO 2019-07-16 17:21] - Analysis finished. - [INFO 2019-07-16 17:21] - To view results in the terminal use the "CodeChecker parse" command. - [INFO 2019-07-16 17:21] - To store results use the "CodeChecker store" command. - [INFO 2019-07-16 17:21] - See --help and the user guide for further options about parsing and storing the reports. - [INFO 2019-07-16 17:21] - ----=================---- - [INFO 2019-07-16 17:21] - Analysis length: 0.659618854523 sec. - $ ls - compile_commands.json foo.cpp foo.cpp.ast main.cpp reports + $ ls -F + compile_commands.json foo.cpp foo.cpp.ast main.cpp reports/ $ tree reports reports ├── compile_cmd.json @@ -174,9 +183,9 @@ Or we can use `CodeChecker parse -e html` to export the results into HTML format $ firefox html_out/index.html Automated CTU Analysis with scan-build-py (don't do it) -------------------------------------------------------- -We actively develop CTU with CodeChecker as a "runner" script, `scan-build-py` is not actively developed for CTU. -`scan-build-py` has various errors and issues, expect it to work with the very basic projects only. +############################################################# +We actively develop CTU with CodeChecker as the driver for this feature, `scan-build-py` is not actively developed for CTU. +`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only. Example usage of scan-build-py: @@ -191,3 +200,176 @@ Example usage of scan-build-py: Opening in existing browser session. ^C $ + +On-demand analysis +__________________ +The analysis produces the necessary AST structure of external TUs during analysis. This requires the +exact compiler invocations for each TU, which can be generated by hand, or by tools driving the analyzer. +The compiler invocation is a shell command that could be used to compile the TU-s main source file. +The mapping from absolute source file paths of a TU to lists of compilation command segments used to +compile said TU are given in YAML format referred to as `invocation list`, and must be passed as an +analyer-config argument. +The index, which maps function USR names to source files containing them must also be generated by the +`clang-extdef-mapping`. Entries in the index must *not* have an `.ast` suffix if the goal +is to use On-demand analysis, as that extension signals that the entry is to be used as an PCH-dump. +The mapping of external definitions implicitly uses a +:doc:`compilation database <../../JSONCompilationDatabase>` to determine the compilation flags used. +The analysis invocation must be provided with the directory which contains the mapping +files, and the `invocation list` which is used to determine compiler flags. + + +Manual CTU Analysis +################### + +Let's consider these source files in our minimal example: + +.. code-block:: cpp + + // main.cpp + int foo(); + + int main() { + return 3 / foo(); + } + +.. code-block:: cpp + + // foo.cpp + int foo() { + return 0; + } + +The compilation database: + +.. code-block:: bash + + [ + { + "directory": "/path/to/your/project", + "command": "clang++ -c foo.cpp -o foo.o", + "file": "foo.cpp" + }, + { + "directory": "/path/to/your/project", + "command": "clang++ -c main.cpp -o main.o", + "file": "main.cpp" + } + ] + +The `invocation list`: + +.. code-block:: bash + + "/path/to/your/project/foo.cpp": + - "clang++" + - "-c" + - "/path/to/your/project/foo.cpp" + - "-o" + - "/path/to/your/project/foo.o" + + "/path/to/your/project/main.cpp": + - "clang++" + - "-c" + - "/path/to/your/project/main.cpp" + - "-o" + - "/path/to/your/project/main.o" + +We'd like to analyze `main.cpp` and discover the division by zero bug. +As we are using On-demand mode, we only need to create a CTU index file which holds the `USR` name and location of +external definitions in the source files: + +.. code-block:: bash + + $ clang-extdef-mapping -p . foo.cpp + c:@F@foo# /path/to/your/project/foo.cpp + $ clang-extdef-mapping -p . foo.cpp > externalDefMap.txt + +Now everything is available for the CTU analysis. +We have to feed Clang with CTU specific extra arguments: + +.. code-block:: bash + + $ pwd + /path/to/your/project + $ clang++ --analyze \ + -Xclang -analyzer-config -Xclang experimental-enable-naive-ctu-analysis=true \ + -Xclang -analyzer-config -Xclang ctu-dir=. \ + -Xclang -analyzer-config -Xclang ctu-invocation-list=invocations.yaml \ + -Xclang -analyzer-output=plist-multi-file \ + main.cpp + main.cpp:5:12: warning: Division by zero + return 3 / foo(); + ~~^~~~~~~ + 1 warning generated. + $ # The plist file with the result is generated. + $ ls -F + compile_commands.json externalDefMap.txt foo.cpp main.cpp main.plist + $ + +This manual procedure is error-prone and not scalable, therefore to analyze real projects it is recommended to use +`CodeChecker` or `scan-build-py`. + +Automated CTU Analysis with CodeChecker +####################################### +The `CodeChecker `_ project fully supports automated CTU analysis with Clang. +Once we have set up the `PATH` environment variable and we activated the python `venv` then it is all it takes: + +.. code-block:: bash + + $ CodeChecker analyze --ctu --ctu-ast-loading-mode on-demand compile_commands.json -o reports + $ ls -F + compile_commands.json foo.cpp main.cpp reports/ + $ tree reports + reports + ├── compile_cmd.json + ├── compiler_info.json + ├── foo.cpp_53f6fbf7ab7ec9931301524b551959e2.plist + ├── main.cpp_23db3d8df52ff0812e6e5a03071c8337.plist + ├── metadata.json + └── unique_compile_commands.json + + 0 directories, 6 files + $ + +The `plist` files contain the results of the analysis, which may be viewed with the regular analysis tools. +E.g. one may use `CodeChecker parse` to view the results in command line: + +.. code-block:: bash + + $ CodeChecker parse reports + [HIGH] /home/egbomrt/ctu_mini_raw_project/main.cpp:5:12: Division by zero [core.DivideZero] + return 3 / foo(); + ^ + + Found 1 defect(s) in main.cpp + + + ----==== Summary ====---- + ----------------------- + Filename | Report count + ----------------------- + main.cpp | 1 + ----------------------- + ----------------------- + Severity | Report count + ----------------------- + HIGH | 1 + ----------------------- + ----=================---- + Total number of reports: 1 + ----=================---- + +Or we can use `CodeChecker parse -e html` to export the results into HTML format: + +.. code-block:: bash + + $ CodeChecker parse -e html -o html_out reports + $ firefox html_out/index.html + +Automated CTU Analysis with scan-build-py (don't do it) +####################################################### +We actively develop CTU with CodeChecker as the driver for feature, `scan-build-py` is not actively developed for CTU. +`scan-build-py` has various errors and issues, expect it to work only with the very basic projects only. + +Currently On-demand analysis is not supported with `scan-build-py`. + diff --git a/clang/include/clang/CrossTU/CrossTranslationUnit.h b/clang/include/clang/CrossTU/CrossTranslationUnit.h index 4d2b710..027c6f1 100644 --- a/clang/include/clang/CrossTU/CrossTranslationUnit.h +++ b/clang/include/clang/CrossTU/CrossTranslationUnit.h @@ -21,6 +21,7 @@ #include "llvm/ADT/SmallPtrSet.h" #include "llvm/ADT/StringMap.h" #include "llvm/Support/Error.h" +#include "llvm/Support/Path.h" namespace clang { class CompilerInstance; @@ -47,7 +48,12 @@ enum class index_error_code { triple_mismatch, lang_mismatch, lang_dialect_mismatch, - load_threshold_reached + load_threshold_reached, + invocation_list_ambiguous, + invocation_list_file_not_found, + invocation_list_empty, + invocation_list_wrong_format, + invocation_list_lookup_unsuccessful }; class IndexError : public llvm::ErrorInfo { @@ -78,7 +84,8 @@ private: }; /// This function parses an index file that determines which -/// translation unit contains which definition. +/// translation unit contains which definition. The IndexPath is not prefixed +/// with CTUDir, so an absolute path is expected for consistent results. /// /// The index file format is the following: /// each line consists of an USR and a filepath separated by a space. @@ -86,17 +93,27 @@ private: /// \return Returns a map where the USR is the key and the filepath is the value /// or an error. llvm::Expected> -parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir); +parseCrossTUIndex(StringRef IndexPath); std::string createCrossTUIndexString(const llvm::StringMap &Index); +using InvocationListTy = llvm::StringMap>; +/// Parse the YAML formatted invocation list file content \p FileContent. +/// The format is expected to be a mapping from from absolute source file +/// paths in the filesystem to a list of command-line parts, which +/// constitute the invocation needed to compile that file. That invocation +/// will be used to produce the AST of the TU. +llvm::Expected parseInvocationList( + StringRef FileContent, + llvm::sys::path::Style PathStyle = llvm::sys::path::Style::posix); + // Returns true if the variable or any field of a record variable is const. bool containsConst(const VarDecl *VD, const ASTContext &ACtx); /// This class is used for tools that requires cross translation /// unit capability. /// -/// This class can load definitions from external AST files. +/// This class can load definitions from external AST sources. /// The loaded definition will be merged back to the original AST using the /// AST Importer. /// In order to use this class, an index file is required that describes @@ -116,7 +133,7 @@ public: /// the current translation unit. A function definition with the same /// declaration will be looked up in the index file which should be in the /// \p CrossTUDir directory, called \p IndexName. In case the declaration is - /// found in the index the corresponding AST file will be loaded and the + /// found in the index the corresponding AST will be loaded and the /// definition will be merged into the original AST using the AST Importer. /// /// \return The declaration with the definition will be returned. @@ -136,7 +153,7 @@ public: /// A definition with the same declaration will be looked up in the /// index file which should be in the \p CrossTUDir directory, called /// \p IndexName. In case the declaration is found in the index the - /// corresponding AST file will be loaded. If the number of TUs imported + /// corresponding AST will be loaded. If the number of TUs imported /// reaches \p CTULoadTreshold, no loading is performed. /// /// \return Returns a pointer to the ASTUnit that contains the definition of @@ -209,14 +226,43 @@ private: /// imported the FileID. ImportedFileIDMap ImportedFileIDs; - /// Functor for loading ASTUnits from AST-dump files. - class ASTFileLoader { + using LoadResultTy = llvm::Expected>; + + /// Loads ASTUnits from AST-dumps or source-files. + class ASTLoader { public: - ASTFileLoader(const CompilerInstance &CI); - std::unique_ptr operator()(StringRef ASTFilePath); + ASTLoader(CompilerInstance &CI, StringRef CTUDir, + StringRef InvocationListFilePath); + + /// Load the ASTUnit by its identifier found in the index file. If the + /// indentifier is suffixed with '.ast' it is considered a dump. Otherwise + /// it is treated as source-file, and on-demand parsed. Relative paths are + /// prefixed with CTUDir. + LoadResultTy load(StringRef Identifier); + + /// Lazily initialize the invocation list information, which is needed for + /// on-demand parsing. + llvm::Error lazyInitInvocationList(); private: - const CompilerInstance &CI; + /// The style used for storage and lookup of filesystem paths. + /// Defaults to posix. + const llvm::sys::path::Style PathStyle = llvm::sys::path::Style::posix; + + /// Loads an AST from a pch-dump. + LoadResultTy loadFromDump(StringRef Identifier); + /// Loads an AST from a source-file. + LoadResultTy loadFromSource(StringRef Identifier); + + CompilerInstance &CI; + StringRef CTUDir; + /// The path to the file containing the invocation list, which is in YAML + /// format, and contains a mapping from source files to compiler invocations + /// that produce the AST used for analysis. + StringRef InvocationListFilePath; + /// In case of on-demand parsing, the invocations for parsing the source + /// files is stored. + llvm::Optional InvocationList; }; /// Maintain number of AST loads and check for reaching the load limit. @@ -242,7 +288,7 @@ private: /// are the concerns of ASTUnitStorage class. class ASTUnitStorage { public: - ASTUnitStorage(const CompilerInstance &CI); + ASTUnitStorage(CompilerInstance &CI); /// Loads an ASTUnit for a function. /// /// \param FunctionName USR name of the function. @@ -287,18 +333,17 @@ private: using IndexMapTy = BaseMapTy; IndexMapTy NameFileMap; - ASTFileLoader FileAccessor; + /// Loads the AST based on the identifier found in the index. + ASTLoader Loader; - /// Limit the number of loaded ASTs. Used to limit the memory usage of the - /// CrossTranslationUnitContext. - /// The ASTUnitStorage has the knowledge about if the AST to load is - /// actually loaded or returned from cache. This information is needed to - /// maintain the counter. + /// Limit the number of loaded ASTs. It is used to limit the memory usage + /// of the CrossTranslationUnitContext. The ASTUnitStorage has the + /// information whether the AST to load is actually loaded or returned from + /// cache. This information is needed to maintain the counter. ASTLoadGuard LoadGuard; }; ASTUnitStorage ASTStorage; - }; } // namespace cross_tu diff --git a/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def b/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def index 597a65c..8944dfe 100644 --- a/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def +++ b/clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def @@ -378,10 +378,25 @@ ANALYZER_OPTION(StringRef, CTUDir, "ctu-dir", "The directory containing the CTU related files.", "") ANALYZER_OPTION(StringRef, CTUIndexName, "ctu-index-name", - "the name of the file containing the CTU index of definitions.", + "The name of the file containing the CTU index of definitions. " + "The index file maps USR-names to identifiers. An identifier " + "can end with an '.ast' suffix, indicating the indentifier is " + "a path to a pch-dump. Otherwise the identifier is regarded as " + "path to a source file which is parsed on-demand. Relative " + "paths are prefixed with ctu-dir, absolute paths are used " + "unmodified during lookup.", "externalDefMap.txt") ANALYZER_OPTION( + StringRef, CTUInvocationList, "ctu-invocation-list", + "The path to the YAML format file containing a mapping from source file " + "paths to command-line invocations represented as a list of arguments. " + "This invocation is used produce the source-file's AST in case on-demand " + "loading is performed. Example file-content: " + "{/main.cpp: [clang++, /main.cpp], other.cpp: [clang++, /other.cpp]}", + "invocations.yaml") + +ANALYZER_OPTION( StringRef, ModelPath, "model-path", "The analyzer can inline an alternative implementation written in C at the " "call site if the called function's body is not available. This is a path " diff --git a/clang/lib/CrossTU/CrossTranslationUnit.cpp b/clang/lib/CrossTU/CrossTranslationUnit.cpp index 689c988..24c3143 100644 --- a/clang/lib/CrossTU/CrossTranslationUnit.cpp +++ b/clang/lib/CrossTU/CrossTranslationUnit.cpp @@ -18,14 +18,19 @@ #include "clang/Frontend/CompilerInstance.h" #include "clang/Frontend/TextDiagnosticPrinter.h" #include "clang/Index/USRGeneration.h" -#include "llvm/ADT/Triple.h" +#include "llvm/ADT/Optional.h" #include "llvm/ADT/Statistic.h" +#include "llvm/ADT/Triple.h" +#include "llvm/Option/ArgList.h" #include "llvm/Support/ErrorHandling.h" #include "llvm/Support/ManagedStatic.h" #include "llvm/Support/Path.h" +#include "llvm/Support/YAMLParser.h" #include "llvm/Support/raw_ostream.h" +#include #include #include +#include namespace clang { namespace cross_tu { @@ -110,6 +115,17 @@ public: return "Language dialect mismatch"; case index_error_code::load_threshold_reached: return "Load threshold reached"; + case index_error_code::invocation_list_ambiguous: + return "Invocation list file contains multiple references to the same " + "source file."; + case index_error_code::invocation_list_file_not_found: + return "Invocation list file is not found."; + case index_error_code::invocation_list_empty: + return "Invocation list file is empty."; + case index_error_code::invocation_list_wrong_format: + return "Invocation list file is in wrong format."; + case index_error_code::invocation_list_lookup_unsuccessful: + return "Invocation list file does not contain the requested source file."; } llvm_unreachable("Unrecognized index_error_code."); } @@ -129,7 +145,7 @@ std::error_code IndexError::convertToErrorCode() const { } llvm::Expected> -parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir) { +parseCrossTUIndex(StringRef IndexPath) { std::ifstream ExternalMapFile{std::string(IndexPath)}; if (!ExternalMapFile) return llvm::make_error(index_error_code::missing_index_file, @@ -139,21 +155,26 @@ parseCrossTUIndex(StringRef IndexPath, StringRef CrossTUDir) { std::string Line; unsigned LineNo = 1; while (std::getline(ExternalMapFile, Line)) { - const size_t Pos = Line.find(" "); - if (Pos > 0 && Pos != std::string::npos) { - StringRef LineRef{Line}; - StringRef LookupName = LineRef.substr(0, Pos); - if (Result.count(LookupName)) + StringRef LineRef{Line}; + const size_t Delimiter = LineRef.find(" "); + if (Delimiter > 0 && Delimiter != std::string::npos) { + StringRef LookupName = LineRef.substr(0, Delimiter); + + // Store paths with posix-style directory separator. + SmallVector FilePath; + llvm::Twine{LineRef.substr(Delimiter + 1)}.toVector(FilePath); + llvm::sys::path::native(FilePath, llvm::sys::path::Style::posix); + + bool InsertionOccured; + std::tie(std::ignore, InsertionOccured) = + Result.try_emplace(LookupName, FilePath.begin(), FilePath.end()); + if (!InsertionOccured) return llvm::make_error( index_error_code::multiple_definitions, IndexPath.str(), LineNo); - StringRef FileName = LineRef.substr(Pos + 1); - SmallString<256> FilePath = CrossTUDir; - llvm::sys::path::append(FilePath, FileName); - Result[LookupName] = std::string(FilePath); } else return llvm::make_error( index_error_code::invalid_index_format, IndexPath.str(), LineNo); - LineNo++; + ++LineNo; } return Result; } @@ -341,30 +362,11 @@ void CrossTranslationUnitContext::emitCrossTUDiagnostics(const IndexError &IE) { } } -CrossTranslationUnitContext::ASTFileLoader::ASTFileLoader( - const CompilerInstance &CI) - : CI(CI) {} - -std::unique_ptr -CrossTranslationUnitContext::ASTFileLoader::operator()(StringRef ASTFilePath) { - // Load AST from ast-dump. - IntrusiveRefCntPtr DiagOpts = new DiagnosticOptions(); - TextDiagnosticPrinter *DiagClient = - new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts); - IntrusiveRefCntPtr DiagID(new DiagnosticIDs()); - IntrusiveRefCntPtr Diags( - new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient)); - - return ASTUnit::LoadFromASTFile( - std::string(ASTFilePath), CI.getPCHContainerOperations()->getRawReader(), - ASTUnit::LoadEverything, Diags, CI.getFileSystemOpts()); -} - CrossTranslationUnitContext::ASTUnitStorage::ASTUnitStorage( - const CompilerInstance &CI) - : FileAccessor(CI), LoadGuard(const_cast(CI) - .getAnalyzerOpts() - ->CTUImportThreshold) {} + CompilerInstance &CI) + : Loader(CI, CI.getAnalyzerOpts()->CTUDir, + CI.getAnalyzerOpts()->CTUInvocationList), + LoadGuard(CI.getAnalyzerOpts()->CTUImportThreshold) {} llvm::Expected CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFile( @@ -380,8 +382,12 @@ CrossTranslationUnitContext::ASTUnitStorage::getASTUnitForFile( index_error_code::load_threshold_reached); } - // Load the ASTUnit from the pre-dumped AST file specified by ASTFileName. - std::unique_ptr LoadedUnit = FileAccessor(FileName); + auto LoadAttempt = Loader.load(FileName); + + if (!LoadAttempt) + return LoadAttempt.takeError(); + + std::unique_ptr LoadedUnit = std::move(LoadAttempt.get()); // Need the raw pointer and the unique_ptr as well. ASTUnit *Unit = LoadedUnit.get(); @@ -461,7 +467,7 @@ llvm::Error CrossTranslationUnitContext::ASTUnitStorage::ensureCTUIndexLoaded( else llvm::sys::path::append(IndexFile, IndexName); - if (auto IndexMapping = parseCrossTUIndex(IndexFile, CrossTUDir)) { + if (auto IndexMapping = parseCrossTUIndex(IndexFile)) { // Initialize member map. NameFileMap = *IndexMapping; return llvm::Error::success(); @@ -494,6 +500,193 @@ llvm::Expected CrossTranslationUnitContext::loadExternalAST( return Unit; } +CrossTranslationUnitContext::ASTLoader::ASTLoader( + CompilerInstance &CI, StringRef CTUDir, StringRef InvocationListFilePath) + : CI(CI), CTUDir(CTUDir), InvocationListFilePath(InvocationListFilePath) {} + +CrossTranslationUnitContext::LoadResultTy +CrossTranslationUnitContext::ASTLoader::load(StringRef Identifier) { + llvm::SmallString<256> Path; + if (llvm::sys::path::is_absolute(Identifier, PathStyle)) { + Path = Identifier; + } else { + Path = CTUDir; + llvm::sys::path::append(Path, PathStyle, Identifier); + } + + // The path is stored in the InvocationList member in posix style. To + // successfully lookup an entry based on filepath, it must be converted. + llvm::sys::path::native(Path, PathStyle); + + // Normalize by removing relative path components. + llvm::sys::path::remove_dots(Path, /*remove_dot_dot*/ true, PathStyle); + + if (Path.endswith(".ast")) + return loadFromDump(Path); + else + return loadFromSource(Path); +} + +CrossTranslationUnitContext::LoadResultTy +CrossTranslationUnitContext::ASTLoader::loadFromDump(StringRef ASTDumpPath) { + IntrusiveRefCntPtr DiagOpts = new DiagnosticOptions(); + TextDiagnosticPrinter *DiagClient = + new TextDiagnosticPrinter(llvm::errs(), &*DiagOpts); + IntrusiveRefCntPtr DiagID(new DiagnosticIDs()); + IntrusiveRefCntPtr Diags( + new DiagnosticsEngine(DiagID, &*DiagOpts, DiagClient)); + return ASTUnit::LoadFromASTFile( + std::string(ASTDumpPath.str()), + CI.getPCHContainerOperations()->getRawReader(), ASTUnit::LoadEverything, + Diags, CI.getFileSystemOpts()); +} + +/// Load the AST from a source-file, which is supposed to be located inside the +/// YAML formatted invocation list file under the filesystem path specified by +/// \p InvocationList. The invocation list should contain absolute paths. +/// \p SourceFilePath is the absolute path of the source file that contains the +/// function definition the analysis is looking for. The Index is built by the +/// \p clang-extdef-mapping tool, which is also supposed to be generating +/// absolute paths. +/// +/// Proper diagnostic emission requires absolute paths, so even if a future +/// change introduces the handling of relative paths, this must be taken into +/// consideration. +CrossTranslationUnitContext::LoadResultTy +CrossTranslationUnitContext::ASTLoader::loadFromSource( + StringRef SourceFilePath) { + + if (llvm::Error InitError = lazyInitInvocationList()) + return std::move(InitError); + assert(InvocationList); + + auto Invocation = InvocationList->find(SourceFilePath); + if (Invocation == InvocationList->end()) + return llvm::make_error( + index_error_code::invocation_list_lookup_unsuccessful); + + const InvocationListTy::mapped_type &InvocationCommand = Invocation->second; + + SmallVector CommandLineArgs(InvocationCommand.size()); + std::transform(InvocationCommand.begin(), InvocationCommand.end(), + CommandLineArgs.begin(), + [](auto &&CmdPart) { return CmdPart.c_str(); }); + + IntrusiveRefCntPtr DiagOpts{&CI.getDiagnosticOpts()}; + auto *DiagClient = new ForwardingDiagnosticConsumer{CI.getDiagnosticClient()}; + IntrusiveRefCntPtr DiagID{ + CI.getDiagnostics().getDiagnosticIDs()}; + IntrusiveRefCntPtr Diags( + new DiagnosticsEngine{DiagID, &*DiagOpts, DiagClient}); + + return std::unique_ptr(ASTUnit::LoadFromCommandLine( + CommandLineArgs.begin(), (CommandLineArgs.end()), + CI.getPCHContainerOperations(), Diags, + CI.getHeaderSearchOpts().ResourceDir)); +} + +llvm::Expected +parseInvocationList(StringRef FileContent, llvm::sys::path::Style PathStyle) { + InvocationListTy InvocationList; + + /// LLVM YAML parser is used to extract information from invocation list file. + llvm::SourceMgr SM; + llvm::yaml::Stream InvocationFile(FileContent, SM); + + /// Only the first document is processed. + llvm::yaml::document_iterator FirstInvocationFile = InvocationFile.begin(); + + /// There has to be at least one document available. + if (FirstInvocationFile == InvocationFile.end()) + return llvm::make_error( + index_error_code::invocation_list_empty); + + llvm::yaml::Node *DocumentRoot = FirstInvocationFile->getRoot(); + if (!DocumentRoot) + return llvm::make_error( + index_error_code::invocation_list_wrong_format); + + /// According to the format specified the document must be a mapping, where + /// the keys are paths to source files, and values are sequences of invocation + /// parts. + auto *Mappings = dyn_cast(DocumentRoot); + if (!Mappings) + return llvm::make_error( + index_error_code::invocation_list_wrong_format); + + for (auto &NextMapping : *Mappings) { + /// The keys should be strings, which represent a source-file path. + auto *Key = dyn_cast(NextMapping.getKey()); + if (!Key) + return llvm::make_error( + index_error_code::invocation_list_wrong_format); + + SmallVector ValueStorage; + StringRef SourcePath = Key->getValue(ValueStorage); + + // Store paths with PathStyle directory separator. + SmallVector NativeSourcePath; + llvm::Twine{SourcePath}.toVector(NativeSourcePath); + llvm::sys::path::native(NativeSourcePath, PathStyle); + + StringRef InvocationKey{NativeSourcePath.begin(), NativeSourcePath.size()}; + + if (InvocationList.find(InvocationKey) != InvocationList.end()) + return llvm::make_error( + index_error_code::invocation_list_ambiguous); + + /// The values should be sequences of strings, each representing a part of + /// the invocation. + auto *Args = dyn_cast(NextMapping.getValue()); + if (!Args) + return llvm::make_error( + index_error_code::invocation_list_wrong_format); + + for (auto &Arg : *Args) { + auto *CmdString = dyn_cast(&Arg); + if (!CmdString) + return llvm::make_error( + index_error_code::invocation_list_wrong_format); + /// Every conversion starts with an empty working storage, as it is not + /// clear if this is a requirement of the YAML parser. + ValueStorage.clear(); + InvocationList[InvocationKey].emplace_back( + CmdString->getValue(ValueStorage)); + } + + if (InvocationList[InvocationKey].empty()) + return llvm::make_error( + index_error_code::invocation_list_wrong_format); + } + + return InvocationList; +} + +llvm::Error CrossTranslationUnitContext::ASTLoader::lazyInitInvocationList() { + /// Lazily initialize the invocation list member used for on-demand parsing. + if (InvocationList) + return llvm::Error::success(); + + llvm::ErrorOr> FileContent = + llvm::MemoryBuffer::getFile(InvocationListFilePath); + if (!FileContent) + return llvm::make_error( + index_error_code::invocation_list_file_not_found); + std::unique_ptr ContentBuffer = std::move(*FileContent); + assert(ContentBuffer && "If no error was produced after loading, the pointer " + "should not be nullptr."); + + llvm::Expected ExpectedInvocationList = + parseInvocationList(ContentBuffer->getBuffer(), PathStyle); + + if (!ExpectedInvocationList) + return ExpectedInvocationList.takeError(); + + InvocationList = *ExpectedInvocationList; + + return llvm::Error::success(); +} + template llvm::Expected CrossTranslationUnitContext::importDefinitionImpl(const T *D, ASTUnit *Unit) { diff --git a/clang/test/Analysis/Inputs/ctu-other.c b/clang/test/Analysis/Inputs/ctu-other.c index 82d29c6..16ebb35 100644 --- a/clang/test/Analysis/Inputs/ctu-other.c +++ b/clang/test/Analysis/Inputs/ctu-other.c @@ -31,10 +31,12 @@ int g(struct S *ctx) { } // Test that asm import does not fail. +// TODO: Support the GNU extension asm keyword as well. +// Example using the GNU extension: asm("mov $42, %0" : "=r"(res)); int inlineAsm() { int res; - asm("mov $42, %0" - : "=r"(res)); + __asm__("mov $42, %0" + : "=r"(res)); return res; } diff --git a/clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.txt b/clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.ast-dump.txt similarity index 100% rename from clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.txt rename to clang/test/Analysis/Inputs/ctu-other.c.externalDefMap.ast-dump.txt diff --git a/clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.txt b/clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt similarity index 100% rename from clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.txt rename to clang/test/Analysis/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt diff --git a/clang/test/Analysis/analyzer-config.c b/clang/test/Analysis/analyzer-config.c index cb3d406..1d4a963 100644 --- a/clang/test/Analysis/analyzer-config.c +++ b/clang/test/Analysis/analyzer-config.c @@ -43,6 +43,7 @@ // CHECK-NEXT: ctu-dir = "" // CHECK-NEXT: ctu-import-threshold = 100 // CHECK-NEXT: ctu-index-name = externalDefMap.txt +// CHECK-NEXT: ctu-invocation-list = invocations.yaml // CHECK-NEXT: deadcode.DeadStores:ShowFixIts = false // CHECK-NEXT: deadcode.DeadStores:WarnForDeadNestedAssignments = true // CHECK-NEXT: debug.AnalysisOrder:* = false diff --git a/clang/test/Analysis/ctu-different-triples.cpp b/clang/test/Analysis/ctu-different-triples.cpp index 20acc31..68c7b0d 100644 --- a/clang/test/Analysis/ctu-different-triples.cpp +++ b/clang/test/Analysis/ctu-different-triples.cpp @@ -2,7 +2,7 @@ // RUN: mkdir -p %t/ctudir // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \ // RUN: -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp -// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt +// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt // RUN: %clang_analyze_cc1 -std=c++14 -triple powerpc64-montavista-linux-gnu \ // RUN: -analyzer-checker=core,debug.ExprInspection \ // RUN: -analyzer-config experimental-enable-naive-ctu-analysis=true \ diff --git a/clang/test/Analysis/ctu-main.c b/clang/test/Analysis/ctu-main.c index 114d694..d991eb7 100644 --- a/clang/test/Analysis/ctu-main.c +++ b/clang/test/Analysis/ctu-main.c @@ -2,7 +2,7 @@ // RUN: mkdir -p %t/ctudir2 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu \ // RUN: -emit-pch -o %t/ctudir2/ctu-other.c.ast %S/Inputs/ctu-other.c -// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.txt %t/ctudir2/externalDefMap.txt +// RUN: cp %S/Inputs/ctu-other.c.externalDefMap.ast-dump.txt %t/ctudir2/externalDefMap.txt // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \ // RUN: -analyzer-checker=core,debug.ExprInspection \ // RUN: -analyzer-config experimental-enable-naive-ctu-analysis=true \ @@ -50,6 +50,10 @@ void testMacro(void) { void testImplicit() { int res = identImplicit(6); // external implicit functions are not inlined clang_analyzer_eval(res == 6); // expected-warning{{TRUE}} + // Call something with uninitialized from the same function in which the implicit was called. + // This is necessary to reproduce a special bug in NoStoreFuncVisitor. + int uninitialized; + h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}} } // Tests the import of functions that have a struct parameter diff --git a/clang/test/Analysis/ctu-main.cpp b/clang/test/Analysis/ctu-main.cpp index 3f095a0..d76b398 100644 --- a/clang/test/Analysis/ctu-main.cpp +++ b/clang/test/Analysis/ctu-main.cpp @@ -4,7 +4,7 @@ // RUN: -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \ // RUN: -emit-pch -o %t/ctudir/ctu-chain.cpp.ast %S/Inputs/ctu-chain.cpp -// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt +// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \ // RUN: -analyzer-checker=core,debug.ExprInspection \ // RUN: -analyzer-config experimental-enable-naive-ctu-analysis=true \ diff --git a/clang/test/Analysis/ctu-on-demand-parsing.c b/clang/test/Analysis/ctu-on-demand-parsing.c new file mode 100644 index 0000000..5adce7f --- /dev/null +++ b/clang/test/Analysis/ctu-on-demand-parsing.c @@ -0,0 +1,83 @@ +// RUN: rm -rf %t +// RUN: mkdir -p %t +// RUN: cp "%s" "%t/ctu-on-demand-parsing.c" +// RUN: cp "%S/Inputs/ctu-other.c" "%t/ctu-other.c" +// +// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file. +// compile_commands.json is only needed for extdef_mapping, not for the analysis itself. +// RUN: echo '[{"directory":"%t","command":"gcc -std=c89 -Wno-visibility ctu-other.c","file":"ctu-other.c"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json +// +// RUN: echo '"%t/ctu-other.c": ["gcc", "-std=c89", "-Wno-visibility", "ctu-other.c"]' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml +// +// RUN: cd "%t" && %clang_extdef_map "%t/ctu-other.c" > externalDefMap.txt +// +// RUN: cd "%t" && %clang_cc1 -fsyntax-only -std=c89 -analyze \ +// RUN: -analyzer-checker=core,debug.ExprInspection \ +// RUN: -analyzer-config experimental-enable-naive-ctu-analysis=true \ +// RUN: -analyzer-config ctu-dir=. \ +// RUN: -analyzer-config ctu-invocation-list=invocations.yaml \ +// RUN: -verify ctu-on-demand-parsing.c +// +// FIXME: Path handling should work on all platforms. +// REQUIRES: linux + +void clang_analyzer_eval(int); + +// Test typedef and global variable in function. +typedef struct { + int a; + int b; +} FooBar; +extern FooBar fb; +int f(int); +void testGlobalVariable() { + clang_analyzer_eval(f(5) == 1); // expected-warning{{TRUE}} +} + +// Test enums. +int enumCheck(void); +enum A { x, + y, + z }; +void testEnum() { + clang_analyzer_eval(x == 0); // expected-warning{{TRUE}} + clang_analyzer_eval(enumCheck() == 42); // expected-warning{{TRUE}} +} + +// Test that asm import does not fail. +int inlineAsm(); +int testInlineAsm() { return inlineAsm(); } + +// Test reporting error in a macro. +struct S; +int g(struct S *); +void testMacro(void) { + g(0); + // expected-warning@ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}} +} + +// The external function prototype is incomplete. +// warning:implicit functions are prohibited by c99 +void testImplicit() { + int res = identImplicit(6); // external implicit functions are not inlined + clang_analyzer_eval(res == 6); // expected-warning{{TRUE}} + // Call something with uninitialized from the same function in which the + // implicit was called. This is necessary to reproduce a special bug in + // NoStoreFuncVisitor. + int uninitialized; + h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}} +} + +// Tests the import of functions that have a struct parameter +// defined in its prototype. +struct DataType { + int a; + int b; +}; +int structInProto(struct DataType *d); +void testStructDefInArgument() { + struct DataType d; + d.a = 1; + d.b = 0; + clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}} +} diff --git a/clang/test/Analysis/ctu-on-demand-parsing.cpp b/clang/test/Analysis/ctu-on-demand-parsing.cpp new file mode 100644 index 0000000..0582696 --- /dev/null +++ b/clang/test/Analysis/ctu-on-demand-parsing.cpp @@ -0,0 +1,111 @@ +// RUN: rm -rf %t +// RUN: mkdir -p %t/Inputs +// RUN: cp %s %t/ctu-on-demand-parsing.cpp +// RUN: cp %S/ctu-hdr.h %t/ctu-hdr.h +// RUN: cp %S/Inputs/ctu-chain.cpp %t/Inputs/ctu-chain.cpp +// RUN: cp %S/Inputs/ctu-other.cpp %t/Inputs/ctu-other.cpp +// +// Path substitutions on Windows platform could contain backslashes. These are escaped in the json file. +// compile_commands.json is only needed for the extdef_mapping, not for the analysis itself. +// RUN: echo '[{"directory":"%t/Inputs","command":"clang++ ctu-chain.cpp","file":"ctu-chain.cpp"},{"directory":"%t/Inputs","command":"clang++ ctu-other.cpp","file":"ctu-other.cpp"}]' | sed -e 's/\\/\\\\/g' > %t/compile_commands.json +// +// RUN: echo '{"%t/Inputs/ctu-chain.cpp": ["g++", "%t/Inputs/ctu-chain.cpp"], "%t/Inputs/ctu-other.cpp": ["g++", "%t/Inputs/ctu-other.cpp"]}' | sed -e 's/\\/\\\\/g' > %t/invocations.yaml +// +// RUN: cd "%t" && %clang_extdef_map Inputs/ctu-chain.cpp Inputs/ctu-other.cpp > externalDefMap.txt +// +// RUN: cd "%t" && %clang_analyze_cc1 \ +// RUN: -analyzer-checker=core,debug.ExprInspection \ +// RUN: -analyzer-config experimental-enable-naive-ctu-analysis=true \ +// RUN: -analyzer-config ctu-dir=. \ +// RUN: -analyzer-config ctu-invocation-list=invocations.yaml \ +// RUN: -verify ctu-on-demand-parsing.cpp +// RUN: cd "%t" && %clang_analyze_cc1 \ +// RUN: -analyzer-checker=core,debug.ExprInspection \ +// RUN: -analyzer-config experimental-enable-naive-ctu-analysis=true \ +// RUN: -analyzer-config ctu-dir=. \ +// RUN: -analyzer-config ctu-invocation-list=invocations.yaml \ +// RUN: -analyzer-config display-ctu-progress=true ctu-on-demand-parsing.cpp 2>&1 | FileCheck %t/ctu-on-demand-parsing.cpp +// +// CHECK: CTU loaded AST file: {{.*}}ctu-other.cpp +// CHECK: CTU loaded AST file: {{.*}}ctu-chain.cpp +// +// FIXME: Path handling should work on all platforms. +// REQUIRES: linux + +#include "ctu-hdr.h" + +void clang_analyzer_eval(int); + +int f(int); +int g(int); +int h(int); + +int callback_to_main(int x) { return x + 1; } + +namespace myns { +int fns(int x); + +namespace embed_ns { +int fens(int x); +} + +class embed_cls { +public: + int fecl(int x); +}; +} // namespace myns + +class mycls { +public: + int fcl(int x); + virtual int fvcl(int x); + static int fscl(int x); + + class embed_cls2 { + public: + int fecl2(int x); + }; +}; + +class derived : public mycls { +public: + virtual int fvcl(int x) override; +}; + +namespace chns { +int chf1(int x); +} + +int fun_using_anon_struct(int); +int other_macro_diag(int); + +void test_virtual_functions(mycls *obj) { + // The dynamic type is known. + clang_analyzer_eval(mycls().fvcl(1) == 8); // expected-warning{{TRUE}} + clang_analyzer_eval(derived().fvcl(1) == 9); // expected-warning{{TRUE}} + // We cannot decide about the dynamic type. + clang_analyzer_eval(obj->fvcl(1) == 8); // expected-warning{{FALSE}} expected-warning{{TRUE}} + clang_analyzer_eval(obj->fvcl(1) == 9); // expected-warning{{FALSE}} expected-warning{{TRUE}} +} + +int main() { + clang_analyzer_eval(f(3) == 2); // expected-warning{{TRUE}} + clang_analyzer_eval(f(4) == 3); // expected-warning{{TRUE}} + clang_analyzer_eval(f(5) == 3); // expected-warning{{FALSE}} + clang_analyzer_eval(g(4) == 6); // expected-warning{{TRUE}} + clang_analyzer_eval(h(2) == 8); // expected-warning{{TRUE}} + + clang_analyzer_eval(myns::fns(2) == 9); // expected-warning{{TRUE}} + clang_analyzer_eval(myns::embed_ns::fens(2) == -1); // expected-warning{{TRUE}} + clang_analyzer_eval(mycls().fcl(1) == 6); // expected-warning{{TRUE}} + clang_analyzer_eval(mycls::fscl(1) == 7); // expected-warning{{TRUE}} + clang_analyzer_eval(myns::embed_cls().fecl(1) == -6); // expected-warning{{TRUE}} + clang_analyzer_eval(mycls::embed_cls2().fecl2(0) == -11); // expected-warning{{TRUE}} + + clang_analyzer_eval(chns::chf1(4) == 12); // expected-warning{{TRUE}} + clang_analyzer_eval(fun_using_anon_struct(8) == 8); // expected-warning{{TRUE}} + + clang_analyzer_eval(other_macro_diag(1) == 1); // expected-warning{{TRUE}} + // expected-warning@Inputs/ctu-other.cpp:93{{REACHABLE}} + MACRODIAG(); // expected-warning{{REACHABLE}} +} diff --git a/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp b/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp index 6bcbd70..f41f07d 100644 --- a/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp +++ b/clang/test/Analysis/ctu-unknown-parts-in-triples.cpp @@ -5,7 +5,7 @@ // RUN: mkdir -p %t/ctudir // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \ // RUN: -emit-pch -o %t/ctudir/ctu-other.cpp.ast %S/Inputs/ctu-other.cpp -// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.txt %t/ctudir/externalDefMap.txt +// RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-unknown-linux-gnu \ // RUN: -analyzer-checker=core,debug.ExprInspection \ // RUN: -analyzer-config experimental-enable-naive-ctu-analysis=true \ diff --git a/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp b/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp index 86ede5e..36a697d 100644 --- a/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp +++ b/clang/unittests/CrossTU/CrossTranslationUnitTest.cpp @@ -7,10 +7,11 @@ //===----------------------------------------------------------------------===// #include "clang/CrossTU/CrossTranslationUnit.h" -#include "clang/Frontend/CompilerInstance.h" #include "clang/AST/ASTConsumer.h" +#include "clang/Frontend/CompilerInstance.h" #include "clang/Frontend/FrontendAction.h" #include "clang/Tooling/Tooling.h" +#include "llvm/ADT/Optional.h" #include "llvm/Support/FileSystem.h" #include "llvm/Support/Path.h" #include "llvm/Support/ToolOutputFile.h" @@ -162,7 +163,7 @@ TEST(CrossTranslationUnit, IndexFormatCanBeParsed) { IndexFile.os().flush(); EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName)); llvm::Expected> IndexOrErr = - parseCrossTUIndex(IndexFileName, ""); + parseCrossTUIndex(IndexFileName); EXPECT_TRUE((bool)IndexOrErr); llvm::StringMap ParsedIndex = IndexOrErr.get(); for (const auto &E : Index) { @@ -173,24 +174,98 @@ TEST(CrossTranslationUnit, IndexFormatCanBeParsed) { EXPECT_TRUE(Index.count(E.getKey())); } -TEST(CrossTranslationUnit, CTUDirIsHandledCorrectly) { - llvm::StringMap Index; - Index["a"] = "/b/c/d"; - std::string IndexText = createCrossTUIndexString(Index); +TEST(CrossTranslationUnit, EmptyInvocationListIsNotValid) { + auto Input = ""; - int IndexFD; - llvm::SmallString<256> IndexFileName; - ASSERT_FALSE(llvm::sys::fs::createTemporaryFile("index", "txt", IndexFD, - IndexFileName)); - llvm::ToolOutputFile IndexFile(IndexFileName, IndexFD); - IndexFile.os() << IndexText; - IndexFile.os().flush(); - EXPECT_TRUE(llvm::sys::fs::exists(IndexFileName)); - llvm::Expected> IndexOrErr = - parseCrossTUIndex(IndexFileName, "/ctudir"); - EXPECT_TRUE((bool)IndexOrErr); - llvm::StringMap ParsedIndex = IndexOrErr.get(); - EXPECT_EQ(ParsedIndex["a"], "/ctudir/b/c/d"); + llvm::Expected Result = parseInvocationList(Input); + EXPECT_FALSE(static_cast(Result)); + bool IsWrongFromatError = false; + llvm::handleAllErrors(Result.takeError(), [&](IndexError &Err) { + IsWrongFromatError = + Err.getCode() == index_error_code::invocation_list_wrong_format; + }); + EXPECT_TRUE(IsWrongFromatError); +} + +TEST(CrossTranslationUnit, AmbiguousInvocationListIsDetected) { + // The same source file occurs twice (for two different architecture) in + // this test case. The disambiguation is the responsibility of the user. + auto Input = R"( + /tmp/main.cpp: + - clang++ + - -c + - -m32 + - -o + - main32.o + - /tmp/main.cpp + /tmp/main.cpp: + - clang++ + - -c + - -m64 + - -o + - main64.o + - /tmp/main.cpp + )"; + + llvm::Expected Result = parseInvocationList(Input); + EXPECT_FALSE(static_cast(Result)); + bool IsAmbiguousError = false; + llvm::handleAllErrors(Result.takeError(), [&](IndexError &Err) { + IsAmbiguousError = + Err.getCode() == index_error_code::invocation_list_ambiguous; + }); + EXPECT_TRUE(IsAmbiguousError); +} + +TEST(CrossTranslationUnit, SingleInvocationCanBeParsed) { + auto Input = R"( + /tmp/main.cpp: + - clang++ + - /tmp/main.cpp + )"; + llvm::Expected Result = parseInvocationList(Input); + EXPECT_TRUE(static_cast(Result)); + + EXPECT_EQ(Result->size(), 1u); + + auto It = Result->find("/tmp/main.cpp"); + EXPECT_TRUE(It != Result->end()); + EXPECT_EQ(It->getValue()[0], "clang++"); + EXPECT_EQ(It->getValue()[1], "/tmp/main.cpp"); +} + +TEST(CrossTranslationUnit, MultipleInvocationsCanBeParsed) { + auto Input = R"( + /tmp/main.cpp: + - clang++ + - /tmp/other.o + - /tmp/main.cpp + /tmp/other.cpp: + - g++ + - -c + - -o + - /tmp/other.o + - /tmp/other.cpp + )"; + llvm::Expected Result = parseInvocationList(Input); + EXPECT_TRUE(static_cast(Result)); + + EXPECT_EQ(Result->size(), 2u); + + auto It = Result->find("/tmp/main.cpp"); + EXPECT_TRUE(It != Result->end()); + EXPECT_EQ(It->getKey(), "/tmp/main.cpp"); + EXPECT_EQ(It->getValue()[0], "clang++"); + EXPECT_EQ(It->getValue()[1], "/tmp/other.o"); + EXPECT_EQ(It->getValue()[2], "/tmp/main.cpp"); + + It = Result->find("/tmp/other.cpp"); + EXPECT_TRUE(It != Result->end()); + EXPECT_EQ(It->getValue()[0], "g++"); + EXPECT_EQ(It->getValue()[1], "-c"); + EXPECT_EQ(It->getValue()[2], "-o"); + EXPECT_EQ(It->getValue()[3], "/tmp/other.o"); + EXPECT_EQ(It->getValue()[4], "/tmp/other.cpp"); } } // end namespace cross_tu