[clang][analyzer][ctu] Make CTU a two phase analysis
authorGabor Marton <gabor.marton@ericsson.com>
Thu, 14 Apr 2022 09:03:44 +0000 (11:03 +0200)
committerGabor Marton <gabor.marton@ericsson.com>
Wed, 18 May 2022 08:35:52 +0000 (10:35 +0200)
This new CTU implementation is the natural extension of the normal single TU
analysis. The approach consists of two analysis phases. During the first phase,
we do a normal single TU analysis. During this phase, if we find a foreign
function (that could be inlined from another TU) then we don’t inline that
immediately, we rather mark that to be analysed later.
When the first phase is finished then we start the second phase, the CTU phase.
In this phase, we continue the analysis from that point (exploded node)
which had been enqueued during the first phase. We gradually extend the
exploded graph of the single TU analysis with the new node that was
created by the inlining of the foreign function.

We count the number of analysis steps of the first phase and we limit the
second (ctu) phase with this number.

This new implementation makes it convenient for the users to run the
single-TU and the CTU analysis in one go, they don't need to run the two
analysis separately. Thus, we name this new implementation as "onego" CTU.

Discussion:
https://discourse.llvm.org/t/rfc-much-faster-cross-translation-unit-ctu-analysis-implementation/61728

Differential Revision: https://reviews.llvm.org/D123773

32 files changed:
clang/docs/ReleaseNotes.rst
clang/include/clang/CrossTU/CrossTranslationUnit.h
clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.def
clang/include/clang/StaticAnalyzer/Core/AnalyzerOptions.h
clang/include/clang/StaticAnalyzer/Core/PathSensitive/CallEvent.h
clang/include/clang/StaticAnalyzer/Core/PathSensitive/CoreEngine.h
clang/include/clang/StaticAnalyzer/Core/PathSensitive/ExprEngine.h
clang/lib/CrossTU/CrossTranslationUnit.cpp
clang/lib/StaticAnalyzer/Core/AnalyzerOptions.cpp
clang/lib/StaticAnalyzer/Core/CallEvent.cpp
clang/lib/StaticAnalyzer/Core/CoreEngine.cpp
clang/lib/StaticAnalyzer/Core/ExprEngine.cpp
clang/lib/StaticAnalyzer/Core/ExprEngineCallAndReturn.cpp
clang/lib/StaticAnalyzer/Frontend/AnalysisConsumer.cpp
clang/test/Analysis/Inputs/ctu-onego-existingdef-other.cpp [new file with mode: 0644]
clang/test/Analysis/Inputs/ctu-onego-existingdef-other.cpp.externalDefMap.ast-dump.txt [new file with mode: 0644]
clang/test/Analysis/Inputs/ctu-onego-indirect-other.cpp [new file with mode: 0644]
clang/test/Analysis/Inputs/ctu-onego-indirect-other.cpp.externalDefMap.ast-dump.txt [new file with mode: 0644]
clang/test/Analysis/Inputs/ctu-onego-small-other.cpp [new file with mode: 0644]
clang/test/Analysis/Inputs/ctu-onego-small-other.cpp.externalDefMap.ast-dump.txt [new file with mode: 0644]
clang/test/Analysis/Inputs/ctu-onego-toplevel-other.cpp [new file with mode: 0644]
clang/test/Analysis/Inputs/ctu-onego-toplevel-other.cpp.externalDefMap.ast-dump.txt [new file with mode: 0644]
clang/test/Analysis/analyzer-config.c
clang/test/Analysis/ctu-implicit.c
clang/test/Analysis/ctu-main.c
clang/test/Analysis/ctu-main.cpp
clang/test/Analysis/ctu-on-demand-parsing.c
clang/test/Analysis/ctu-on-demand-parsing.cpp
clang/test/Analysis/ctu-onego-existingdef.cpp [new file with mode: 0644]
clang/test/Analysis/ctu-onego-indirect.cpp [new file with mode: 0644]
clang/test/Analysis/ctu-onego-small.cpp [new file with mode: 0644]
clang/test/Analysis/ctu-onego-toplevel.cpp [new file with mode: 0644]

index 2477a80..2092094 100644 (file)
@@ -479,6 +479,13 @@ libclang
 
 Static Analyzer
 ---------------
+- `New CTU implementation
+  <https://discourse.llvm.org/t/rfc-much-faster-cross-translation-unit-ctu-analysis-implementation/61728>`_
+  that keeps the slow-down around 2x compared to the single-TU analysis, even
+  in case of complex C++ projects. Still, it finds the majority of the "old"
+  CTU findings. Besides, not more than ~3% of the bug reports are lost compared
+  to single-TU analysis, the lost reports are highly likely to be false
+  positives.
 
 - Added a new checker ``alpha.unix.cstring.UninitializedRead`` this will check for uninitialized reads
   from common memory copy/manipulation functions such as ``memcpy``, ``mempcpy``, ``memmove``, ``memcmp``, `
index f94f246..3a0178a 100644 (file)
@@ -197,6 +197,14 @@ public:
   getMacroExpansionContextForSourceLocation(
       const clang::SourceLocation &ToLoc) const;
 
+  /// Returns true if the given Decl is newly created during the import.
+  bool isImportedAsNew(const Decl *ToDecl) const;
+
+  /// Returns true if the given Decl is mapped (or created) during an import
+  /// but there was an unrecoverable error (the AST node cannot be erased, it
+  /// is marked with an Error object in this case).
+  bool hasError(const Decl *ToDecl) const;
+
 private:
   void lazyInitImporterSharedSt(TranslationUnitDecl *ToTU);
   ASTImporter &getOrCreateASTImporter(ASTUnit *Unit);
index be356b5..d7075fb 100644 (file)
@@ -410,6 +410,34 @@ ANALYZER_OPTION_DEPENDS_ON_USER_MODE(
     /* SHALLOW_VAL */ 75000, /* DEEP_VAL */ 225000)
 
 ANALYZER_OPTION(
+    unsigned, CTUMaxNodesPercentage, "ctu-max-nodes-pct",
+    "The percentage of single-TU analysed nodes that the CTU analysis is "
+    "allowed to visit.", 50)
+
+ANALYZER_OPTION(
+    unsigned, CTUMaxNodesMin, "ctu-max-nodes-min",
+    "The maximum number of nodes in CTU mode is determinded by "
+    "'ctu-max-nodes-pct'. However, if the number of nodes in single-TU "
+    "analysis is too low, it is meaningful to provide a minimum value that "
+    "serves as an upper bound instead.", 10000)
+
+ANALYZER_OPTION(
+    StringRef, CTUPhase1InliningMode, "ctu-phase1-inlining",
+    "Controls which functions will be inlined during the first phase of the ctu "
+    "analysis. "
+    "If the value is set to 'all' then all foreign functions are inlinied "
+    "immediately during the first phase, thus rendering the second phase a noop. "
+    "The 'ctu-max-nodes-*' budge has no effect in this case. "
+    "If the value is 'small' then only functions with a linear CFG and with a "
+    "limited number of statements would be inlined during the first phase. The "
+    "long and/or nontrivial functions are handled in the second phase and are "
+    "controlled by the 'ctu-max-nodes-*' budge. "
+    "The value 'none' means that all foreign functions are inlined only in the "
+    "second phase, 'ctu-max-nodes-*' budge limits the second phase. "
+    "Value: \"none\", \"small\", \"all\".",
+    "small")
+
+ANALYZER_OPTION(
     unsigned, RegionStoreSmallStructLimit, "region-store-small-struct-limit",
     "The largest number of fields a struct can have and still be considered "
     "small This is currently used to decide whether or not it is worth forcing "
index 7514eee..0c4f9e6 100644 (file)
@@ -138,6 +138,8 @@ enum UserModeKind {
   UMK_Deep = 2
 };
 
+enum class CTUPhase1InliningKind { None, Small, All };
+
 /// Stores options for the analyzer from the command line.
 ///
 /// Some options are frontend flags (e.g.: -analyzer-output), but some are
@@ -379,6 +381,7 @@ public:
   UserModeKind getUserMode() const;
 
   ExplorationStrategyKind getExplorationStrategy() const;
+  CTUPhase1InliningKind getCTUPhase1Inlining() const;
 
   /// Returns the inter-procedural analysis mode.
   IPAKind getIPAMode() const;
index fba9aa7..00f66d8 100644 (file)
@@ -113,12 +113,18 @@ class RuntimeDefinition {
   /// precise.
   const MemRegion *R = nullptr;
 
+  /// A definition is foreign if it has been imported and newly created by the
+  /// ASTImporter. This can be true only if CTU is enabled.
+  const bool Foreign = false;
+
 public:
   RuntimeDefinition() = default;
   RuntimeDefinition(const Decl *InD): D(InD) {}
+  RuntimeDefinition(const Decl *InD, bool Foreign) : D(InD), Foreign(Foreign) {}
   RuntimeDefinition(const Decl *InD, const MemRegion *InR): D(InD), R(InR) {}
 
   const Decl *getDecl() { return D; }
+  bool isForeign() const { return Foreign; }
 
   /// Check if the definition we have is precise.
   /// If not, it is possible that the call dispatches to another definition at
@@ -147,6 +153,7 @@ private:
   ProgramStateRef State;
   const LocationContext *LCtx;
   llvm::PointerUnion<const Expr *, const Decl *> Origin;
+  mutable Optional<bool> Foreign; // Set by CTU analysis.
 
 protected:
   // This is user data for subclasses.
@@ -208,6 +215,12 @@ public:
     return Origin.dyn_cast<const Decl *>();
   }
 
+  bool isForeign() const {
+    assert(Foreign.hasValue() && "Foreign must be set before querying");
+    return *Foreign;
+  }
+  void setForeign(bool B) const { Foreign = B; }
+
   /// The state in which the call is being evaluated.
   const ProgramStateRef &getState() const {
     return State;
index 760dc1d..220aee7 100644 (file)
@@ -78,6 +78,7 @@ private:
   ///  worklist algorithm.  It is up to the implementation of WList to decide
   ///  the order that nodes are processed.
   std::unique_ptr<WorkList> WList;
+  std::unique_ptr<WorkList> CTUWList;
 
   /// BCounterFactory - A factory object for created BlockCounter objects.
   ///   These are used to record for key nodes in the ExplodedGraph the
@@ -101,6 +102,8 @@ private:
   /// tags.
   DataTag::Factory DataTags;
 
+  void setBlockCounter(BlockCounter C);
+
   void generateNode(const ProgramPoint &Loc,
                     ProgramStateRef State,
                     ExplodedNode *Pred);
@@ -170,6 +173,7 @@ public:
   }
 
   WorkList *getWorkList() const { return WList.get(); }
+  WorkList *getCTUWorkList() const { return CTUWList.get(); }
 
   BlocksExhausted::const_iterator blocks_exhausted_begin() const {
     return blocksExhausted.begin();
index feb4a72..fe558fc 100644 (file)
@@ -135,6 +135,7 @@ public:
 
 private:
   cross_tu::CrossTranslationUnitContext &CTU;
+  bool IsCTUEnabled;
 
   AnalysisManager &AMgr;
 
@@ -805,8 +806,14 @@ private:
                         const ExplodedNode *Pred,
                         const EvalCallOptions &CallOpts = {});
 
-  bool inlineCall(const CallEvent &Call, const Decl *D, NodeBuilder &Bldr,
-                  ExplodedNode *Pred, ProgramStateRef State);
+  void inlineCall(WorkList *WList, const CallEvent &Call, const Decl *D,
+                  NodeBuilder &Bldr, ExplodedNode *Pred, ProgramStateRef State);
+
+  void ctuBifurcate(const CallEvent &Call, const Decl *D, NodeBuilder &Bldr,
+                    ExplodedNode *Pred, ProgramStateRef State);
+
+  /// Returns true if the CTU analysis is running its second phase.
+  bool isSecondPhaseCTU() { return IsCTUEnabled && !Engine.getCTUWorkList(); }
 
   /// Conservatively evaluate call by invalidating regions and binding
   /// a conjured return value.
index ee6cc60..8bc478b 100644 (file)
@@ -801,5 +801,18 @@ CrossTranslationUnitContext::getMacroExpansionContextForSourceLocation(
   return llvm::None;
 }
 
+bool CrossTranslationUnitContext::isImportedAsNew(const Decl *ToDecl) const {
+  if (!ImporterSharedSt)
+    return false;
+  return ImporterSharedSt->isNewDecl(const_cast<Decl *>(ToDecl));
+}
+
+bool CrossTranslationUnitContext::hasError(const Decl *ToDecl) const {
+  if (!ImporterSharedSt)
+    return false;
+  return static_cast<bool>(
+      ImporterSharedSt->getImportDeclErrorIfAny(const_cast<Decl *>(ToDecl)));
+}
+
 } // namespace cross_tu
 } // namespace clang
index 8cd7f75..7b9b27e 100644 (file)
@@ -81,6 +81,17 @@ AnalyzerOptions::getExplorationStrategy() const {
   return K.getValue();
 }
 
+CTUPhase1InliningKind AnalyzerOptions::getCTUPhase1Inlining() const {
+  auto K = llvm::StringSwitch<llvm::Optional<CTUPhase1InliningKind>>(
+               CTUPhase1InliningMode)
+               .Case("none", CTUPhase1InliningKind::None)
+               .Case("small", CTUPhase1InliningKind::Small)
+               .Case("all", CTUPhase1InliningKind::All)
+               .Default(None);
+  assert(K.hasValue() && "CTU inlining mode is invalid.");
+  return K.getValue();
+}
+
 IPAKind AnalyzerOptions::getIPAMode() const {
   auto K = llvm::StringSwitch<llvm::Optional<IPAKind>>(IPAMode)
           .Case("none", IPAK_None)
index 1e61c54..135e541 100644 (file)
@@ -515,20 +515,28 @@ RuntimeDefinition AnyFunctionCall::getRuntimeDefinition() const {
       llvm::dbgs() << "Using autosynthesized body for " << FD->getName()
                    << "\n";
   });
-  if (Body) {
-    const Decl* Decl = AD->getDecl();
-    return RuntimeDefinition(Decl);
-  }
 
   ExprEngine &Engine = getState()->getStateManager().getOwningEngine();
+  cross_tu::CrossTranslationUnitContext &CTUCtx =
+      *Engine.getCrossTranslationUnitContext();
+
   AnalyzerOptions &Opts = Engine.getAnalysisManager().options;
 
+  if (Body) {
+    const Decl* Decl = AD->getDecl();
+    if (Opts.IsNaiveCTUEnabled && CTUCtx.isImportedAsNew(Decl)) {
+      // A newly created definition, but we had error(s) during the import.
+      if (CTUCtx.hasError(Decl))
+        return {};
+      return RuntimeDefinition(Decl, /*Foreign=*/true);
+    }
+    return RuntimeDefinition(Decl, /*Foreign=*/false);
+  }
+
   // Try to get CTU definition only if CTUDir is provided.
   if (!Opts.IsNaiveCTUEnabled)
     return {};
 
-  cross_tu::CrossTranslationUnitContext &CTUCtx =
-      *Engine.getCrossTranslationUnitContext();
   llvm::Expected<const FunctionDecl *> CTUDeclOrError =
       CTUCtx.getCrossTUDefinition(FD, Opts.CTUDir, Opts.CTUIndexName,
                                   Opts.DisplayCTUProgress);
@@ -541,7 +549,7 @@ RuntimeDefinition AnyFunctionCall::getRuntimeDefinition() const {
     return {};
   }
 
-  return RuntimeDefinition(*CTUDeclOrError);
+  return RuntimeDefinition(*CTUDeclOrError, /*Foreign=*/true);
 }
 
 void AnyFunctionCall::getInitialStackFrameContents(
index d57bab1..de90f4a 100644 (file)
@@ -43,6 +43,8 @@ using namespace ento;
 
 STATISTIC(NumSteps,
             "The # of steps executed.");
+STATISTIC(NumSTUSteps, "The # of STU steps executed.");
+STATISTIC(NumCTUSteps, "The # of CTU steps executed.");
 STATISTIC(NumReachedMaxSteps,
             "The # of times we reached the max number of steps.");
 STATISTIC(NumPathsExplored,
@@ -73,11 +75,18 @@ static std::unique_ptr<WorkList> generateWorkList(AnalyzerOptions &Opts) {
 CoreEngine::CoreEngine(ExprEngine &exprengine, FunctionSummariesTy *FS,
                        AnalyzerOptions &Opts)
     : ExprEng(exprengine), WList(generateWorkList(Opts)),
+      CTUWList(Opts.IsNaiveCTUEnabled ? generateWorkList(Opts) : nullptr),
       BCounterFactory(G.getAllocator()), FunctionSummaries(FS) {}
 
+void CoreEngine::setBlockCounter(BlockCounter C) {
+  WList->setBlockCounter(C);
+  if (CTUWList)
+    CTUWList->setBlockCounter(C);
+}
+
 /// ExecuteWorkList - Run the worklist algorithm for a maximum number of steps.
-bool CoreEngine::ExecuteWorkList(const LocationContext *L, unsigned Steps,
-                                   ProgramStateRef InitState) {
+bool CoreEngine::ExecuteWorkList(const LocationContext *L, unsigned MaxSteps,
+                                 ProgramStateRef InitState) {
   if (G.num_roots() == 0) { // Initialize the analysis by constructing
     // the root if none exists.
 
@@ -100,7 +109,7 @@ bool CoreEngine::ExecuteWorkList(const LocationContext *L, unsigned Steps,
     BlockEdge StartLoc(Entry, Succ, L);
 
     // Set the current block counter to being empty.
-    WList->setBlockCounter(BCounterFactory.GetEmptyCounter());
+    setBlockCounter(BCounterFactory.GetEmptyCounter());
 
     if (!InitState)
       InitState = ExprEng.getInitialState(L);
@@ -118,34 +127,54 @@ bool CoreEngine::ExecuteWorkList(const LocationContext *L, unsigned Steps,
   }
 
   // Check if we have a steps limit
-  bool UnlimitedSteps = Steps == 0;
+  bool UnlimitedSteps = MaxSteps == 0;
+
   // Cap our pre-reservation in the event that the user specifies
   // a very large number of maximum steps.
   const unsigned PreReservationCap = 4000000;
   if(!UnlimitedSteps)
-    G.reserve(std::min(Steps,PreReservationCap));
-
-  while (WList->hasWork()) {
-    if (!UnlimitedSteps) {
-      if (Steps == 0) {
-        NumReachedMaxSteps++;
-        break;
+    G.reserve(std::min(MaxSteps, PreReservationCap));
+
+  auto ProcessWList = [this, UnlimitedSteps](unsigned MaxSteps) {
+    unsigned Steps = MaxSteps;
+    while (WList->hasWork()) {
+      if (!UnlimitedSteps) {
+        if (Steps == 0) {
+          NumReachedMaxSteps++;
+          break;
+        }
+        --Steps;
       }
-      --Steps;
-    }
 
-    NumSteps++;
+      NumSteps++;
 
-    const WorkListUnit& WU = WList->dequeue();
+      const WorkListUnit &WU = WList->dequeue();
 
-    // Set the current block counter.
-    WList->setBlockCounter(WU.getBlockCounter());
+      // Set the current block counter.
+      setBlockCounter(WU.getBlockCounter());
 
-    // Retrieve the node.
-    ExplodedNode *Node = WU.getNode();
+      // Retrieve the node.
+      ExplodedNode *Node = WU.getNode();
 
-    dispatchWorkItem(Node, Node->getLocation(), WU);
+      dispatchWorkItem(Node, Node->getLocation(), WU);
+    }
+    return MaxSteps - Steps;
+  };
+  const unsigned STUSteps = ProcessWList(MaxSteps);
+
+  if (CTUWList) {
+    NumSTUSteps += STUSteps;
+    const unsigned MinCTUSteps =
+        this->ExprEng.getAnalysisManager().options.CTUMaxNodesMin;
+    const unsigned Pct =
+        this->ExprEng.getAnalysisManager().options.CTUMaxNodesPercentage;
+    unsigned MaxCTUSteps = std::max(STUSteps * Pct / 100, MinCTUSteps);
+
+    WList = std::move(CTUWList);
+    const unsigned CTUSteps = ProcessWList(MaxCTUSteps);
+    NumCTUSteps += CTUSteps;
   }
+
   ExprEng.processEndWorklist();
   return WList->hasWork();
 }
@@ -282,7 +311,7 @@ void CoreEngine::HandleBlockEntrance(const BlockEntrance &L,
   BlockCounter Counter = WList->getBlockCounter();
   Counter = BCounterFactory.IncrementCount(Counter, LC->getStackFrame(),
                                            BlockId);
-  WList->setBlockCounter(Counter);
+  setBlockCounter(Counter);
 
   // Process the entrance of the block.
   if (Optional<CFGElement> E = L.getFirstElement()) {
index e748a35..a8bffd3 100644 (file)
@@ -200,24 +200,17 @@ REGISTER_TRAIT_WITH_PROGRAMSTATE(ObjectsUnderConstruction,
 static const char* TagProviderName = "ExprEngine";
 
 ExprEngine::ExprEngine(cross_tu::CrossTranslationUnitContext &CTU,
-                       AnalysisManager &mgr,
-                       SetOfConstDecls *VisitedCalleesIn,
-                       FunctionSummariesTy *FS,
-                       InliningModes HowToInlineIn)
-    : CTU(CTU), AMgr(mgr),
-      AnalysisDeclContexts(mgr.getAnalysisDeclContextManager()),
+                       AnalysisManager &mgr, SetOfConstDecls *VisitedCalleesIn,
+                       FunctionSummariesTy *FS, InliningModes HowToInlineIn)
+    : CTU(CTU), IsCTUEnabled(mgr.getAnalyzerOptions().IsNaiveCTUEnabled),
+      AMgr(mgr), AnalysisDeclContexts(mgr.getAnalysisDeclContextManager()),
       Engine(*this, FS, mgr.getAnalyzerOptions()), G(Engine.getGraph()),
       StateMgr(getContext(), mgr.getStoreManagerCreator(),
-               mgr.getConstraintManagerCreator(), G.getAllocator(),
-               this),
-      SymMgr(StateMgr.getSymbolManager()),
-      MRMgr(StateMgr.getRegionManager()),
-      svalBuilder(StateMgr.getSValBuilder()),
-      ObjCNoRet(mgr.getASTContext()),
-      BR(mgr, *this),
-      VisitedCallees(VisitedCalleesIn),
-      HowToInline(HowToInlineIn)
-  {
+               mgr.getConstraintManagerCreator(), G.getAllocator(), this),
+      SymMgr(StateMgr.getSymbolManager()), MRMgr(StateMgr.getRegionManager()),
+      svalBuilder(StateMgr.getSValBuilder()), ObjCNoRet(mgr.getASTContext()),
+      BR(mgr, *this), VisitedCallees(VisitedCalleesIn),
+      HowToInline(HowToInlineIn) {
   unsigned TrimInterval = mgr.options.GraphTrimInterval;
   if (TrimInterval != 0) {
     // Enable eager node reclamation when constructing the ExplodedGraph.
index e6918e0..04de505 100644 (file)
@@ -427,10 +427,39 @@ namespace {
 
 REGISTER_MAP_WITH_PROGRAMSTATE(DynamicDispatchBifurcationMap,
                                const MemRegion *, unsigned)
+REGISTER_TRAIT_WITH_PROGRAMSTATE(CTUDispatchBifurcation, bool)
+
+void ExprEngine::ctuBifurcate(const CallEvent &Call, const Decl *D,
+                              NodeBuilder &Bldr, ExplodedNode *Pred,
+                              ProgramStateRef State) {
+  ProgramStateRef ConservativeEvalState = nullptr;
+  if (Call.isForeign() && !isSecondPhaseCTU()) {
+    const auto IK = AMgr.options.getCTUPhase1Inlining();
+    const bool DoInline = IK == CTUPhase1InliningKind::All ||
+                          (IK == CTUPhase1InliningKind::Small &&
+                           isSmall(AMgr.getAnalysisDeclContext(D)));
+    if (DoInline) {
+      inlineCall(Engine.getWorkList(), Call, D, Bldr, Pred, State);
+      return;
+    }
+    const bool BState = State->get<CTUDispatchBifurcation>();
+    if (!BState) { // This is the first time we see this foreign function.
+      // Enqueue it to be analyzed in the second (ctu) phase.
+      inlineCall(Engine.getCTUWorkList(), Call, D, Bldr, Pred, State);
+      // Conservatively evaluate in the first phase.
+      ConservativeEvalState = State->set<CTUDispatchBifurcation>(true);
+      conservativeEvalCall(Call, Bldr, Pred, ConservativeEvalState);
+    } else {
+      conservativeEvalCall(Call, Bldr, Pred, State);
+    }
+    return;
+  }
+  inlineCall(Engine.getWorkList(), Call, D, Bldr, Pred, State);
+}
 
-bool ExprEngine::inlineCall(const CallEvent &Call, const Decl *D,
-                            NodeBuilder &Bldr, ExplodedNode *Pred,
-                            ProgramStateRef State) {
+void ExprEngine::inlineCall(WorkList *WList, const CallEvent &Call,
+                            const Decl *D, NodeBuilder &Bldr,
+                            ExplodedNode *Pred, ProgramStateRef State) {
   assert(D);
 
   const LocationContext *CurLC = Pred->getLocationContext();
@@ -465,7 +494,7 @@ bool ExprEngine::inlineCall(const CallEvent &Call, const Decl *D,
   if (ExplodedNode *N = G.getNode(Loc, State, false, &isNew)) {
     N->addPredecessor(Pred, G);
     if (isNew)
-      Engine.getWorkList()->enqueue(N);
+      WList->enqueue(N);
   }
 
   // If we decided to inline the call, the successor has been manually
@@ -475,11 +504,17 @@ bool ExprEngine::inlineCall(const CallEvent &Call, const Decl *D,
   NumInlinedCalls++;
   Engine.FunctionSummaries->bumpNumTimesInlined(D);
 
-  // Mark the decl as visited.
-  if (VisitedCallees)
-    VisitedCallees->insert(D);
-
-  return true;
+  // Do not mark as visited in the 2nd run (CTUWList), so the function will
+  // be visited as top-level, this way we won't loose reports in non-ctu
+  // mode. Considering the case when a function in a foreign TU calls back
+  // into the main TU.
+  // Note, during the 1st run, it doesn't matter if we mark the foreign
+  // functions as visited (or not) because they can never appear as a top level
+  // function in the main TU.
+  if (!isSecondPhaseCTU())
+    // Mark the decl as visited.
+    if (VisitedCallees)
+      VisitedCallees->insert(D);
 }
 
 static ProgramStateRef getInlineFailedState(ProgramStateRef State,
@@ -1068,6 +1103,7 @@ void ExprEngine::defaultEvalCall(NodeBuilder &Bldr, ExplodedNode *Pred,
     State = InlinedFailedState;
   } else {
     RuntimeDefinition RD = Call->getRuntimeDefinition();
+    Call->setForeign(RD.isForeign());
     const Decl *D = RD.getDecl();
     if (shouldInlineCall(*Call, D, Pred, CallOpts)) {
       if (RD.mayHaveOtherDefinitions()) {
@@ -1085,10 +1121,8 @@ void ExprEngine::defaultEvalCall(NodeBuilder &Bldr, ExplodedNode *Pred,
           return;
         }
       }
-
-      // We are not bifurcating and we do have a Decl, so just inline.
-      if (inlineCall(*Call, D, Bldr, Pred, State))
-        return;
+      ctuBifurcate(*Call, D, Bldr, Pred, State);
+      return;
     }
   }
 
@@ -1110,8 +1144,7 @@ void ExprEngine::BifurcateCall(const MemRegion *BifurReg,
   if (BState) {
     // If we are on "inline path", keep inlining if possible.
     if (*BState == DynamicDispatchModeInlined)
-      if (inlineCall(Call, D, Bldr, Pred, State))
-        return;
+      ctuBifurcate(Call, D, Bldr, Pred, State);
     // If inline failed, or we are on the path where we assume we
     // don't have enough info about the receiver to inline, conjure the
     // return value and invalidate the regions.
@@ -1124,7 +1157,7 @@ void ExprEngine::BifurcateCall(const MemRegion *BifurReg,
   ProgramStateRef IState =
       State->set<DynamicDispatchBifurcationMap>(BifurReg,
                                                DynamicDispatchModeInlined);
-  inlineCall(Call, D, Bldr, Pred, IState);
+  ctuBifurcate(Call, D, Bldr, Pred, IState);
 
   ProgramStateRef NoIState =
       State->set<DynamicDispatchBifurcationMap>(BifurReg,
index fcc73b3..ad438f6 100644 (file)
@@ -476,6 +476,18 @@ void AnalysisConsumer::HandleDeclsCallGraph(const unsigned LocalTUDeclsSize) {
     if (shouldSkipFunction(D, Visited, VisitedAsTopLevel))
       continue;
 
+    // The CallGraph might have declarations as callees. However, during CTU
+    // the declaration might form a declaration chain with the newly imported
+    // definition from another TU. In this case we don't want to analyze the
+    // function definition as toplevel.
+    if (const auto *FD = dyn_cast<FunctionDecl>(D)) {
+      // Calling 'hasBody' replaces 'FD' in place with the FunctionDecl
+      // that has the body.
+      FD->hasBody(FD);
+      if (CTU.isImportedAsNew(FD))
+        continue;
+    }
+
     // Analyze the function.
     SetOfConstDecls VisitedCallees;
 
diff --git a/clang/test/Analysis/Inputs/ctu-onego-existingdef-other.cpp b/clang/test/Analysis/Inputs/ctu-onego-existingdef-other.cpp
new file mode 100644 (file)
index 0000000..b9e1714
--- /dev/null
@@ -0,0 +1,7 @@
+int bar() {
+  return 0;
+}
+
+void other() {
+  bar();
+}
diff --git a/clang/test/Analysis/Inputs/ctu-onego-existingdef-other.cpp.externalDefMap.ast-dump.txt b/clang/test/Analysis/Inputs/ctu-onego-existingdef-other.cpp.externalDefMap.ast-dump.txt
new file mode 100644 (file)
index 0000000..e785e61
--- /dev/null
@@ -0,0 +1,2 @@
+9:c:@F@bar# ctu-onego-existingdef-other.cpp.ast
+11:c:@F@other# ctu-onego-existingdef-other.cpp.ast
diff --git a/clang/test/Analysis/Inputs/ctu-onego-indirect-other.cpp b/clang/test/Analysis/Inputs/ctu-onego-indirect-other.cpp
new file mode 100644 (file)
index 0000000..b9e1714
--- /dev/null
@@ -0,0 +1,7 @@
+int bar() {
+  return 0;
+}
+
+void other() {
+  bar();
+}
diff --git a/clang/test/Analysis/Inputs/ctu-onego-indirect-other.cpp.externalDefMap.ast-dump.txt b/clang/test/Analysis/Inputs/ctu-onego-indirect-other.cpp.externalDefMap.ast-dump.txt
new file mode 100644 (file)
index 0000000..4d5fbd2
--- /dev/null
@@ -0,0 +1,2 @@
+11:c:@F@other# ctu-onego-indirect-other.cpp.ast
+9:c:@F@bar# ctu-onego-indirect-other.cpp.ast
diff --git a/clang/test/Analysis/Inputs/ctu-onego-small-other.cpp b/clang/test/Analysis/Inputs/ctu-onego-small-other.cpp
new file mode 100644 (file)
index 0000000..a4957da
--- /dev/null
@@ -0,0 +1,3 @@
+int bar() {
+  return 0;
+}
diff --git a/clang/test/Analysis/Inputs/ctu-onego-small-other.cpp.externalDefMap.ast-dump.txt b/clang/test/Analysis/Inputs/ctu-onego-small-other.cpp.externalDefMap.ast-dump.txt
new file mode 100644 (file)
index 0000000..fa84974
--- /dev/null
@@ -0,0 +1 @@
+9:c:@F@bar# ctu-onego-small-other.cpp.ast
diff --git a/clang/test/Analysis/Inputs/ctu-onego-toplevel-other.cpp b/clang/test/Analysis/Inputs/ctu-onego-toplevel-other.cpp
new file mode 100644 (file)
index 0000000..6682938
--- /dev/null
@@ -0,0 +1,4 @@
+void b(int x);
+void other(int y) {
+  b(1);
+}
diff --git a/clang/test/Analysis/Inputs/ctu-onego-toplevel-other.cpp.externalDefMap.ast-dump.txt b/clang/test/Analysis/Inputs/ctu-onego-toplevel-other.cpp.externalDefMap.ast-dump.txt
new file mode 100644 (file)
index 0000000..2db5f54
--- /dev/null
@@ -0,0 +1 @@
+13:c:@F@other#I# ctu-onego-toplevel-other.cpp.ast
index 9fed04e..440547b 100644 (file)
@@ -50,6 +50,9 @@
 // CHECK-NEXT: ctu-import-threshold = 24
 // CHECK-NEXT: ctu-index-name = externalDefMap.txt
 // CHECK-NEXT: ctu-invocation-list = invocations.yaml
+// CHECK-NEXT: ctu-max-nodes-min = 10000
+// CHECK-NEXT: ctu-max-nodes-pct = 50
+// CHECK-NEXT: ctu-phase1-inlining = small
 // CHECK-NEXT: deadcode.DeadStores:ShowFixIts = false
 // CHECK-NEXT: deadcode.DeadStores:WarnForDeadNestedAssignments = true
 // CHECK-NEXT: debug.AnalysisOrder:* = false
index 9250448..c0b28fc 100644 (file)
@@ -5,6 +5,7 @@
 // RUN: cp %S/Inputs/ctu-import.c.externalDefMap.ast-dump.txt %t/ctudir2/externalDefMap.txt
 // RUN: %clang_cc1 -analyze \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config  display-ctu-progress=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir2 \
@@ -15,6 +16,7 @@ void clang_analyzer_eval(int);
 int testStaticImplicit(void);
 int func(void) {
   int ret = testStaticImplicit();
-  clang_analyzer_eval(ret == 4); // expected-warning{{TRUE}}
+  clang_analyzer_eval(ret == 4); // expected-warning{{TRUE}} ctu
+                                 // expected-warning@-1{{UNKNOWN}} stu
   return testStaticImplicit();
 }
index 00ee7e4..46ae5e1 100644 (file)
@@ -3,14 +3,36 @@
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir2/ctu-other.c.ast %S/Inputs/ctu-other.c
 // RUN: cp %S/Inputs/ctu-other.c.externalDefMap.ast-dump.txt %t/ctudir2/externalDefMap.txt
+
+// RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir2 \
+// RUN:   -analyzer-config ctu-phase1-inlining=none \
+// RUN:   -verify=newctu %s
+
+// Simulate the behavior of the previous CTU implementation by inlining all
+// functions during the first phase. This way, the second phase is a noop.
 // RUN: %clang_cc1 -triple x86_64-pc-linux-gnu -fsyntax-only -std=c89 -analyze \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir2 \
-// RUN:   -verify %s
+// RUN:   -analyzer-config ctu-phase1-inlining=all \
+// RUN:   -verify=oldctu %s
 
 void clang_analyzer_eval(int);
 
+// A function that's definition is unknown both for single-tu (stu) and ctu
+// mode.
+int unknown(int);
+void test_unknown() {
+  int res = unknown(6);
+  clang_analyzer_eval(res == 6); // newctu-warning{{UNKNOWN}}
+                                 // oldctu-warning@-1{{UNKNOWN}}
+}
+
 // Test typedef and global variable in function.
 typedef struct {
   int a;
@@ -18,8 +40,10 @@ typedef struct {
 } FooBar;
 extern FooBar fb;
 int f(int);
-void testGlobalVariable(void) {
-  clang_analyzer_eval(f(5) == 1);         // expected-warning{{TRUE}}
+void testGlobalVariable() {
+  clang_analyzer_eval(f(5) == 1);         // newctu-warning{{TRUE}} ctu
+                                          // newctu-warning@-1{{UNKNOWN}} stu
+                                          // oldctu-warning@-2{{TRUE}}
 }
 
 // Test enums.
@@ -28,8 +52,11 @@ enum A { x,
          y,
          z };
 void testEnum(void) {
-  clang_analyzer_eval(x == 0);            // expected-warning{{TRUE}}
-  clang_analyzer_eval(enumCheck() == 42); // expected-warning{{TRUE}}
+  clang_analyzer_eval(x == 0);            // newctu-warning{{TRUE}}
+                                          // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(enumCheck() == 42); // newctu-warning{{TRUE}} ctu
+                                          // newctu-warning@-1{{UNKNOWN}} stu
+                                          // oldctu-warning@-2{{TRUE}}
 }
 
 // Test that asm import does not fail.
@@ -42,18 +69,22 @@ int testInlineAsm(void) {
 struct S;
 int g(struct S *);
 void testMacro(void) {
-  g(0); // expected-warning@Inputs/ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
+  g(0); // newctu-warning@Inputs/ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
+        // oldctu-warning@Inputs/ctu-other.c:29 {{Access to field 'a' results in a dereference of a null pointer (loaded from variable 'ctx')}}
 }
 
 // The external function prototype is incomplete.
 // warning:implicit functions are prohibited by c99
 void testImplicit(void) {
   int res = identImplicit(6);   // external implicit functions are not inlined
-  clang_analyzer_eval(res == 6); // expected-warning{{TRUE}}
+  clang_analyzer_eval(res == 6); // newctu-warning{{TRUE}} ctu
+                                 // newctu-warning@-1{{UNKNOWN}} stu
+                                 // oldctu-warning@-2{{TRUE}}
   // Call something with uninitialized from the same function in which the implicit was called.
   // This is necessary to reproduce a special bug in NoStoreFuncVisitor.
   int uninitialized;
-  h(uninitialized); // expected-warning{{1st function call argument is an uninitialized value}}
+  h(uninitialized); // newctu-warning{{1st function call argument is an uninitialized value}}
+                    // oldctu-warning@-1{{1st function call argument is an uninitialized value}}
 }
 
 // Tests the import of functions that have a struct parameter
@@ -67,7 +98,9 @@ void testStructDefInArgument(void) {
   struct DataType d;
   d.a = 1;
   d.b = 0;
-  clang_analyzer_eval(structInProto(&d) == 0); // expected-warning{{TRUE}} expected-warning{{FALSE}}
+  // Not imported, thus remains unknown both in stu and ctu.
+  clang_analyzer_eval(structInProto(&d) == 0); // newctu-warning{{UNKNOWN}}
+                                               // oldctu-warning@-1{{UNKNOWN}}
 }
 
 int switchWithoutCases(int);
index 5bf212a..27369b8 100644 (file)
@@ -5,11 +5,25 @@
 // RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -emit-pch -o %t/ctudir/ctu-chain.cpp.ast %S/Inputs/ctu-chain.cpp
 // RUN: cp %S/Inputs/ctu-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
+
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-config ctu-phase1-inlining=none \
+// RUN:   -verify=newctu %s
+
+// Simulate the behavior of the previous CTU implementation by inlining all
+// functions during the first phase. This way, the second phase is a noop.
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=%t/ctudir \
-// RUN:   -verify %s
+// RUN:   -analyzer-config ctu-phase1-inlining=all \
+// RUN:   -verify=oldctu %s
+
 // RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
@@ -113,10 +127,17 @@ extern const U extU;
 
 void test_virtual_functions(mycls* obj) {
   // The dynamic type is known.
-  clang_analyzer_eval(mycls().fvcl(1) == 8);   // expected-warning{{TRUE}}
-  clang_analyzer_eval(derived().fvcl(1) == 9); // expected-warning{{TRUE}}
+  clang_analyzer_eval(mycls().fvcl(1) == 8);   // newctu-warning{{TRUE}} ctu
+                                               // newctu-warning@-1{{UNKNOWN}} stu
+                                               // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(derived().fvcl(1) == 9); // newctu-warning{{TRUE}} ctu
+                                               // newctu-warning@-1{{UNKNOWN}} stu
+                                               // oldctu-warning@-2{{TRUE}}
   // We cannot decide about the dynamic type.
-  clang_analyzer_eval(obj->fvcl(1) == 8);      // expected-warning{{FALSE}} expected-warning{{TRUE}}
+  clang_analyzer_eval(obj->fvcl(1) == 8);      // newctu-warning{{TRUE}} ctu
+                                               // newctu-warning@-1{{UNKNOWN}} ctu, stu
+                                               // oldctu-warning@-2{{TRUE}}
+                                               // oldctu-warning@-3{{UNKNOWN}}
 }
 
 class TestAnonUnionUSR {
@@ -137,44 +158,92 @@ extern int testImportOfIncompleteDefaultParmDuringImport(int);
 extern int testImportOfDelegateConstructor(int);
 
 int main() {
-  clang_analyzer_eval(f(3) == 2); // expected-warning{{TRUE}}
-  clang_analyzer_eval(f(4) == 3); // expected-warning{{TRUE}}
-  clang_analyzer_eval(f(5) == 3); // expected-warning{{FALSE}}
-  clang_analyzer_eval(g(4) == 6); // expected-warning{{TRUE}}
-  clang_analyzer_eval(h(2) == 8); // expected-warning{{TRUE}}
-
-  clang_analyzer_eval(myns::fns(2) == 9);                   // expected-warning{{TRUE}}
-  clang_analyzer_eval(myns::embed_ns::fens(2) == -1);       // expected-warning{{TRUE}}
-  clang_analyzer_eval(mycls().fcl(1) == 6);                 // expected-warning{{TRUE}}
-  clang_analyzer_eval(mycls::fscl(1) == 7);                 // expected-warning{{TRUE}}
-  clang_analyzer_eval(myns::embed_cls().fecl(1) == -6);     // expected-warning{{TRUE}}
-  clang_analyzer_eval(mycls::embed_cls2().fecl2(0) == -11); // expected-warning{{TRUE}}
-
-  clang_analyzer_eval(chns::chf1(4) == 12); // expected-warning{{TRUE}}
-  clang_analyzer_eval(fun_using_anon_struct(8) == 8); // expected-warning{{TRUE}}
-
-  clang_analyzer_eval(other_macro_diag(1) == 1); // expected-warning{{TRUE}}
-  // expected-warning@Inputs/ctu-other.cpp:93{{REACHABLE}}
-  MACRODIAG(); // expected-warning{{REACHABLE}}
-
-  clang_analyzer_eval(extInt == 2); // expected-warning{{TRUE}}
-  clang_analyzer_eval(intns::extInt == 3); // expected-warning{{TRUE}}
-  clang_analyzer_eval(extS.a == 4); // expected-warning{{TRUE}}
-  clang_analyzer_eval(extNonConstS.a == 4); // expected-warning{{TRUE}} expected-warning{{FALSE}}
+  clang_analyzer_eval(f(3) == 2); // newctu-warning{{TRUE}} ctu
+                                  // newctu-warning@-1{{UNKNOWN}} stu
+                                  // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(f(4) == 3); // newctu-warning{{TRUE}} ctu
+                                  // newctu-warning@-1{{UNKNOWN}} stu
+                                  // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(f(5) == 3); // newctu-warning{{FALSE}} ctu
+                                  // newctu-warning@-1{{UNKNOWN}} stu
+                                  // oldctu-warning@-2{{FALSE}}
+  clang_analyzer_eval(g(4) == 6); // newctu-warning{{TRUE}} ctu
+                                  // newctu-warning@-1{{UNKNOWN}} stu
+                                  // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(h(2) == 8); // newctu-warning{{TRUE}} ctu
+                                  // newctu-warning@-1{{UNKNOWN}} stu
+                                  // oldctu-warning@-2{{TRUE}}
+
+  clang_analyzer_eval(myns::fns(2) == 9);                   // newctu-warning{{TRUE}} ctu
+                                                            // newctu-warning@-1{{UNKNOWN}} stu
+                                                            // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(myns::embed_ns::fens(2) == -1);       // newctu-warning{{TRUE}} ctu
+                                                            // newctu-warning@-1{{UNKNOWN}} stu
+                                                            // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(mycls().fcl(1) == 6);                 // newctu-warning{{TRUE}} ctu
+                                                            // newctu-warning@-1{{UNKNOWN}} stu
+                                                            // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(mycls::fscl(1) == 7);                 // newctu-warning{{TRUE}} ctu
+                                                            // newctu-warning@-1{{UNKNOWN}} stu
+                                                            // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(myns::embed_cls().fecl(1) == -6);     // newctu-warning{{TRUE}} ctu
+                                                            // newctu-warning@-1{{UNKNOWN}} stu
+                                                            // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(mycls::embed_cls2().fecl2(0) == -11); // newctu-warning{{TRUE}} ctu
+                                                            // newctu-warning@-1{{UNKNOWN}} stu
+                                                            // oldctu-warning@-2{{TRUE}}
+
+  clang_analyzer_eval(chns::chf1(4) == 12); // newctu-warning{{TRUE}} ctu
+                                            // newctu-warning@-1{{UNKNOWN}} stu
+                                            // oldctu-warning@-2{{TRUE}}
+  clang_analyzer_eval(fun_using_anon_struct(8) == 8); // newctu-warning{{TRUE}} ctu
+                                                      // newctu-warning@-1{{UNKNOWN}} stu
+                                                      // oldctu-warning@-2{{TRUE}}
+
+  clang_analyzer_eval(other_macro_diag(1) == 1); // newctu-warning{{TRUE}} ctu
+                                                 // newctu-warning@-1{{UNKNOWN}} stu
+                                                 // oldctu-warning@-2{{TRUE}}
+  // newctu-warning@Inputs/ctu-other.cpp:93{{REACHABLE}}
+  // oldctu-warning@Inputs/ctu-other.cpp:93{{REACHABLE}}
+  MACRODIAG(); // newctu-warning{{REACHABLE}}
+               // oldctu-warning@-1{{REACHABLE}}
+
+  // FIXME we should report an UNKNOWN as well for all external variables!
+  clang_analyzer_eval(extInt == 2); // newctu-warning{{TRUE}}
+                                    // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(intns::extInt == 3); // newctu-warning{{TRUE}}
+                                           // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(extS.a == 4); // newctu-warning{{TRUE}}
+                                    // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(extNonConstS.a == 4); // newctu-warning{{UNKNOWN}}
+                                            // oldctu-warning@-1{{UNKNOWN}}
   // Do not import non-trivial classes' initializers.
-  clang_analyzer_eval(extNTS.a == 4); // expected-warning{{TRUE}} expected-warning{{FALSE}}
-  clang_analyzer_eval(extHere == 6); // expected-warning{{TRUE}}
-  clang_analyzer_eval(A::a == 3); // expected-warning{{TRUE}}
-  clang_analyzer_eval(extSC.a == 8); // expected-warning{{TRUE}}
-  clang_analyzer_eval(ST::sc.a == 2); // expected-warning{{TRUE}}
+  clang_analyzer_eval(extNTS.a == 4); // newctu-warning{{UNKNOWN}}
+                                      // oldctu-warning@-1{{UNKNOWN}}
+  clang_analyzer_eval(extHere == 6); // newctu-warning{{TRUE}}
+                                     // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(A::a == 3); // newctu-warning{{TRUE}}
+                                  // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(extSC.a == 8); // newctu-warning{{TRUE}}
+                                     // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(ST::sc.a == 2); // newctu-warning{{TRUE}}
+                                      // oldctu-warning@-1{{TRUE}}
   // clang_analyzer_eval(extSCN.scn.a == 9); // TODO
-  clang_analyzer_eval(extSubSCN.a == 1); // expected-warning{{TRUE}}
+  clang_analyzer_eval(extSubSCN.a == 1); // newctu-warning{{TRUE}}
+                                         // oldctu-warning@-1{{TRUE}}
   // clang_analyzer_eval(extSCC.a == 7); // TODO
-  clang_analyzer_eval(extU.a == 4); // expected-warning{{TRUE}}
-
-  clang_analyzer_eval(TestAnonUnionUSR::Test == 5); // expected-warning{{TRUE}}
-
-  clang_analyzer_eval(testImportOfIncompleteDefaultParmDuringImport(9) == 9); // expected-warning{{TRUE}}
-
-  clang_analyzer_eval(testImportOfDelegateConstructor(10) == 10); // expected-warning{{TRUE}}
+  clang_analyzer_eval(extU.a == 4); // newctu-warning{{TRUE}}
+                                    // oldctu-warning@-1{{TRUE}}
+  clang_analyzer_eval(TestAnonUnionUSR::Test == 5); // newctu-warning{{TRUE}}
+                                                    // oldctu-warning@-1{{TRUE}}
+
+  clang_analyzer_eval(testImportOfIncompleteDefaultParmDuringImport(9) == 9);
+  // newctu-warning@-1{{TRUE}} ctu
+  // newctu-warning@-2{{UNKNOWN}} stu
+  // oldctu-warning@-3{{TRUE}}
+
+  clang_analyzer_eval(testImportOfDelegateConstructor(10) == 10);
+  // newctu-warning@-1{{TRUE}} ctu
+  // newctu-warning@-2{{UNKNOWN}} stu
+  // oldctu-warning@-3{{TRUE}}
 }
index 07a72a1..8159210 100644 (file)
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=. \
 // RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -analyzer-config ctu-phase1-inlining=all \
 // RUN:   -verify ctu-on-demand-parsing.c
 //
+// FIXME: On-demand ctu should be tested in the same file that we have for the
+// PCH version, but with a different verify prefix (e.g. -verfiy=on-demand-ctu)
+//
 // FIXME: Path handling should work on all platforms.
 // REQUIRES: system-linux
 
index e4e998c..d28d3c2 100644 (file)
@@ -18,6 +18,7 @@
 // RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
 // RUN:   -analyzer-config ctu-dir=. \
 // RUN:   -analyzer-config ctu-invocation-list=invocations.yaml \
+// RUN:   -analyzer-config ctu-phase1-inlining=all \
 // RUN:   -verify ctu-on-demand-parsing.cpp
 // RUN: cd "%t" && %clang_analyze_cc1 \
 // RUN:   -analyzer-checker=core,debug.ExprInspection \
@@ -28,6 +29,9 @@
 //
 // CHECK: CTU loaded AST file: {{.*}}ctu-other.cpp
 // CHECK: CTU loaded AST file: {{.*}}ctu-chain.cpp
+
+// FIXME: On-demand ctu should be tested in the same file that we have for the
+// PCH version, but with a different verify prefix (e.g. -verfiy=on-demand-ctu)
 //
 // FIXME: Path handling should work on all platforms.
 // REQUIRES: system-linux
diff --git a/clang/test/Analysis/ctu-onego-existingdef.cpp b/clang/test/Analysis/ctu-onego-existingdef.cpp
new file mode 100644 (file)
index 0000000..e60c9b3
--- /dev/null
@@ -0,0 +1,67 @@
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyze-function='baruser(int)' -x c++ \
+// RUN:   -verify=nonctu %s
+
+// RUN: rm -rf %t && mkdir %t
+// RUN: mkdir -p %t/ctudir
+// RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -emit-pch -o %t/ctudir/ctu-onego-existingdef-other.cpp.ast %S/Inputs/ctu-onego-existingdef-other.cpp
+// RUN: cp %S/Inputs/ctu-onego-existingdef-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
+
+// Existing and equal function definition in both TU. `other` calls `bar` thus
+// `bar` will be indirectly imported. During the import we recognize that there
+// is an existing definition in the main TU, so we don't create a new Decl.
+// Thus, ctu should not bifurcate on the call of `bar` it should directly
+// inlinie that as in the case of nonctu.
+// Note, we would not get a warning below, if `bar` is conservatively evaluated.
+int bar() {
+  return 0;
+}
+
+//Here we completely supress the CTU work list execution. We should not
+//bifurcate on the call of `bar`. (We do not load the foreign AST at all.)
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -verify=stu %s \
+// RUN:   -analyze-function='baruser(int)' -x c++ \
+// RUN:   -analyzer-config ctu-max-nodes-pct=0 \
+// RUN:   -analyzer-config ctu-max-nodes-min=0
+
+//Here we enable the CTU work list execution. We should not bifurcate on the
+//call of `bar`.
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -verify=ctu %s \
+// RUN:   -analyze-function='baruser(int)' -x c++ \
+// RUN:   -analyzer-config ctu-max-nodes-pct=100 \
+// RUN:   -analyzer-config ctu-max-nodes-min=1000
+//Check that the AST file is loaded.
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyze-function='baruser(int)' -x c++ \
+// RUN:   -analyzer-config ctu-max-nodes-pct=100 \
+// RUN:   -analyzer-config display-ctu-progress=true \
+// RUN:   -analyzer-config ctu-max-nodes-min=1000 2>&1 %s | FileCheck %s
+// CHECK: CTU loaded AST file
+
+void other(); // Defined in the other TU.
+
+void baruser(int) {
+  other();
+  int x = bar();
+  (void)(1 / x);
+  // ctu-warning@-1{{Division by zero}}
+  // stu-warning@-2{{Division by zero}}
+  // nonctu-warning@-3{{Division by zero}}
+}
diff --git a/clang/test/Analysis/ctu-onego-indirect.cpp b/clang/test/Analysis/ctu-onego-indirect.cpp
new file mode 100644 (file)
index 0000000..9a95ce6
--- /dev/null
@@ -0,0 +1,58 @@
+// RUN: rm -rf %t && mkdir %t
+// RUN: mkdir -p %t/ctudir
+// RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -emit-pch -o %t/ctudir/ctu-onego-indirect-other.cpp.ast %S/Inputs/ctu-onego-indirect-other.cpp
+// RUN: cp %S/Inputs/ctu-onego-indirect-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
+
+int bar();
+
+// Here we have a foreign function `bar` that is imported when we analyze
+// `adirectbaruser`. During the subsequent toplevel analysis of `baruser` we
+// should bifurcate on the call of `bar`.
+
+//Ensure the order of the toplevel analyzed functions.
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-display-progress \
+// RUN:   -analyzer-inlining-mode=all \
+// RUN:   -analyzer-config ctu-phase1-inlining=none \
+// RUN:   -analyzer-config ctu-max-nodes-pct=100 \
+// RUN:   -analyzer-config ctu-max-nodes-min=1000 2>&1 %s | FileCheck %s
+// CHECK: ANALYZE (Path,  Inline_Regular):{{.*}}adirectbaruser(int)
+// CHECK: ANALYZE (Path,  Inline_Regular):{{.*}}baruser(int)
+
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-display-progress \
+// RUN:   -analyzer-inlining-mode=all \
+// RUN:   -analyzer-config ctu-phase1-inlining=none \
+// RUN:   -verify %s \
+// RUN:   -analyzer-config ctu-max-nodes-pct=100 \
+// RUN:   -analyzer-config ctu-max-nodes-min=1000
+
+
+void other(); // Defined in the other TU.
+
+void clang_analyzer_eval(int);
+
+void baruser(int x) {
+  if (x == 1)
+    return;
+  int y = bar();
+  clang_analyzer_eval(y == 0); // expected-warning{{TRUE}}
+                               // expected-warning@-1{{UNKNOWN}}
+  other();
+}
+
+void adirectbaruser(int) {
+  int y = bar();
+  (void)y;
+  baruser(1);
+}
+
diff --git a/clang/test/Analysis/ctu-onego-small.cpp b/clang/test/Analysis/ctu-onego-small.cpp
new file mode 100644 (file)
index 0000000..c228757
--- /dev/null
@@ -0,0 +1,51 @@
+// RUN: rm -rf %t && mkdir %t
+// RUN: mkdir -p %t/ctudir
+// RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -emit-pch -o %t/ctudir/ctu-onego-small-other.cpp.ast %S/Inputs/ctu-onego-small-other.cpp
+// RUN: cp %S/Inputs/ctu-onego-small-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
+
+// Small function defined in another TU.
+int bar();
+
+// Here we limit the ctu analysis to the first phase only (via the
+// ctu-max-nodes config options). And we check whether the small foreign
+// function `bar` is inlined.
+
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-config display-ctu-progress=true \
+// RUN:   -analyzer-display-progress \
+// RUN:   -analyzer-config ctu-max-nodes-pct=0 \
+// RUN:   -analyzer-config ctu-max-nodes-min=0 2>&1 %s | FileCheck %s
+// CHECK: ANALYZE (Path,  Inline_Regular): {{.*}} baruser(int){{.*}}CTU loaded AST file
+
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-config ctu-max-nodes-pct=0 \
+// RUN:   -analyzer-config ctu-phase1-inlining=none \
+// RUN:   -analyzer-config ctu-max-nodes-min=0 -verify=inline-none %s
+
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-config ctu-max-nodes-pct=0 \
+// RUN:   -analyzer-config ctu-phase1-inlining=small \
+// RUN:   -analyzer-config ctu-max-nodes-min=0 -verify=inline-small %s
+
+
+void clang_analyzer_eval(int);
+
+void baruser(int x) {
+  int y = bar();
+  // inline-none-warning@+2{{UNKNOWN}}
+  // inline-small-warning@+1{{TRUE}}
+  clang_analyzer_eval(y == 0);
+}
diff --git a/clang/test/Analysis/ctu-onego-toplevel.cpp b/clang/test/Analysis/ctu-onego-toplevel.cpp
new file mode 100644 (file)
index 0000000..0cc313c
--- /dev/null
@@ -0,0 +1,54 @@
+// RUN: rm -rf %t && mkdir %t
+// RUN: mkdir -p %t/ctudir
+// RUN: %clang_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -emit-pch -o %t/ctudir/ctu-onego-toplevel-other.cpp.ast %S/Inputs/ctu-onego-toplevel-other.cpp
+// RUN: cp %S/Inputs/ctu-onego-toplevel-other.cpp.externalDefMap.ast-dump.txt %t/ctudir/externalDefMap.txt
+
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-config ctu-phase1-inlining=none \
+// RUN:   -verify=ctu %s
+
+// RUN: %clang_analyze_cc1 -std=c++14 -triple x86_64-pc-linux-gnu \
+// RUN:   -analyzer-checker=core,debug.ExprInspection \
+// RUN:   -analyzer-config eagerly-assume=false \
+// RUN:   -analyzer-config experimental-enable-naive-ctu-analysis=true \
+// RUN:   -analyzer-config ctu-dir=%t/ctudir \
+// RUN:   -analyzer-config ctu-phase1-inlining=none \
+// RUN:   -analyzer-config display-ctu-progress=true \
+// RUN:   -analyzer-display-progress \
+// RUN:   -verify=ctu %s 2>&1 | FileCheck %s
+
+// CallGraph: c->b
+// topological sort: c, b
+// Note that `other` calls into `b` but that is not visible in the CallGraph
+// because that happens in another TU.
+
+// During the onego CTU analysis, we start with c() as top level function.
+// Then we visit b() as non-toplevel during the processing of the FWList, thus
+// that would not be visited as toplevel without special care.
+
+// `c` is analyzed as toplevel and during that the other TU is loaded:
+// CHECK: ANALYZE (Path,  Inline_Regular): {{.*}} c(int){{.*}}CTU loaded AST file
+// next, `b` is analyzed as toplevel:
+// CHECK: ANALYZE (Path,  Inline_Regular): {{.*}} b(int)
+
+void b(int x);
+void other(int y);
+void c(int y) {
+  other(y);
+  return;
+  // The below call is here to form the proper CallGraph, but will not be
+  // analyzed.
+  b(1);
+}
+
+void b(int x) {
+  if (x == 0)
+    (void)(1 / x);
+    // ctu-warning@-1{{Division by zero}}
+    // We receive the above warning only if `b` is analyzed as top-level.
+}