Modules: Cache PCMs in memory and avoid a use-after-free
authorDuncan P. N. Exon Smith <dexonsmith@apple.com>
Fri, 17 Mar 2017 22:55:13 +0000 (22:55 +0000)
committerDuncan P. N. Exon Smith <dexonsmith@apple.com>
Fri, 17 Mar 2017 22:55:13 +0000 (22:55 +0000)
commit079c40e8860ccbc80b5a04a26e474b2923d92d48
treeadc13f151cb13cd30bce6a5fed9c035dfa4e8179
parent77e6ebe748e5a36dd12d44d31c3987ca56af2d43
Modules: Cache PCMs in memory and avoid a use-after-free

Clang's internal build system for implicit modules uses lock files to
ensure that after a process writes a PCM it will read the same one back
in (without contention from other -cc1 commands).  Since PCMs are read
from disk repeatedly while invalidating, building, and importing, the
lock is not released quickly.  Furthermore, the LockFileManager is not
robust in every environment.  Other -cc1 commands can stall until
timeout (after about eight minutes).

This commit changes the lock file from being necessary for correctness
to a (possibly dubious) performance hack.  The remaining benefit is to
reduce duplicate work in competing -cc1 commands which depend on the
same module.  Follow-up commits will change the internal build system to
continue after a timeout, and reduce the timeout.  Perhaps we should
reconsider blocking at all.

This also fixes a use-after-free, when one part of a compilation
validates a PCM and starts using it, and another tries to swap out the
PCM for something new.

The PCMCache is a new type called MemoryBufferCache, which saves memory
buffers based on their filename.  Its ownership is shared by the
CompilerInstance and ModuleManager.

  - The ModuleManager stores PCMs there that it loads from disk, never
    touching the disk if the cache is hot.

  - When modules fail to validate, they're removed from the cache.

  - When a CompilerInstance is spawned to build a new module, each
    already-loaded PCM is assumed to be valid, and is frozen to avoid
    the use-after-free.

  - Any newly-built module is written directly to the cache to avoid the
    round-trip to the filesystem, making lock files unnecessary for
    correctness.

Original patch by Manman Ren; most testcases by Adrian Prantl!

llvm-svn: 298165
34 files changed:
clang/include/clang/Basic/DiagnosticSerializationKinds.td
clang/include/clang/Basic/MemoryBufferCache.h [new file with mode: 0644]
clang/include/clang/Frontend/ASTUnit.h
clang/include/clang/Frontend/CompilerInstance.h
clang/include/clang/Lex/Preprocessor.h
clang/include/clang/Serialization/ASTReader.h
clang/include/clang/Serialization/ASTWriter.h
clang/include/clang/Serialization/Module.h
clang/include/clang/Serialization/ModuleManager.h
clang/lib/Basic/CMakeLists.txt
clang/lib/Basic/MemoryBufferCache.cpp [new file with mode: 0644]
clang/lib/Frontend/ASTUnit.cpp
clang/lib/Frontend/CompilerInstance.cpp
clang/lib/Lex/Preprocessor.cpp
clang/lib/Serialization/ASTReader.cpp
clang/lib/Serialization/ASTWriter.cpp
clang/lib/Serialization/GeneratePCH.cpp
clang/lib/Serialization/ModuleManager.cpp
clang/test/Modules/Inputs/system-out-of-date/X.h [new file with mode: 0644]
clang/test/Modules/Inputs/system-out-of-date/Y.h [new file with mode: 0644]
clang/test/Modules/Inputs/system-out-of-date/Z.h [new file with mode: 0644]
clang/test/Modules/Inputs/system-out-of-date/module.map [new file with mode: 0644]
clang/test/Modules/Inputs/warning-mismatch/Mismatch.h [new file with mode: 0644]
clang/test/Modules/Inputs/warning-mismatch/System.h [new file with mode: 0644]
clang/test/Modules/Inputs/warning-mismatch/module.modulemap [new file with mode: 0644]
clang/test/Modules/outofdate-rebuild.m [new file with mode: 0644]
clang/test/Modules/system-out-of-date-test.m [new file with mode: 0644]
clang/test/Modules/warning-mismatch.m [new file with mode: 0644]
clang/unittests/Basic/CMakeLists.txt
clang/unittests/Basic/MemoryBufferCacheTest.cpp [new file with mode: 0644]
clang/unittests/Basic/SourceManagerTest.cpp
clang/unittests/Lex/LexerTest.cpp
clang/unittests/Lex/PPCallbacksTest.cpp
clang/unittests/Lex/PPConditionalDirectiveRecordTest.cpp