[modules] Support zstd in .pcm file
authorFangrui Song <i@maskray.me>
Wed, 23 Nov 2022 19:27:49 +0000 (11:27 -0800)
committerFangrui Song <i@maskray.me>
Wed, 23 Nov 2022 19:27:49 +0000 (11:27 -0800)
commitfa7bc386ec7565ccfdfa75b272079ee03604e3ba
tree1d84aa9fafdf47163c42c74d7d83b15b0f7c2650
parent4f9c9295a6057d752c1fe0220d797ee94b0a18cf
[modules] Support zstd in .pcm file

Extend SM_SLOC_BUFFER_BLOB_COMPRESSED to allow zstd, which is much faster
(compression/decompression) than zlib with a similar compression ratio.

An alternative is to add a value beside SM_SLOC_BUFFER_BLOB_COMPRESSED, but
reusing SM_SLOC_BUFFER_BLOB_COMPRESSED slightly simplifies the implementation
and leads to better diagnostics when a slightly older Clang consumes zstd
compressed blob.

Compressing AST takes a small portion of WriteAST, so we can pick a higher
compression level.

Compiling a relatively large .pcm (absl endian) with -fmodules-embed-all-files,
zstd level 9 has comparable performance with zlib-chromium level 6 (default),
but provides smaller output (5809156 => 5796016). Higher zstd levels will make
"Compress AST" notably slower and do not provide significant more size saving.

```
2.219345 Total ExecuteCompiler
0.746799 Total Frontend
0.736862 Total Source
0.339434 Total ReadAST
0.165452 Total WriteAST
0.043045 Total Compress AST
0.008236 Total ParseClass
0.00633 Total InstantiateClass
0.001887 Total isPotentialConstantExpr
0.001808 Total InstantiateFunction
0.001535 Total EvaluateForOverflow
0.000986 Total EvaluateAsRValue
0.000536 Total EvaluateAsBooleanCondition
0.000308 Total EvaluateAsConstantExpr
0.000156 Total EvaluateAsInt
3.4e-05 Total EvaluateKnownConstInt
8e-06 Total EvaluateAsInitializer
0 Total PerformPendingInstantiations
```

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D137885
clang/lib/Serialization/ASTReader.cpp
clang/lib/Serialization/ASTWriter.cpp
clang/test/Modules/embed-files-compressed.cpp