erofs-utils: mkfs: introduce inner-file multi-threaded compression
authorYifan Zhao <zhaoyifan@sjtu.edu.cn>
Fri, 15 Mar 2024 01:10:19 +0000 (09:10 +0800)
committerGao Xiang <hsiangkao@linux.alibaba.com>
Sun, 7 Apr 2024 07:35:52 +0000 (15:35 +0800)
commit830b27bc233421b91ea66b55a3fcc69a99870af8
tree7b0b87005952d088e9c9292e33d8601bd47ff12e
parent0d6f9835ce27ead0f23848408366d9ca7f2b0d15
erofs-utils: mkfs: introduce inner-file multi-threaded compression

Currently, the creation of EROFS compressed image creation is
single-threaded, which suffers from performance issues. This patch
attempts to address it by compressing the large file in parallel.

Specifically, each input file larger than 16MiB is splited into
segments, and each worker thread compresses a segment as if it were
a separate file.  Finally, the main thread merges all the compressed
segments.

Multi-threaded compression is not compatible with -Ededupe,
-E(all-)fragments for now.

Signed-off-by: Yifan Zhao <zhaoyifan@sjtu.edu.cn>
Co-authored-by: Tong Xin <xin_tong@sjtu.edu.cn>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240315011019.610442-5-hsiangkao@linux.alibaba.com
Link: https://lore.kernel.org/r/ZfaW3oLe8Q2621DV@debian
include/erofs/compress.h
lib/compress.c
lib/compressor.c
mkfs/main.c