review.tizen.org Git - platform/upstream/erofs-utils.git/log

erofs-utils: lib: fix compressed packed inodes

Commit 2fdbd28 fixed uncompressed packed inodes by not always writing
compressed data, but it broke compressed packed inodes because now
uncompressed file data is always written after the compressed data.

The new error handling always rewinds with lseek and falls through to
write_uncompressed_file_from_fd, regardless of whether the compressed
data was written successfully (ret = 0) or not (ret = -ENOSPC). This
can result in corrupted files.

Fix it by simplifying the error handling to better match the old code.

Fixes: 2fdbd28 ("erofs-utils: lib: fix uncompressed packed inode")
Co-authored-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Signed-off-by: Danny Lin <danny@orbstack.dev>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240923051759.7563-2-danny@orbstack.dev
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: mkfs: add `--sort=none`

Currently, `--tar=f` writes file data twice due to unseekable streams
and EROFS data ordering requirements. Some use cases may need to avoid
unnecessary data writes for performance and do NOT require a strict
data ordering.

It adds `--sort=none` to address this. The image is still reproducible;
it simply means no specific file data ordering is implied.

Currently, It comes into effect if `-E^inline_data` is specified and no
compression is applied.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240923074929.2445674-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: get rid of outdated subpage compression warning

It has been supported since Linux 6.8.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240919105519.3656878-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: fix a regression where rebuild mode does not work

It will fail unexpectedly if any sub-image lacks an extra device (blob).

Fixes: 7550a30c332c ("erofs-utils: enable incremental builds")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240919021635.2922387-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix off-by-one issue with invalid device ID

The device ID should be no less than `1 + number of blobs`. In other
words, it should be greater than `number of blobs`.

Fixes: 89dfe997c2ee ("erofs-utils: lib: fix global-buffer-overflow due to invalid device")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240917091115.3920734-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix sorting shared xattrs

The length of xattr_item->kvbuf is calculated by EROFS_XATTR_KVSIZE,
and the key part has a trailing '\0' before the value part. When qsort
compares two xattr_items, the key-value length should be calculated by
EROFS_XATTR_KVSIZE, and use memcmp instead of strncmp to avoid key-value
string being cut by '\0'.

Fixes: 5df285cf405d ("erofs-utils: lib: refactor extended attribute name prefixes")
Signed-off-by: Sheng Yong <shengyong@oppo.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240913143542.3265071-1-shengyong@oppo.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: fsck: introduce exporting xattrs

Currently `fsck --extract` does not support exporting
extended attributes. This patch adds the `--xattrs` option
to dump extended attributes and the `--no-xattrs` option to
omit them (the default behavior).

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240913072100.577753-2-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: expose erofs_xattr_prefix_matches()

Prepare for the feature of exporting extended attributes for
`fsck.erofs`.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240913072100.577753-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: fix incorrect nblocks in block list for chunked inodes

Currently, the number of physical blocks (nblocks) for the last chunk
written to the block list file is incorrectly recorded as the inode
chunksize.

This patch writes the actual number of physical blocks for the inode in
the last chunk to the block list file.

Fixes: 7b46f7a0160a ("erofs-utils: lib: merge consecutive chunks if possible")
Fixes: b6749839e710 ("erofs-utils: generate preallocated extents for tarerofs")
Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240911085531.2133723-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: use another way to check power-of-2

Otherwise, Coverity will complain with "Operands don't affect result
(CONSTANT_EXPRESSION_RESULT)", which I don't think is an issue.

Coverity-id: 508261
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240910075256.3423180-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: fix inaccurate assertion of hardlinks in rebuild mode

`erofs_parent_inode(inode) == dir` is only true for non-hardlink
inodes and directories.

Only debug builds are impacted.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240909111305.1850391-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: tar: allow pax headers with empty names

Usually, `exthdr.name`s are set as "%d/PaxHeaders.%p/%f" [1].

However, since both `GNU tar` and `bsdtar` can process empty
`exthdr.name`s, let's allow them too.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html

Reported-by: Colin Walters <walters@verbum.org>
Fixes: 95d315fd7958 ("erofs-utils: introduce tarerofs")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240909093354.1712460-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: fix an undefined behavior of memcpy

Call trace:
- erofs_mkfs_build_tree
- erofs_mkfs_go(sbi, ~0, NULL, 0);

inode.c:1395:20: runtime error: null pointer passed as argument 2, which is declared to never be null
/usr/include/string.h:44:28: note: nonnull attribute specified here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior inode.c:1395:20 in

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240906083651.341555-1-hsiangkao@linux.alibaba.com

erofs-utils: fix invalid argument type in erofs_err()

Fix several issues found by Coverity regarding "Invalid type in argument
for printf format specifier".

Coverity-id: 502374, 502367, 502362, 502348, 502342, 502341,
502340, 502358
Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240830100354.2093735-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: don't include <lzma.h> and <zlib.h> in external headers

Applications don't need internal header dependencies.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240830065542.94908-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: actually skip the unidentified xattrs

parse_one_xattr() will return null if it detects unidentified xattr.
In such cases we need to skip this xattr which was our intention than
try to add it in erofs_xattr_add() which results in null pointer
dereference.

Fixes: 3037f8958f3b ("erofs-utils: skip all unidentified xattrs from local paths")
Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240820210123.2684886-2-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: mkfs: fix an indefinite wait race

Coverity reports: In erofs_mkfs_flushjobs, a thread waits for
a thread-shared condition that may have already been satisfied,
causing a hang.

It might indeed happen if the dfops worker runs with the specific
timing. Let's try to fix it and see if the report is cleared.

Coverity-id: 502330
Fixes: 37e5abcd8720 ("erofs-utils: mkfs: assign root NID in the main thread")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240821034326.2464146-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix potential overflow issue

Change 8U to 8ULL to avoid arithmetic multiplication overflow.

Coverity-id: 502353
Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240813121023.781122-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: adjust volume label maximum length to the kernel implementation

The erofs implementation of kernel has limitation of the volume label length.
The volume label data size of super block is 16 bytes.
The kernel implementation requires to null terminate inside a that 16 bytes.

Logs:
  $ ./mkfs/mkfs.erofs test16.erofs -L 0123456789abcdef test/
  $ mount -o loop ./test16.erofs ./mnt/
  $ dmesg
  [26477.019283] erofs: (device loop0): erofs_read_superblock: bad volume name without NIL terminator

  $ ./mkfs/mkfs.erofs test15.erofs -L 0123456789abcde test/
  $ mount -o loop ./test15.erofs ./mnt/
  $ dmesg
  [26500.516871] erofs: (device loop0): mounted with root inode @ nid 36.

This patch adjusts volume label maximum length to the kernel implementation.

Signed-off-by: Naoto Yamaguchi <naoto.yamaguchi@aisin.co.jp>
Link: https://lore.kernel.org/r/20240816031601.45848-1-naoto.yamaguchi@aisin.co.jp
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: use $EROFS_UTILS_VERSION, if set, as the version

This lets downstreams embed the unpolluted downstream version;
when building the Debian package (from gbp) the resulting binary yields
  $ mkfs.erofs
  <E> erofs: missing argument: FILE
  mkfs.erofs 1.8.1-fead89d91-dirty
  Try 'mkfs.erofs --help' for more information.

Now, d/rules can
  export EROFS_UTILS_VERSION := $(shell IFS="$$IFS()" read -r _ v _ < debian/changelog; echo "$$v")
yielding
  $ mkfs.erofs
  <E> erofs: missing argument: FILE
  mkfs.erofs 1.8.1-1
  Try 'mkfs.erofs --help' for more information.

Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Link: https://lore.kernel.org/r/gho2b67qax222ewv5xb5cjkkgjgzftr3pyecl536g6jshcfexa@tarta.nabijaczleweli.xyz
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: exclude: #include PATH_MAX workaround

Fixes build on the hurd.

Signed-off-by: Ahelenia Ziemiańska <nabijaczleweli@nabijaczleweli.xyz>
Link: https://lore.kernel.org/r/xs4azw3vs7oryqnpkvzsl6qbmma6p646igoklia2fextt6pdiw@tarta.nabijaczleweli.xyz
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: release 1.8.1

Signed-off-by: Gao Xiang <xiang@kernel.org>

erofs-utils: lib: fix heap-buffer-overflow on read

Wrap up into kite_mf_hc3_skip() for now.

Fixes: 861037f4fc15 ("erofs-utils: add a built-in DEFLATE compressor")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240809161536.3961647-1-hsiangkao@linux.alibaba.com

erofs-utils: fuse: fix partial decompression for libdeflate

Actually, libdeflate doesn't support partial decompression; therefore,
fix it by reallocating larger decompression buffers.

Although a better approach would be to obtain the exact decompressed
length instead for libdeflate decompressor, which requires more changes,
a quick fix is needed.

Fixes: 29b9e7140162 ("erofs-utils: fuse,fsck: add DEFLATE algorithm support")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240809105636.3641536-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix truncated uncompressed files

Some uncompressed files which are more than 4GiB can be truncated
incorrectly.

Fixes: 358177730598 ("erofs-utils: optimize write_uncompressed_file_from_fd()")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240809033747.3178912-1-hsiangkao@linux.alibaba.com

erofs-utils: release 1.8

Signed-off-by: Gao Xiang <xiang@kernel.org>

erofs-utils: lib: fix global-buffer-overflow due to invalid device

Fuzzer generates an image with crafted chunks of some invalid device.
Also refine the printed message of EOD.

Closes: https://github.com/erofs/erofsnightly/actions/runs/10172576269/job/28135408276
Closes: https://github.com/erofs/erofs-utils/issues/11
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240808160343.2544426-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: add `--mkfs-time` option

Some users need a fixed build time in the superblock for reproducible
builds rather than a fixed timestamp everywhere.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240808094522.2161075-1-hsiangkao@linux.alibaba.com

erofs-utils: update README for the upcoming 1.8

Add descriptions to multi-threaded compression and reproducible builds.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240808050818.1822583-1-hsiangkao@linux.alibaba.com

erofs-utils: enable multi-threaded support for `-Eall-fragments`

Since `-Eall-fragments` packs the whole data into the special inode,
it's possible to use the multi-threaded compression for this.

Some users may be interested in `-Eall-fragments` for extreme
compression anyway.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240807085413.717066-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix potential overflow issue

Coverity-id: 502377
Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240805032510.2637488-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: disallow new algorithms on incremental builds

On-disk compression configurations are not rewritten on incremental
builds, therefore there is no way to add new algorithms in this mode.

Clean builds should be used instead.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240806000859.66658-1-xiang@kernel.org

erofs-utils: mkfs: fix uninitialized nblocks

Coverity-id: 502376
Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240805073953.2864289-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: fix invalid argument type in erofs_err()

Coverity-id: 502378
Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240805024408.2598464-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: fix missing argument to erofs_err()

Fixes: cdd34def670b ("erofs-utils: mkfs: introduce rebuild mode")
Coverity-id: 502380
Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240805022826.2581887-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: fix potential memory leak in erofs_export_xattr_ibody()

Although it won't happen in reality except for bugs.

Fixes: 8f93c2f83962 ("erofs-utils: mkfs: support inline xattr reservation for rootdirs")
Coverity-id: 507395
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240802062316.2368403-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix fd leak on failure in erofs_dev_open()

Coverity-id: 502356
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240802015527.2113797-3-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix out-of-bounds in erofs_io_xcopy()

Coverity-id: 502334
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240802015527.2113797-2-hsiangkao@linux.alibaba.com

erofs-utils: fsck: fix fd leak on failure in erofs_extract_file()

Ignore the return values as other close()s instead.

Coverity-id: 502331
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240802015527.2113797-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: uuid: fix compilation error if __NR_getrandom doesn't exist

Let's use rand() for insecure randomness without getrandom() support
on very old kernels to resolve issues as below:

uuid.c: In function ‘s_getrandom’:
uuid.c:44:32: error: ‘__NR_getrandom’ undeclared (first use in this function); did you mean ‘s_getrandom’?
   44 |   ssize_t r = (ssize_t)syscall(__NR_getrandom, out, size, flags);
      |                                ^~~~~~~~~~~~~~
      |                                s_getrandom

I'm not sure who cares since most users just use `--with-uuid` instead.

Fixes: 5de439566bc5 ("erofs-utils: Provide identical functionality without libuuid")
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240729112524.930460-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: support inline xattr reservation for rootdirs

Due to the current on-disk limitation (16-bit on-disk root_nid), on-disk
root inodes must be updated in place for now.

If rootdir xattr sizes are expanded during incremental updates, there
may be insufficient space to keep additional extended attributes.

To work around this, let's add a mkfs option `--root-xattr-isize=#` to
specify the minimum inline xattr size of root directories in advance.

Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240729075027.712339-2-hsiangkao@linux.alibaba.com

erofs-utils: lib: allow xattr e_name_index as 0

Since it's implicitly supported since the initial EROFS kernel version.

It is particularly useful as an on-disk inode xattr placeholder for
on-disk core inode in-place updates, especially for root inodes because
"root_nid" recorded in the on-disk superblock is limited to 16 bits
for now.

Fixes: 1e429b74bff8 ("erofs-utils: lib: fix potential out-of-bound in xattr_entrylist()")
Reviewed-by: Sandeep Dhavale <dhavale@google.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240729075027.712339-1-hsiangkao@linux.alibaba.com

erofs-utils: misc: Fix potential memory leak in realloc failure path

As realloc returns NULL on failure, the original value will be
overwritten if it is used as lvalue. Fix this by using a temporary
variable to hold the return value and exit with -ENOMEM in case of
failure. This patch fixes 2 of the realloc blocks with similar fix.

Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Link: https://lore.kernel.org/r/20240718202204.1224620-1-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: manpage: add more description for --extract option

Especially, extract files to a specific directory.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240715033829.2338056-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: tar: fix garbage ns timestamps

Some "#if" directives were used incorrectly.

Fixes: 95d315fd7958 ("erofs-utils: introduce tarerofs")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240714044119.1119717-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: fix -U option

`-U <UUID>` option cannot work properly now.

Fixes: 7550a30c332c ("erofs-utils: enable incremental builds")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240712093808.2986196-2-hsiangkao@linux.alibaba.com

erofs-utils: fix reproducible builds for multi-threaded libdeflate

`last_uncompressed_size` should be reset on the basis of segments.

Fixes: 830b27bc2334 ("erofs-utils: mkfs: introduce inner-file multi-threaded compression")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240712093808.2986196-1-hsiangkao@linux.alibaba.com

erofs-utils: add per-sbi buffer support

It updates all relevant function definitions and callers to get rid of
the global g_sbi, which will be used for external use.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240710082906.203180-2-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib/cache.c: replace &g_sbi with sbi

Prepare for the upcoming per-sbi buffers.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240710082906.203180-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: tar: support ddtaridx format informally

`ddtaridx` is a customized tar meta-only format implemented in
Alibaba's OverlayBD project [1].

Please don't use it externally if you have no idea of this except for
the OverlayBD project. It will be removed if some better way exists.

[1] https://github.com/containerd/overlaybd

Cc: Yifan Yuan <tuji.yyf@alibaba-inc.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240713064028.4134602-1-hsiangkao@linux.alibaba.com

erofs-utils: fix bitops fls_long()

`__builtin_clz` is for unsigned int, while it is now applied to
unsigned long. This fixes it by using `__builtin_clzl`.

`roundup_pow_of_two()` in `erofs_init_devices()` could give wrong
results although the current compiler optimization level "-O2"
covers it up.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240709073819.3061805-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: rename the global sbi to g_sbi

Rename the global `sbi` to `g_sbi` to prepare for
the upcoming per-sbi buffer management.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240704050259.520618-2-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: get rid of global sbi in lib/inode.c

Get rid of the global sbi when EROFS_MT_ENABLED is defined.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240704050259.520618-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: change function definition of erofs_blocklist_open()

Modify erofs_blocklist_open() to accept a file pointer instead of
a file path, making it suitable for external use in liberofs.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240703030327.3280503-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: rebuild: only update dev/i_ino[1] pairs for directories

Since the underlying dev/i_ino[1] pairs are only useful for merged
sub-directories, don't bother with other types of inodes.

Otherwise, the original i_ino[1] could be overwritten unexpectedly,
which impacts resvsp mode at least..

Fixes: f64d9d02576b ("erofs-utils: introduce incremental builds")
Reported-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240702083144.2120808-1-hsiangkao@linux.alibaba.com

erofs-utils: fix "non-trivial designated initializers not supported"

This partially reverts commit 79f6e168d94c ("erofs-utils: improve
compatibility and reduce header conflicts") since some C++ compiler
will complain:

include/erofs_fs.h: In function 'void erofs_check_ondisk_layout_definitions()':
include/erofs_fs.h:460:2: sorry, unimplemented: non-trivial designated initializers not supported

Let's just bypass this compile-time check for the C++ language since
only external programs may be written in C++.

Fixes: 79f6e168d94c ("erofs-utils: improve compatibility and reduce header conflicts")
Cc: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240627031343.3424030-1-hsiangkao@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: add erofs_get_configure()

This adds `erofs_get_configure()` to get the global configuration
`cfg`. It allows external entities to change the global configuration
through this helper, thereby controlling the EROFS mkfs process.

It is just a temporary helper for liberofs and it will be deprecated
in the future. Don't rely on it too much.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240627022741.3912785-1-hongzhen@linux.alibaba.com
[ Gao Xiang: minor commit message update. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: add erofs_{rebuild_make_root,enable_sb_chksum}

Move erofs_sb_csum_set() and erofs_mkfs_alloc_root() into liberofs
for external use.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240625062458.1514209-1-hongzhen@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: improve compatibility and reduce header conflicts

Adjust initializers of union in erofs-utils to ensure compatibility
with various compilers. The original C99 designated initializer style
was not supported in other compilers (e.g., C++11), leading to build
failures. Additionally, change the codebase to minimize potential
conflicts with headers from other projects.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240624063217.170251-1-hongzhen@linux.alibaba.com
[ Gao Xiang: minor commit message update. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: introduce `payload` field in `struct erofs_vfile`

Allow customized `vfile` with non-NULL `ops` utilizing `payload`
for additional information.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240624115923.4090196-2-hsiangkao@linux.alibaba.com

erofs-utils: derive i_srcpath for erofs_rebuild_mkdir()

Also add missing erofs_iput() on errors.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240624115923.4090196-1-hsiangkao@linux.alibaba.com

erofs-utils: optimize write_uncompressed_file_from_fd()

Utilize copy offloading to speed up copying data from the source
filesystem to the target EROFS filesystem.

This method improves build speed by approximately 9% (tested with
Linux 5.4.140 source code dataset).

Reported-by: Daan De Meyer <daan.j.demeyer@gmail.com>
Closes: https://lore.kernel.org/r/CAO8sHcmZZORnrJXA=QzmGkYNkNWn7M+amAK_DZ19-WL4kLUvpw@mail.gmail.com
Link: https://lore.kernel.org/r/20240623115932.2696312-1-hsiangkao@linux.alibaba.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: skip all unidentified xattrs from local paths

Just warn out but continue. Don't over-complicate for now.

Reported-by: Gael Donval <gael.donval@manchester.ac.uk>
Closes: https://lore.kernel.org/r/4abed942399fb29933f0fa85cc55d3d795ae8bcd.camel@manchester.ac.uk
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618111700.267702-1-hsiangkao@linux.alibaba.com

erofs-utils: enable mapfile for `--tar=f`

The data offsets in the tar streams can always be looked up now:
mkfs.erofs --tar=f,MAPFILE IMAGE TARBALL

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-9-hsiangkao@linux.alibaba.com

erofs-utils: enable incremental builds

`--incremental=<data|rvsp>` are now supported for tarerofs but
only `--incremental=rvsp` works for the rebuild mode.

For example:
$ mkfs.erofs --tar=f --gzip --aufs --clean=data foo.erofs f0.tgz
$ mkfs.erofs --tar=f --gzip --aufs --incremental=data foo.erofs f1.tgz
...
$ mkfs.erofs --tar=f --gzip --aufs --incremental=data foo.erofs fn-1.tgz

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-8-hsiangkao@linux.alibaba.com

erofs-utils: fix incremental builds for tarerofs index mode

The blob data area should be considered in the total block number to
prevent overlap during incremental builds.

Fixes: b6749839e710 ("erofs-utils: generate preallocated extents for tarerofs")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-7-hsiangkao@linux.alibaba.com

erofs-utils: support building image with reserved space

A new mode is prepared in order to preallocate/reserve data blocks only
since some applications tend to fill data after EROFS images are built.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-6-hsiangkao@linux.alibaba.com

erofs-utils: fix up unchanged directory pNIDs for incremental builds

For incremental builds, it's unnecessary to dump all unchanged
directories, yet the pNIDs of those directories need to be fixed to
the new parent on-disk inodes.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-5-hsiangkao@linux.alibaba.com

erofs-utils: introduce incremental builds

This introduces incremental build support for mkfs, where new on-disk
(meta)data will be appended in a log-structured manner, except for the
root inode (due to current on-disk limitations), as illustrated below:

___________________________________________
| base | delta 0 | delta 1 | .. | delta n-1 |
|______|_________|_________|____|___________|
---> image/data growth direction

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-4-hsiangkao@linux.alibaba.com

erofs-utils: fix up root inode for incremental builds

Move the new root inode to the original location if it cannot
be accessed by the super block.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-3-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: minor cleanup & rearrangement

Introduce erofs_flush_packed_inode() and more for exporting liberofs
APIs later.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-2-hsiangkao@linux.alibaba.com

erofs-utils: simplify erofs_insert_ihash

Get rid of unnecessary arguments for simplicity.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240618082414.47876-1-hsiangkao@linux.alibaba.com

erofs-utils: fix erofs_io_p{read,write} and erofs_dev_close

erofs_io_p{read,write} should return the number of bytes
successfully {read,write}.

This also fixes `erofs_dev_close` which could close random
fds if `vf->ops` is NULL.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240619025024.1109782-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: get rid of global sbi in lib/inode.c

In order to prepare for incremental builds.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240612161826.711279-5-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: assign root NID in the main thread

Thus it can be customized (skipped), especially for incremental builds.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240612161826.711279-4-hsiangkao@linux.alibaba.com

erofs-utils: wrap up superblock reservation for incremental builds

Refactor `erofs_buffer_init()` to wrap up necessary operations for full
builds.

Introduce another `erofs_buffer_init()` to specify start block address
for the upcoming incremental builds.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240612161826.711279-3-hsiangkao@linux.alibaba.com

erofs-utils: lib: use filesystem UUID if the device name is not specified

The device name is not always valid.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240612161826.711279-2-hsiangkao@linux.alibaba.com

erofs-utils: lib: get rid of erofs_prepare_dir_layout()

Just open-code the previous erofs_prepare_dir_file() and rename
`erofs_prepare_dir_layout()` to `erofs_prepare_dir_file()`.

No logic changes.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240612161826.711279-1-hsiangkao@linux.alibaba.com

erofs-utils: fix incorrect i_nlink in the unified rebuild logic

Fixes: 203c847cc7d1 ("erofs-utils: unify the tree traversal for the rebuild mode")
Closes: https://github.com/erofs/erofsnightly/actions/runs/9492427961/job/26159566596
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240613023706.1269816-1-hsiangkao@linux.alibaba.com

erofs-utils: add I/O control for tarerofs stream via `erofs_vfile`

This adds I/O control for tarerofs stream.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240612031124.1227558-1-hongzhen@linux.alibaba.com
[ Gao Xiang: code styling fixups. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: fix the current rebuild mode

`inode->with_diskbuf` can be false in the rebuild mode since
inode data has been mapped before.

Fixes: 203c847cc7d1 ("erofs-utils: unify the tree traversal for the rebuild mode")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240612021617.4025762-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: split erofs_iflush()

So that external programs can directly use it.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240607095319.2169172-2-hsiangkao@linux.alibaba.com

erofs-utils: move erofs_writesb() into lib/

So that external programs can directly use it.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240607095319.2169172-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: support virtual files

The current erofs-utils I/O implementation is through file descriptors.
The new `erofs_vfile` provides a more flexible way to perform I/Os.

Signed-off-by: Hongzhen Luo <hongzhen@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240606111833.2389455-1-hongzhen@linux.alibaba.com
[ Gao Xiang: minor styling fixes. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: build: support building static library liberofsfuse

Add new option '--enable-static-fuse' so that we
could import erofsfuse as a static library directly
into other projects

Signed-off-by: ComixHe <heyuming@deepin.org>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/3399126AB01D5AB6+2bad5767fc035a7a2234408b0fffa53b3a07aa51.1717659178.git.heyuming@deepin.org
[ Gao Xiang: the target static library shouldn't have dependencies. ]
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: support Intel Query Processing Library

This adds the preliminary Intel QPL [1] support to enable built-in
In-Memory Analytics Accelerator [2] started from Sapphire Rapids.

It just leverages the synchronous APIs for the sake of simplicity
for now, thus performance for small compressed clusters can still
be improved in the future if needed anyway.

[ QPL 1.5.0+ is strictly needed for pkg-config detection and
  it can be explicitly enabled by `--with-qpl`. ]

Here are some performance numbers for reference:

Processors: Intel(R) Xeon(R) Platinum 8475B (192 cores)
Memory:     512 GiB
Dataset:    enwik9 (1000000000) [3]

Single-threaded decompression:
______________________________________________________________
|                 |_ Cluster size _|_ Image size _|_ Time (s) _|
| LZ4             |     65536      |  391581696   |   0.364    |
| LZ4             |    1048576     |  373309440   |   0.376    |
| Intel QPL (IAA) |    1048576     |  374816768   |   0.386    |
| Intel QPL (IAA) |     65536      |  376057856   |   0.396    |
| Intel QPL (IAA) |      4096      |  399650816   |   0.675    |
| libdeflate (4k) |    1048576     |  374816768   |   1.862    |
| libdeflate (4k) |     65536      |  376057856   |   1.859    |
| libdeflate (4k) |      4096      |  399749120   |   2.203    |
| libdeflate      |    1048576     |  323457024   |   1.318    |
| libdeflate      |     65536      |  328712192   |   1.358    |
| libdeflate      |      4096      |  389943296   |   2.103    |
| Zstd            |      N/A       |  312548986   |   1.047    |
| Zstd (fast)     |      N/A       |  453096980   |   0.740    |
|_________________|________________|______________|____________|

LZ4 1.9.4: [ mkfs.erofs -zlz4hc,12 -C65536 ]
           [ mkfs.erofs -zlz4hc,12 -C1048576 ]
    time fsck/fsck.erofs --extract

QPL 1.5.0 (IAA) / libdeflate 1.20 (4k):
           [ mkfs.erofs -zdeflate,level=9,dictsize=4096 -C1048576 ]
           [ mkfs.erofs -zdeflate,level=9,dictsize=4096 -C65536 ]
           [ mkfs.erofs -zdeflate,level=9,dictsize=4096 -C4096 ]
    time fsck/fsck.erofs --extract

libdeflate 1.20:
           [ mkfs.erofs -zdeflate,level=9 -C1048576 ]
           [ mkfs.erofs -zdeflate,level=9 -C65536 ]
           [ mkfs.erofs -zdeflate,level=9 -C4096 ]
    time fsck/fsck.erofs --extract

Zstd 1.5.6: [ zstd -k ] [ zstd -k --fast ]
    time zstd -d -k -f -c --no-progress > /dev/null

[1] https://github.com/intel/qpl
[2] https://www.intel.com/content/www/us/en/products/docs/accelerator-engines/in-memory-analytics-accelerator.html
[3] https://www.mattmahoney.net/dc/textdata.html

Cc: "Feghali, Wajdi K" <wajdi.k.feghali@intel.com>
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240605123233.3833332-1-hsiangkao@linux.alibaba.com

erofs-utils: introduce z_erofs_parse_cfgs()

This userspace implementation will be mainly used for the upcoming
Intel In-Memory Analytics Accelerator integration.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240605121448.3816160-1-hsiangkao@linux.alibaba.com

erofs-utils: record sb_size instead of sb_extslots

Just follow the kernel implementation.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240604084015.2291157-2-hsiangkao@linux.alibaba.com

erofs-utils: lib: wrap up zeropadding calculation

Use a simple helper instead of open-coding.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240604084015.2291157-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: fix incorrect xattr sharing

There are off-by-one issues after refactoring, and the size of kvbuf
should be calculated by EROFS_XATTR_KVSIZE instead.

Fixes: 5df285cf405d ("erofs-utils: lib: refactor extended attribute name prefixes")
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240531071305.1183728-1-hsiangkao@linux.alibaba.com

erofs-utils: fix false-positive errors on gcc 4.8.5

Just old compiler bugs.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240528064313.1352565-1-hsiangkao@linux.alibaba.com

erofs-utils: lib: improve freeing hashmap in erofs_blob_exit()

Depending on size of the filesystem being built there can be huge number
of elements in the hashmap. Currently we call hashmap_iter_first() in
while loop to iterate and free the elements. However technically
correct, this is inefficient in 2 aspects.

- As we are iterating the elements for removal, we do not need overhead of
rehashing.
- Second part which contributes hugely to the performance is using
hashmap_iter_first() as it starts scanning from index 0 throwing away
the previous successful scan. For sparsely populated hashmap this becomes
O(n^2) in worst case.

Lets fix this by disabling hashmap shrink which avoids rehashing
and use hashmap_iter_next() which is now guaranteed to iterate over
all the elements while removing while avoiding the performance pitfalls
of using hashmap_iter_first().

Test with random data shows performance improvement as:

fs_size  Before   After
1G 23s   7s
2G 81s      15s
4G 272s     31s
8G 1252s   61s

Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240523210131.3126753-3-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: provide helper to disable hashmap shrinking

This helper sets hasmap.shrink_at to 0. This is helpful to iterate over
hashmap using hashmap_iter_next() and use hashmap_remove() in single
pass efficeintly.

Signed-off-by: Sandeep Dhavale <dhavale@google.com>
Reviewed-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240523210131.3126753-2-dhavale@google.com
Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>

erofs-utils: lib: fix uncompressed packed inode

Currently, packed inode can also be used in the unencoded way
such as xattr prefixes.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240523025550.2447091-1-hsiangkao@linux.alibaba.com

erofs-utils: unify the tree traversal for the rebuild mode

Let's drop the legacy approach and `tarerofs` will be applied too.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240520060301.2642650-1-hsiangkao@linux.alibaba.com

erofs-utils: mkfs: add `--zfeature-bits` option

Thus, we could traverse all compression features with continuous
numbers easily in the testcases.

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240517090048.3039594-1-hsiangkao@linux.alibaba.com

erofs-utils: add preliminary zstd support [x]

This patch just adds a preliminary Zstandard support to erofs-utils
since currently Zstandard doesn't support fixed-sized output compression
officially.  Mkfs could take more time to finish but it works at least.

The built-in zstd compressor for erofs-utils is slowly WIP, therefore
apparently it will take more efforts.

[ TODO: Later I tend to add another way to generate fixed-sized input
        pclusters temporarily for relatively large pcluster sizes as
        an option since it will have minor impacts to the results. ]

Signed-off-by: Gao Xiang <hsiangkao@linux.alibaba.com>
Link: https://lore.kernel.org/r/20240515051641.3929058-1-hsiangkao@linux.alibaba.com