David Sterba [Tue, 12 Jul 2016 15:47:23 +0000 (17:47 +0200)]
btrfs-progs: docs: update btrfs-subvolume manual page
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Tue, 12 Jul 2016 15:13:16 +0000 (17:13 +0200)]
btrfs-progs: docs: update btrfs-balance manual page
Signed-off-by: David Sterba <dsterba@suse.com>
Wang Shilong [Thu, 7 Jul 2016 09:43:38 +0000 (18:43 +0900)]
btrfs-progs: du: fix to skip not btrfs dir/file
'btrfs file du' is a very useful tool to watch my system
file usage information with snapshot aware.
when trying to run following commands:
[root@localhost btrfs-progs]# btrfs file du /
Total Exclusive Set shared Filename
ERROR: Failed to lookup root id - Inappropriate ioctl for device
ERROR: cannot check space of '/': Unknown error -1
and My Filesystem looks like this:
[root@localhost btrfs-progs]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 16G 0 16G 0% /dev
tmpfs tmpfs 16G 368K 16G 1% /dev/shm
tmpfs tmpfs 16G 1.4M 16G 1% /run
tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/sda3 btrfs 60G 19G 40G 33% /
tmpfs tmpfs 16G 332K 16G 1% /tmp
/dev/sdc btrfs 2.8T 166G 1.7T 9% /data
/dev/sda2 xfs 2.0G 452M 1.6G 23% /boot
/dev/sda1 vfat 1.9G 11M 1.9G 1% /boot/efi
tmpfs tmpfs 3.2G 24K 3.2G 1% /run/user/1000
So I installed Btrfs as my root partition, but boot partition
can be other fs.
We can Let btrfs tool aware of this is not a btrfs file or
directory and skip those files, so that someone like me
could just run 'btrfs file du /' to scan all btrfs filesystems.
After patch, it will look like:
Total Exclusive Set shared Filename
0.00B 0.00B - //root/.bash_logout
0.00B 0.00B - //root/.bash_profile
0.00B 0.00B - //root/.bashrc
0.00B 0.00B - //root/.cshrc
0.00B 0.00B - //root/.tcshrc
This works for me to analysis system usage and analysis
performaces.
Signed-off-by: Wang Shilong <wangshilong1991@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Tsutomu Itoh [Thu, 9 Jun 2016 01:23:15 +0000 (10:23 +0900)]
btrfs-progs: add the error message when open fails
When open in btrfs_open_devices failed, only the following message is
displayed. Therefore the user doesn't understand the reason why open
failed.
# btrfs check /dev/sdb8
Couldn't open file system
This patch adds the error message when open fails.
Signed-off-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 8 Jul 2016 15:04:25 +0000 (17:04 +0200)]
btrfs-progs: docs: man5, document control device
Signed-off-by: David Sterba <dsterba@suse.com>
Luis Henriques [Mon, 4 Jul 2016 22:48:47 +0000 (23:48 +0100)]
btrfs-progs: tests: 006-image-on-missing-device: fix btrfs tool path
If btrfs isn't in the path, this test will fail with:
[TEST/misc] 006-image-on-missing-device
failed: btrfs fi show /dev/loop0
test failed for case 006-image-on-missing-device
Makefile:226: recipe for target 'test-misc' failed
make: *** [test-misc] Error 1
Fix the test script by adding $TOP to the path.
Signed-off-by: Luis Henriques <henrix@camandro.org>
[ updated to full command names ]
Signed-off-by: David Sterba <dsterba@suse.com>
Mark Fasheh [Mon, 4 Jul 2016 12:57:38 +0000 (14:57 +0200)]
btrfs-progs: check: write corrected qgroup info to disk
Now that we can verify all qgroups, we can write the corrected qgroups out
to disk when '--repair' is specified. The qgroup status item is also updated
to clear any out-of-date state. The repair_ functions were modeled after the
inode repair code in cmds-check.c.
I also renamed the 'scan' member of qgroup_status_item to 'rescan' in order
to keep consistency with the kernel.
Testing this was easy, I just reproduced qgroup inconsistencies via the
usual routes and had btrfsck fix them.
Signed-off-by: Mark Fasheh <mfasheh@suse.de>
Signed-off-by: David Sterba <dsterba@suse.com>
Mark Fasheh [Fri, 17 Jun 2016 20:37:48 +0000 (13:37 -0700)]
btrfs-progs: check: verify qgroups above level 0
At the moment we only check subvolume quota groups (level 0). With this
patch we can check groups above 0, thus verifying the entire qgroup
hierarchy on a file system. The accounting portion of this patch is modeled
after the kernel - we are essentially reimplementing the 'quota rescan' case
here. Most other sections of this code went unchanged, in particular the
root counting works independently of the accounting.
Signed-off-by: Mark Fasheh <mfasheh@suse.de>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Mon, 4 Jul 2016 13:22:53 +0000 (15:22 +0200)]
btrfs-progs: factor out repair mode
Signed-off-by: David Sterba <dsterba@suse.com>
Jeff Mahoney [Thu, 23 Jun 2016 19:26:06 +0000 (15:26 -0400)]
btrfs-progs: check: switch to iterating over the backref_tree
We now have two data structures that can be used to iterate the same data
set, and there may be quite a few of them in memory. Eliminating the
list_head member will reduce memory consumption while iterating over
the extent backrefs.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Jeff Mahoney [Thu, 23 Jun 2016 19:26:05 +0000 (15:26 -0400)]
btrfs-progs: check: supplement extent backref list with rbtree
For the pathlogical case, like xfstests generic/297 that creates a
large file consisting of one, repeating reflinked extent, fsck can
take hours. The root cause is that calling find_data_backref while
iterating the extent records is an O(n^2) algorithm. For my
example test run, n was 2*2^20 and fsck was at 8 hours and counting.
This patch supplements the list with an rbtree and drops the runtime
of that testcase to about 20 seconds.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Jeff Mahoney [Thu, 23 Jun 2016 19:26:04 +0000 (15:26 -0400)]
btrfs-progs: check: add helpers for converting between structures
We either open code list_entry calls or outright cast between types. The
compiler will do the right thing if we use static inlines to do
typesafe conversions.
Signed-off-by: Jeff Mahoney <jeffm@suse.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Hans van Kranenburg [Sun, 3 Jul 2016 21:22:26 +0000 (23:22 +0200)]
btrfs-progs: use the correct struct for BTRFS_IOC_LOGICAL_INO
BTRFS_IOC_LOGICAL_INO takes a btrfs_ioctl_logical_ino_args as argument,
not a btrfs_ioctl_ino_path_args. The lines were probably copy/pasted
when the code was written.
Since btrfs_ioctl_logical_ino_args and btrfs_ioctl_ino_path_args have
the same size, the actual IOCTL definition here does not change.
But, it makes the code less confusing for the reader.
Signed-off-by: Hans van Kranenburg <hans.van.kranenburg@mendix.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Wang Xiaoguang [Fri, 1 Jul 2016 05:26:25 +0000 (13:26 +0800)]
btrfs-progs: mkfs: fix allocation information output of block group types
When cleanup_temp_chunks() removes block groups, it forgot to update
mkfs_allocation accordingly, fix this.
Signed-off-by: Wang Xiaoguang <wangxg.fnst@cn.fujitsu.com>
[ minor adjustments ]
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Mon, 4 Jul 2016 11:45:15 +0000 (13:45 +0200)]
btrfs-progs: tests: use /bin/bash for scripts
Since we use 'source' in scripts, let's use bash everywhere.
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Mon, 27 Jun 2016 07:50:11 +0000 (15:50 +0800)]
btrfs-progs: convert-test: Add test case for discontinuous hole extent
For ext* fs containing a large hole(larger than 128M), btrfs-convert
will only insert one 128M hole extent and skip the remaining.
This leads to discontinuous file extents.
Add test case for it, and since it's a pinpoint regression test case, no
combination of convert options nor checksum verification.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Mon, 27 Jun 2016 07:50:10 +0000 (15:50 +0800)]
btrfs-progs: convert: Fix a bug leads to discontinuous extents
Btrfs_record_file_extent() will split extents using max extent size(128M).
It works well for real file extents, but not that well for large
hole extent, as hole doesn't have extent size limit.
In that case, it will only insert one 128M hole, and skip the rest,
leading to discontinuous extent error for converted btrfs.
Fix it by not splitting hole extents.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Luis Henriques [Fri, 24 Jun 2016 23:47:29 +0000 (00:47 +0100)]
btrfs-progs: tests: 001-simple-unmounted: fix test failure due to bashism
The usage of 'source' is a bashism, and '.' should be used instead. This
is causing fuzz-tests/001-simple-unmounted to fail in systems where
/bin/sh isn't bash:
[TEST/fuzz] 001-simple-unmounted
./test.sh: 5: ./test.sh: source: not found
./test.sh: 7: ./test.sh: setup_root_helper: not found
./test.sh: 8: ./test.sh: check_prereq: not found
./test.sh: 18: ./test.sh: check_all_images: not found
Since most (all?) tests actually use /bin/bash, change this test to use
bash too.
Signed-off-by: Luis Henriques <henrix@camandro.org>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 24 Jun 2016 12:10:31 +0000 (14:10 +0200)]
Btrfs progs v4.6.1
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 8 Jun 2016 09:40:02 +0000 (11:40 +0200)]
btrfs-progs: update CHANGES for 4.6.1
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 24 Jun 2016 12:04:04 +0000 (14:04 +0200)]
btrfs-progs: build: check if FIEMAP_EXTENT_SHARED is defined
Detect the macro at configure time rather than during compilation.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 24 Jun 2016 12:03:36 +0000 (14:03 +0200)]
btrfs-progs: build: add m4 macros for AC_CHECK_DEFINE
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 24 Jun 2016 11:54:45 +0000 (13:54 +0200)]
btrfs-progs: build: let autotools include m4
We'll need some non-standard macro definitions.
Signed-off-by: David Sterba <dsterba@suse.com>
Satoru Takeuchi [Fri, 24 Jun 2016 08:24:47 +0000 (17:24 +0900)]
btrfs-progs: fi show: print error message if no valid Btrfs is specified
* Before this patch
===============================
# ./btrfs fi show foo # "foo" doesn't mean any valid Btrfs
# # no error message
# echo $?
1
===============================
* After this patch
===============================
# ./btrfs fi show foo
ERROR: foo is not a valid Btrfs
#
# echo $?
1
===============================
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 22 Jun 2016 13:52:17 +0000 (15:52 +0200)]
btrfs-progs: tests: fix filesytem type creation for convert tests
The extN filesystem type was lost when the separate tests were created
and we've been testing only ext2. The tests pass for ext3 and ext4
though.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 22 Jun 2016 13:50:17 +0000 (15:50 +0200)]
btrfs-progs: tests: add 005-delete-all-rollback
Test if a rollback works after deleing all files from btrfs.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 22 Jun 2016 12:28:23 +0000 (14:28 +0200)]
btrfs-progs: tests: update README
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 22 Jun 2016 12:07:46 +0000 (14:07 +0200)]
btrfs-progs: tests: add test console
Add a wrapper that sets up environment the same way a test would use it.
Use it for quick prototyping or testing, the commands and output is
logged.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 22 Jun 2016 09:30:24 +0000 (11:30 +0200)]
btrfs-progs: tests: print shorter test name in the output
The full path is printed, we're interested in the last path component
only.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 22 Jun 2016 09:25:33 +0000 (11:25 +0200)]
btrfs-progs: tests: split convert_test
Split the big function to several helpers so we can use them separately.
Add comments and do minor tweaks.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Tue, 21 Jun 2016 00:48:39 +0000 (02:48 +0200)]
btrfs-progs: docs: update 'btrfs-device' manual page
Signed-off-by: David Sterba <dsterba@suse.com>
Mark Fasheh [Wed, 15 Jun 2016 22:50:01 +0000 (15:50 -0700)]
btrfs-progs: free qgroup counts in btrfsck
Signed-off-by: Mark Fasheh <mfasheh@suse.de>
Signed-off-by: David Sterba <dsterba@suse.com>
Chandan Rajendra [Fri, 17 Jun 2016 05:37:54 +0000 (11:07 +0530)]
Btrfs-progs: Initialize stripesize to the value of sectorsize
stripesize should ideally be set to the value of sectorsize. However
previous versions of btrfs-progs/mkfs.btrfs had set stripesize to a
value of 4096. On machines with PAGE_SIZE other than 4096, This could
lead to the following scenario,
- /dev/loop0, /dev/loop1 and /dev/loop2 are mounted as a single
filesystem. The filesystem was created by an older version of mkfs.btrfs
which set stripesize to 4k.
- losetup -a
/dev/loop0: [0030]:19477 (/root/disk-imgs/file-0.img)
/dev/loop1: [0030]:16577 (/root/disk-imgs/file-1.img)
/dev/loop2: [64770]:
3423229 (/root/disk-imgs/file-2.img)
- /etc/mtab lists only /dev/loop0
- losetup /dev/loop4 /root/disk-imgs/file-1.img
The new mkfs.btrfs invoked as 'mkfs.btrfs -f /dev/loop4' succeeds even
though /dev/loop1 has already been mounted and has
/root/disk-imgs/file-1.img as its backing file.
The above behaviour occurs because check_super() function returns an
error code (due to stripesize not being set to 4096) and hence
check_mounted_where() function treats /dev/loop1 as a disk containing a
filesystem other than Btrfs.
Hence as a workaround this commit allows 4096 as a valid stripesize.
Signed-off-by: Chandan Rajendra <chandan@linux.vnet.ibm.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Thu, 16 Jun 2016 01:15:51 +0000 (09:15 +0800)]
btrfs-progs: qgroup: Fix a bug that fails to skip rescan running case
Commit
6bdf962fe35a8648d(btrfs-progs: Read qgroup status for qgroup
verify) will read qgroup status, and then use it to skip qgroup
reporting.
However since the rescan_running/inconsistent flags are only 1 bit long,
while qgroup flags & BTRFS_QGROUP_FLAGS returns value longer than 1bit,
it doesn't work.
Fix it by doing double negation on (flags & BTRFS_QGROUP_FLAGS) to
convert it to either 1 or 0.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Thu, 16 Jun 2016 16:48:44 +0000 (18:48 +0200)]
btrfs-progs: docs: update btrfs-restore manual page
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Thu, 16 Jun 2016 11:50:18 +0000 (13:50 +0200)]
btrfs-progs: tests: add 003-fi-resize-args
Check various resize option combinations.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Thu, 16 Jun 2016 11:27:26 +0000 (13:27 +0200)]
btrfs-progs: fi resize: fix parsing of negative argument
The cleanup of option parsing caused a regression where the negative
resize argument is recognized as short options and the command fails.
Use the new helper to allow that.
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=119971
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Thu, 16 Jun 2016 11:27:18 +0000 (13:27 +0200)]
btrfs-progs: new helper for option parsing, more permissive for "no options"
Signed-off-by: David Sterba <dsterba@suse.com>
Satoru Takeuchi [Tue, 14 Jun 2016 05:50:19 +0000 (14:50 +0900)]
btrfs-progs: doc: correct the destination of btrfs-receive
We can set not only btrfs mount point but also any path belong to
btrfs mount point as btrfs-receive's destination.
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Noah Massey [Fri, 10 Jun 2016 13:57:55 +0000 (09:57 -0400)]
btrfs-progs: doc: add missing newline in btrfs-convert
Signed-off-by: Noah Massey <noah.massey@gmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 10 Jun 2016 13:03:18 +0000 (15:03 +0200)]
Btrfs progs v4.6
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 8 Jun 2016 09:40:02 +0000 (11:40 +0200)]
btrfs-progs: update CHANGES for 4.6
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Thu, 2 Jun 2016 07:22:49 +0000 (15:22 +0800)]
btrfs-progs: convert: Fix a bug which fails to insert hole file extent
When copying inode, if there is a file referring part of a hole range,
convert will fail.
The problem is, when calculating real extent bytenr, it doesn't check if
the original extent is a hole.
In case the orinal extent is a hole, we still calculate bytenr using
file_pos - found_extent_file_pos, causing non-zero value, and later
btrfs_record_file_extent() detects that we are pointing to non-exist
extent and aborts convert.
Fix it by checking the disk_bytenr before calculating real disk bytenr.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Wed, 1 Jun 2016 08:29:43 +0000 (16:29 +0800)]
btrfs-progs: convert: Insert needed holes for superblock migration
New convert doesn't insert holes for superblock migration range.
Unlike the old design, which only relocates 4K (superblock size) to
other places.
In the new design, to make sure convert can handle different page sizes
and align chunks bytenr, we relocate the whole 64K range.
And if there is only a 4K used block inside 64K superblock migration
range, it will make converted the fs have discontiguous file extents.
This patch will fix it by inserting needed holes to avoid such
discontinuous error.
Reported-by: Tsutomu Itoh <t-itoh@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Tue, 31 May 2016 08:49:33 +0000 (16:49 +0800)]
btrfs-progs: convert: Fix bugs in backup superblock migration
New convert has several bugs with backup superblock migration
1) Backup superblocks are not migrated due to incorrect condition
Two wrong checks cause backup superblocks not to be migrated at all
2) Converted ext* image doesn't keep hole for backup superblocks
Since we are creating file extents according to tmp_used, which has
wiped out backup superblock ranges.
In that case, later superblock migration will fail, since migration
will insert file extent range into ext* image.
Fix above bugs will make convert on ext2 image filled about 100M data
successful.
Reported-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Tue, 31 May 2016 07:51:13 +0000 (15:51 +0800)]
btrfs-progs: convert: Fix a bug that makes convert asserts at scan time
When a ext2 fs filled with a 57M file, it's possible that convert fails
with assert in add_merge_cache_extent().
The problem is that the ext2 used space just takes some of the second
superblock.
And due to a bug in reserving superblock space, it corrupted used space
tree and cause assert.
Fix in by doing better used space merging for case where superblock
range is inside the ext2 used space.
Reported-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Tue, 31 May 2016 02:46:00 +0000 (10:46 +0800)]
btrfs-progs: convert: Add support to rollback new convert image
For new btrfs-convert, it's less restrict for metadata chunk allocation.
While the may_rollback() function is still following the restrict 1:1
mapping check for all chunks, it will not allow some new convert image
to be rolled back.
Add new per extent check for new convert to allow it to be rolled back.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:32 +0000 (13:03 +0800)]
btrfs-progs: Cleanup old btrfs-convert
Cleanup all the old btrfs-convert facilities, including:
1) btrfs_convert_operations->alloc/free/test_extents*
No need to do non-standard extent allocation.
After init_btrfs() everything can be done by normal routine.
Now only 4 functions are needed in btrfs_convert_operations.
1) open_fs
2) read_used_space
3) copy_inodes
4) close_fs
2) fs_info->extent_ops
Same as above.
3) Old init_btrfs(), create_image(), create_file_image_range()
Replaced with newer and cleaner one.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:31 +0000 (13:03 +0800)]
btrfs-progs: convert: Strictly avoid meta or system chunk allocation
Before this patch, btrfs-convert only rely on large enough initial
system/metadata chunk size to ensure no newer system/meta chunk will be
created.
But that's not safe enough. So add two new members in fs_info,
avoid_sys/meta_chunk_alloc flags to prevent any newer system or meta
chunks to be created before init_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:30 +0000 (13:03 +0800)]
btrfs-progs: Convert: Add support for rollback new convert behavior
Add support to rollback new btrfs-convert.
The support is quite easy unlike the new convert behavior, which in fact
makes the rollback less restricted.
The core is, rollback must support both old and new convert result.
For old convert, all fs image extents can be covered by any chunk type.
But any extents inside reserved range must be covered by chunk.
For new convert, all fs image extents are covered by data chunk.
Allowing any condition will just make another fail to pass.
So make the convert condition a little less restricted to allow both can
be converted.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:29 +0000 (13:03 +0800)]
btrfs-progs: Introduce do_convert_v2 function
Introduce new function do_convert_v2() to do new convert.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ use __strncpy_null for label ]
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:28 +0000 (13:03 +0800)]
btrfs-progs: convert: Introduce init_btrfs_v2 function.
Introduce new init_btrfs_v2() function for later newer do_convert().
Since we have good enough chunk allocation, a lot of wired chunk hack
won't ever be used.
We only need to insert data chunks and create needed subvolume.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:27 +0000 (13:03 +0800)]
btrfs-progs: convert: Enhance record_file_blocks to handle reserved ranges
Enhance record_file_blocks() to handle reserved ranges.
Old file system can use the space in btrfs reserved ranges.
So we could not use the bytenr of old filesystem directly.
Thanks to previous patches, we have a full fs image in convert_root, and
it has already relocated the blocks in reserved ranges.
So here we just search the convert_root to get correct disk_bytenr and
use it.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:26 +0000 (13:03 +0800)]
btrfs-progs: convert: Introduce function to migrate reserved ranges
Introduce new function, migrate_reserved_ranges() to migrate used fs
data in btrfs reserved space.
Unlike old implementation, which will need to relocate all the
complicated csum and reference relocation, previous patches already
ensure such reserved ranges won't be allocated. So here we only need
copy these data out and create new extent/csum/reference.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:25 +0000 (13:03 +0800)]
btrfs-progs: convert: Introduce new function to create converted image
Use new function, create_convert_image_v2() to create snapshot of old
filesystem.
Unlike old function which is called after copying all inodes, this
function need to be called before copying inodes.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:24 +0000 (13:03 +0800)]
btrfs-progs: extent-tree: Enhance btrfs_record_file_extent
Btrfs_record_file_extent() has some small problems like:
1) Can't handle overlapping extents
2) May create extent larger than BTRFS_MAX_EXTENT_SIZE
So enhance it using previously added facilites.
This is used for later btrfs-convert, as for new convert, we create
saved image first, then copy inode.
Which will also cause extent overlapping.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:23 +0000 (13:03 +0800)]
btrfs-progs: extent-tree: Introduce function to find the first overlapping extent
Introduce a new function, btrfs_search_overlap_extent() to find the first
overlapping extent.
It's useful for later btrfs-convert rework.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:22 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to create convert data chunks
Introduce new function, make_convert_data_chunks(), to build up data
chunks for convert.
It will call a modified version of btrfs_alloc_data_chunk() to force
data chunks to covert all known ext* data.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:21 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to setup temporary extent tree
Introduce new function, setup_temporary_extent_tree() to build a
temporary extent tree for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:20 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to initialize csum tree
Introduce new function, setup_temp_csum_tree(), to setup temporary
csum tree for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:19 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to initialize fs tree
Introudce new function, setup_temp_fs_tree(), to setup temporary fs tree
for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:18 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to initialize device tree
Introduce new function, setup_temp_device_tree(), to setup temporary
device tree for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:17 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to setup temporary chunk root
Introduce new function, setup_temp_chunk_root(), to initialize temporary
chunk root for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:16 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to setup temporary tree root
Introduce new function, setup_temp_tree_root(), to initialize temporary
tree root for make_btrfs_v2().
The new function will setup tree root at metadata chunk and ensure data
won't be written into metadata chunk.
Also, new make_btrfs_v2() will have a much better code structure than
old make_btrfs().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:15 +0000 (13:03 +0800)]
btrfs-progs: Introduce function to setup temporary superblock
Introduce a new function, setup_temp_super(), to setup temporary super
for make_btrfs_v2().
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ use __strncpy_null in setup_temp_super ]
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:14 +0000 (13:03 +0800)]
btrfs-progs: utils: Introduce new function for convert
Introduce new function make_convert_btrfs() for convert.
This new function will have the following features:
1) Allocate temporary sb/metadata/system chunk, avoiding old used data
2) More structured functions
No more over 1000 lines function, better function split and code
reuse
This will finally replace current make_btrfs(), but now only used for
convert.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:13 +0000 (13:03 +0800)]
btrfs-progs: convert: Introduce function to calculate the available space
Introduce a new function, calculate_available_space() to get available
space cache_tree data_chunks cache_tree.
Unlike old implementation, this function will do the new work:
1) batch used ext* data space.
To ensure data chunks will recovery them all.
And restore the result into mkfs_cfg->convert_data_chunks for later
use.
2) avoid SB and reserved space at chunk level
Both batched data space or free space will not cover reserved space,
like SB or the first 1M.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:12 +0000 (13:03 +0800)]
btrfs-progs: convert: Introduce new function to remove reserved ranges
Introduce functions to remove reserved ranges for later btrfs-convert
rework.
The reserved ranges includes:
1. [0,1M)
2. [btrfs_sb_offset(1), +BTRFS_STRIPE_LEN)
3. [btrfs_sb_offset(2), +BTRFS_STRIPE_LEN)
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 29 Jan 2016 05:03:11 +0000 (13:03 +0800)]
btrfs-progs: convert: Introduce functions to read used space
Before we do real convert, we need to read and build up used space cache
tree for later data/meta separate chunk layout.
This patch will iterate all used blocks in ext2 filesystem and record it
into cctx->used cache tree, for later use.
This provides the very basic of later btrfs-convert rework.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Mon, 9 May 2016 04:46:46 +0000 (12:46 +0800)]
btrfs-progs: convert: add compatibility layer for e2fsprogs < 1.42
The new convert framework copies code from current dumpe2fs, which uses
BIGALLOC feature introduced in e2fsprogs v1.42.
While there are a lot of enterprise distributions which are still using
v1.41 e2fsprogs, this will cause compile error for them.
This patch introduces backward compatibility for new convert framework,
by manually introduce macros for ext2 BIGALLOC feature.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 15:08:43 +0000 (17:08 +0200)]
btrfs-progs: tests: 004-ext2-backup-superblock-ranges, use common helpers for image loop
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 14:52:57 +0000 (16:52 +0200)]
btrfs-progs: tests: 004-ext2-backup-superblock-ranges, drop unnecessary root privs
We really use root only for mount/umount and access to the ext2_saved
image (that has 0600). Also switch to common variable so we can use
helpers.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 14:01:51 +0000 (16:01 +0200)]
btrfs-progs: tests: unify test drivers
Remove unnecessary code, add exports to all common variables.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 13:54:32 +0000 (15:54 +0200)]
btrfs-progs: tests: convert: set common variables
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 13:46:39 +0000 (15:46 +0200)]
btrfs-progs: tests: clean up the test driver of convert tests
Everything is now in separate tests, and TEST=mask now works.
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 13:37:25 +0000 (15:37 +0200)]
btrfs-progs: tests: convert: separate ext4 tests
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 13:37:25 +0000 (15:37 +0200)]
btrfs-progs: tests: convert: separate ext3 tests
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 13:37:25 +0000 (15:37 +0200)]
btrfs-progs: tests: convert: separate ext2 tests
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 3 Jun 2016 13:28:44 +0000 (15:28 +0200)]
btrfs-progs: tests: move convert helpers to a separate file
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 3 Jun 2016 02:34:26 +0000 (10:34 +0800)]
btrfs-progs: convert-tests: Add test case for backup superblock migration
New convert introduced simpler chunk/extent allocation algorithm, at the
cost of complex backup superblock migration codes.
Use specially built ext2 images to test if btrfs-convert can convert and
rollback images without problem.
All these special ext2 image have blocks/holes across 2nd btrfs backup
superblock.
The naming of test image is like the following:
|<------superblock migration range----->|
64M 64M + 64K
|-Data--|-Data--|/Hole//|-Data--|/Hole//|-Data--|--Data--| = drdhdhdrd
These test cases should check all typical layouts and make sure new
convert works.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Fri, 3 Jun 2016 02:34:25 +0000 (10:34 +0800)]
btrfs-progs: convert-tests: Add support for custom test scripts
Add support for custom convert test scripts, just like fsck tests.
Instead of generic convert tests, we need more specifically created images
for new convert tests.
This patch provide the needed infrastructure for later convert test
cases.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Satoru Takeuchi [Thu, 2 Jun 2016 08:14:57 +0000 (17:14 +0900)]
btrfs-progs: btrfs-crc: make argc check more strict
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Satoru Takeuchi [Thu, 2 Jun 2016 08:13:10 +0000 (17:13 +0900)]
btrfs-progs: btrfs-crc: improve usage message
- If -c is set, filename argument is ignored.
- Describe about -h option
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Satoru Takeuchi [Thu, 2 Jun 2016 08:11:51 +0000 (17:11 +0900)]
btrfs-progs: btrfs-crc: print usage on receiving invalid arguments
Usage is only printed if -h option is set. However it's nice to
do it when wrong option is set or the number of argument is wrong.
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Satoru Takeuchi [Thu, 2 Jun 2016 08:10:18 +0000 (17:10 +0900)]
btrfs-progs: btrfs-crc should be ignored by git
It's a binary built from btrfs-crc.c
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Satoru Takeuchi [Thu, 2 Jun 2016 08:06:37 +0000 (17:06 +0900)]
btrfs-progs: btrfs-crc: fix build error
Remove the following build error.
====================================
$ make btrfs-crc
[CC] btrfs-crc.o
[LD] btrfs-crc
btrfs-crc.o: In function `usage':
/home/sat/src/btrfs-progs/btrfs-crc.c:26: multiple definition of `usage'
help.o:/home/sat/src/btrfs-progs/help.c:125: first defined here
collect2: error: ld returned 1 exit status
Makefile:294: recipe for target 'btrfs-crc' failed
make: *** [btrfs-crc] Error 1
=====================================
Signed-off-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Thu, 2 Jun 2016 16:18:41 +0000 (18:18 +0200)]
btrfs-progs: use wider int type in btrfs_min_global_blk_rsv_size
We know nodesize should not overflow with the shift, but le'ts make the
code correct if the resulting type can store the full value.
Resolves-coverity-id:
1358120
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Thu, 26 May 2016 02:56:51 +0000 (10:56 +0800)]
btrfs-progs: Use new random number API
Replace old and not so informal srand()/rand() calls to new random
number API.
Including btrfs-corrupt-block(main user), btrfs-image and btrfs-crc.
Some tests like dir-test/random-test/quick-test is not modified.
As random-test itself can't even pass build.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Thu, 26 May 2016 02:56:50 +0000 (10:56 +0800)]
btrfs-progs: utils: Introduce new pseudo random API
David has reported some quite chaos usage of pseudo random numbers.
Like using static srand seed, or even calling rand() without setting
seed correctly.
The new pseudo random API will initialize the random seed on its first
calling and use uniformly distributed pseudo random number generator as
backend.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
[ renamed variables and functions, added prefixes ]
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Thu, 26 May 2016 02:36:02 +0000 (10:36 +0800)]
btrfs-progs: corrupt-block: Add support to corrupt extent for skinny metadata
For skinny metadata case, btrfs-corrupt-block can't corrupt a metadata
extent due to the type check missing BTRFS_METADATA_ITEM_KEY.
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Qu Wenruo [Wed, 1 Jun 2016 01:51:21 +0000 (09:51 +0800)]
btrfs-progs: convert-tests: Add test for backup superblock migration
New convert framework uses new and simpler chunk layout, while the cost
is the more complex superblock range migration logical, compared to old
convert.
Enhance the convert test script to create file which will takes up 2nd
backup superblock space, to ensure the superblock migration is working
as expected.
Suggested-by: David Sterba <dsterba@suse.cz>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Wed, 1 Jun 2016 11:22:44 +0000 (13:22 +0200)]
btrfs-progs: tests: document cli-tests in readme
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Tue, 31 May 2016 17:17:02 +0000 (19:17 +0200)]
btrfs-progs: tests: convert: dump all superblocks after conversion
We want to see all of them, even if they're not valid.
Signed-off-by: David Sterba <dsterba@suse.com>
Lu Fengqi [Thu, 26 May 2016 09:43:00 +0000 (17:43 +0800)]
btrfs-progs: make btrfs-image restore to support dup
Previously btrfs-image restore would set the chunk items to have 1 stripe,
even if the chunk is dup. If you use btrfsck on the restored file system,
some dev_extent will not find any relative chunk stripe, and the
bytes-used of dev_item will not equal to the dev_extents's total_bytes.
This patch store a additional physical just for the dup case when build
the in-memory chunk-tree.
Currently btrfsck on the restored file system, only single and dup is no
problem. raid* support should be added in the future.
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Mon, 30 May 2016 15:11:19 +0000 (17:11 +0200)]
btrfs-progs: tests: run rollback after conversion
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Mon, 30 May 2016 15:07:35 +0000 (17:07 +0200)]
btrfs-progs: tests: convert, run md5sum with sudo helper
Some of the files might not end up in the checksum list because of
permissions. This is reported by md5sum as incorrectly formatted lines.
Signed-off-by: David Sterba <dsterba@suse.com>
Lu Fengqi [Mon, 30 May 2016 02:58:13 +0000 (10:58 +0800)]
btrfs-progs: tests: add 020-extent-ref-cases
In order to confirm that btrfsck supports to check a variety of
refs, add the
following cases:
* keyed_block_ref
* keyed_data_ref
* shared_block_ref
* shared_data_ref
* no_inline_ref (a extent item without inline ref)
* no_skinny_ref
Signed-off-by: Lu Fengqi <lufq.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.com>
Merlin Hartley [Thu, 19 May 2016 14:17:24 +0000 (15:17 +0100)]
btrfs-progs: doc: fix typo in btrfs-subvolume
Signed-off-by: Merlin Hartley <merlinhartley@hotmail.com>
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 20 May 2016 11:39:02 +0000 (13:39 +0200)]
btrfs-progs: switch to common message helpers in utils.c
Signed-off-by: David Sterba <dsterba@suse.com>
David Sterba [Fri, 20 May 2016 11:20:28 +0000 (13:20 +0200)]
btrfs-progs: fix type mismatch in backtrace dumping functions
Signed-off-by: David Sterba <dsterba@suse.com>