Artem Bityutskiy [Wed, 28 Nov 2012 16:02:21 +0000 (18:02 +0200)]
Add RELEASE_NOTES
Not wired to the packaging so far - it needs some more work.
Change-Id: I82eae2e79acd55472f099a9178678f0aec4e452d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 15:38:33 +0000 (17:38 +0200)]
Add a small README file
Change-Id: I83285512d949bce6254d87091adc275e98d861c3
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 13:01:43 +0000 (15:01 +0200)]
test_fiemap: implement a test for part of the file
Test no only the entire file, but also parts of the file.
Change-Id: I6a64199c371d8c343384d508856ea99a496a6c4a
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 13:27:44 +0000 (15:27 +0200)]
Fiemap: fix get_(un)mapped_ranges for file parts
The functions worked incorrectly when the ranges were requested for part of the
file. Fix this.
Change-Id: Ib979c06a05dcbad1825b98199d23441d236fe5e4
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 12:28:57 +0000 (14:28 +0200)]
test_fiemap: code re-structuring
This patch re-structures the _do_test() function of the 'test_fiemap' unit
test. The re-structuring is needed because I am going to extend the test and
verify the 'get_(un)mapped_ranges()' finctions fro parts of the file, not only
for the entire file. Without the re-structuring there will be a lot of code
duplication.
So, this patch introduces a '_check_ranges()' helper function which verifies
the 'get_(un)mapped_ranges()' function for a given range of blocks.
Additinally, to make the code look uniform, this patch renames all the 'holes'
list variables into 'unmapped', to match the existing 'mapped' list.
Change-Id: Ie1a30941eea9004e742621e87bf325a7f464ab27
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 09:36:54 +0000 (11:36 +0200)]
test_fiemap: test with different buffer sizes
Test the 'Fiemap' class with several different buffer sizes.
Change-Id: I250f576c4804c02e3691197d8542430571fcb6d4
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 09:35:24 +0000 (11:35 +0200)]
Fiemap: amend things about the default buffer size
Introduce a constant for the default buffer size.
Change the constructor interface - now None 'buf_size' means the default
buffer size, which is a nicer interface.
Change-Id: I1bc7dca57e9d36e4a9d6e8cc402b3f5038bcf460
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 09:26:39 +0000 (11:26 +0200)]
test_fiemap: increase maximum file size to 16MiB
The default 4MiB are probably too few because the internal ext4 extent size
seems to be 2MiB, so 4MiB are just 2 extents.
Change-Id: If40f0ad70c6787659a01ebc6473ed4189069d81e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 07:02:22 +0000 (09:02 +0200)]
Fiemap.py: implement new get_mapped_ranges
Use full power of the FIEMAP ioctl and call it for large areas of the file,
instead of doing it block-after-block. This version is several times faster
than the old version.
The 'get_unmapped_ranges()' generator is build on top of 'get_mapped_ranges()'.
Change-Id: I8bbaf7ccb755904a2bcaef71ba53d82b4fa2c60b
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 09:07:42 +0000 (11:07 +0200)]
tests: helpers: mark create_random_sparse_file as local
Mark the 'create_random_sparse_file()' function as local by addin a leading
'_'.
Change-Id: Id17557c0c8b4268555d71610c1a12def8e8b64bc
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 09:05:54 +0000 (11:05 +0200)]
test_api_base: correct tar.bz2 files extension
This is just a copy-paste error - we gave .tar.bz2 files a '.tar.gz' extension.
Change-Id: Ic4a0fbb40c4519bb12c91c42b3e546b1952a82db
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 08:57:46 +0000 (10:57 +0200)]
tests: helpers: do not always write entire blocks
In 'create_random_sparse_file()', when we have decided to map a block, we
always fill it entirely (write 4096 bytes). However, filling only part of this
block from a random offset withing a block is more realistic.
Change-Id: I825e9fb093e7c9c7be309b2563c2be6049783811
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 08:37:02 +0000 (10:37 +0200)]
tests: helpers: a small nicification
getrandbits(1) is a bit more clever than randint(0, 1). Also, remove useless
bool().
Change-Id: I169c21f8e986dd6579a46d6eb0b214ec7e11611a
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 08:30:07 +0000 (10:30 +0200)]
tests: helpers: use less random data
We do not need really random data in the files we generate - this only slows
the tests down a lot. Let's fill the files with semi-random data instead - just
pick a random byte and fill large regions with this byt. This also make
compression work faster, and it is good enough for our purposes.
Besides, this allows to reproduce tests by setting a known seed, which does not
work with os.urandom.
Change-Id: I0459ee44d271b8e19581e84cc33b7167e4ada166
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 06:59:53 +0000 (08:59 +0200)]
Fiemap: fix internal buffer size calculations
... I used 'buf_size' instead of 'self._buf_size' by a mistake.
Change-Id: If92b92e4aa09653fe27e41e5a2cc236fce3042af
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 06:58:31 +0000 (08:58 +0200)]
Fiemap: lower the default buffer size
The default 1MiB buffer is a bit too large, make it 256KiB.
Change-Id: I7827889d6201f224033c77b5396901f65c766e8d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 28 Nov 2012 06:56:58 +0000 (08:56 +0200)]
Fiemap: introduce a constant for minimum buffer size
Change-Id: Ic029ac27afa9c3994b9a59594ca52c280c3b24b9
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 14:05:36 +0000 (16:05 +0200)]
Suppress few pylint recommendations
Change-Id: Ia757396ffe5d39b3ff5d35b9639417c662f74706
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 13:58:34 +0000 (15:58 +0200)]
tests: helpers: generate fully-mapped files as well
Improve tests coverage by generating fully-mapped files as well.
Change-Id: Icf3a8a246efadd32a0dcbc28ccc279caaaced143
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 13:11:21 +0000 (15:11 +0200)]
test_fiemap: add Fiemap module unit test
Change-Id: I0da8bb80f8b15fd27695381324bb213726cfeca2
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 13:44:05 +0000 (15:44 +0200)]
tests: helpers: fix holes area for a 4097 bytes file
... should be (0, 1), not (0, 0).
Change-Id: Ia611057680623360191973aa2691c9d35161bcef
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 13:19:33 +0000 (15:19 +0200)]
tests: helpers: teach generate_test_files to return mapped areas
This is a preparation to the Fiemap test. In the Fiemap module we have 2
functions: get_mapped_ranges() and get_unmapped_ranges(). And to test both of
them it is convenient to have 2 lists from the
'tests.helpers.generate_test_files()' function: mapped list and unmapped list.
Thus, teach 'tests.helpers.generate_test_files()' to also generate the list of
mapped areas, not only the holes.
Change-Id: Id000ba0b278ccceb821f540755cdd94165eeecc9
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 12:38:32 +0000 (14:38 +0200)]
test_api_base: test .tar.gz and .tar.bz2 compression
This actually revealed a bug...
Change-Id: Ife39862d4b9e71beea5b5150970da9066782145b
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 12:39:40 +0000 (14:39 +0200)]
BmapCopy: fix tar.gz compression support
This is a nasty bug - I used 'if' instead of 'elif' ... Thanks to
'test_api_base' for revealing it.
Change-Id: I43e1954cb02c46ac27e9d9e7324e86ef9dec2974
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 10:06:29 +0000 (12:06 +0200)]
tests: add a possibility to avoid tmp files deletion
This is needed for debugging purposes. When there are issues, it is very handy
to be able to leave the temporary files and then investigate them. Do lets add
a simple way to do this. In the future, this may become a test parameter.
Change-Id: I067fc9d5a522057290c5c9ee190cf15ee7fa19f1
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 09:14:03 +0000 (11:14 +0200)]
tests: introduce a possibility to change the directory for tmp files
The default temporary directory is choosen by 'NamedTemporaryFile' and it is
usually '/tmp'. However, for debugging purposes it is nice to sometimes change
that to a different directory. In the future, this may be a test parameter.
Change-Id: I12428f99a14eb7467948e08048bd0108c8bf5bc1
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 08:52:54 +0000 (10:52 +0200)]
tests: assign better names to temporary files
When creating temporary files, add prefixes and suffixes which make it easy to
understand what is the file. This makes debugging a lot easier, although the
code becomes a bit more complex.
Change-Id: Ieba77821c3d97ed080fbc22b889d77cb640312c6
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 07:21:54 +0000 (09:21 +0200)]
TODO: add an entry about compression types detection
Change-Id: I4a12477bd07a7b6d25be662929a0053cda170cb7
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 07:16:18 +0000 (09:16 +0200)]
tests: helpers: move compress_test_file to test_api_base.py
The 'compress_test_file()' function is only used by 'test_api_base' and I am
not planning any other tests which would need it. So let's move it to that
file.
While on this, also rename the function to '_generate_compressed_files()',
which is more consistent with the other iterator we have:
'generate_test_files()'.
Change-Id: I9191b413a82545facf9507d4c72ee070e0487be4
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 27 Nov 2012 07:11:15 +0000 (09:11 +0200)]
test_api_base: make compare_holes local
The 'compare_holes()' function is only used in this test, so it is cleaner to
add a leading '_' to the name to show that it is a local function.
Change-Id: I8f2a446fefbc314640830e734db088e8ce39e05a
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 15:58:48 +0000 (17:58 +0200)]
test_api_base: also test compressed files
Change-Id: Iae27bcea6fac825a2c6d1f82b125afda5db31859
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 14:34:59 +0000 (16:34 +0200)]
tests: helpers: create a compression iterator
Create an iterator which compresses a file to different format. Not used
so far.
Change-Id: I23a707667a918c2e07f5c85f32f86936454c997b
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 15:34:15 +0000 (17:34 +0200)]
BmapCopy: fix image size validation
... which did not work correctly because it should only be run for uncompressed
images in __init__, as well as for any images when there is no bmap.
Change-Id: I3e1dad253889df7fa8980c30df6bcb9dd4b5652d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 15:12:48 +0000 (17:12 +0200)]
test_api_base: compare files using sha1
Instead of comparing files all the time using 'filecmp', calculate
sha1 of the image once and compare it to the copies.
This will be needed soon in order to implement compressed files testing - the
'filecmp' module does not accept file-like objects, only file names, so you
cannot feed it a compressed image.
Change-Id: I0c4db36fb89f9c5b8ba5aac116e9f25d9a17b4f6
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 14:57:45 +0000 (16:57 +0200)]
test_api_base: move _do_test() out of the test class
It really does not need to be there. Let the class be just a minimum
for nosetests to be happy.
Change-Id: I8181f7f9ffa02d9e1277154bdd9a0d5fc8af84a3
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 14:17:34 +0000 (16:17 +0200)]
tests: helpers: improve generate_test_files
Make 'generate_test_files()' generate more files - files consisting of a single
hole with different sizes, and randomly generated sparse files of different
sizes.
Change-Id: I16f3f84d778f2653f72cbd5d34ce57deff2cdd2f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 13:44:32 +0000 (15:44 +0200)]
tests: move images generation to helpers.py
Introduce a common iterator in helpers.py which generates test images.
The iterator will be used in the fiemap test as well. I will also add more
interesting files to the iterator soon.
Change-Id: I38b957389d39fa99e11cd44699d636028bf12109
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 13:34:23 +0000 (15:34 +0200)]
tests: rename test_helpers.py into helpers.py
... in order to prevent nostests to try to run functions from this file.
Change-Id: I5517a6e82b36d75ce33a25aeb03406ec05416351
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 11:41:39 +0000 (13:41 +0200)]
test_api_base: run the test for more than one file
Instead of running the test of one file of 64MiB, run it 3 times for files of
the following sizes:
1. 8MiB
2. 8MiB + 1 byte
3. 8MiB - 1 byte
This gives better coverage.
Change-Id: I4ea4f5f92dc32c1d8fbf9ea0173b7c39a294968d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 11:39:05 +0000 (13:39 +0200)]
test_api_base: re-structure the test some more
Kill the 'setUp()' and 'tearDown()' methods and open/close the temporary files
in '_do_test()' instead. This works better for me because I will call
'_do_test()' many times and it is simpler when it starts with new temporary
files every time, rather than teach it cleaning-up the old ones.
Change-Id: I135b8423d1e7d95a6c2de8d8bc0d3f8597ca5418
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 11:26:06 +0000 (13:26 +0200)]
BmapCopy: validate image size
Add some validation of image size - when we have the bmap, bmake sure that real
image size is the same as image size from the bmap.
Change-Id: I8503e0a14d7eb1df071780058b71b7ef6be225a4
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 11:01:47 +0000 (13:01 +0200)]
BmapCopy: bugfix: handle unaligned files properly
When the image size is not aligned to the block size, we made an aligned copy
anyway, which is wrong, because the copy has to have the same length as the
image. This patch fixes the issue.
Change-Id: I012496e315b47c1381715fdd1f8921d347213b29
Artem Bityutskiy [Mon, 26 Nov 2012 11:02:59 +0000 (13:02 +0200)]
TODO: add a reminder about testing compressed files.
Change-Id: I56506ee85f11c2fe03828b06f538313533971759
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 10:51:24 +0000 (12:51 +0200)]
BmapCopy: simplify the code a bit
Stop returning 'length' in '_get_data()', because it is redundant and identical
to 'end - start + 1'.
Also add another assertion to improve robustness.
Change-Id: I693d40882fd1f47709db2687446be284de35c01f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 10:15:29 +0000 (12:15 +0200)]
test_api_base: re-structure the test a bit
I want to run the test for various different files, not just for one file.
Re-structure the test to allow this:
1. Rename the old 'test' function to '_do_test()'
2. Make the '_f_image' attribute to be an argument of '_do_test()'. So
'self._f_image' is renamed to 'f_image'.
3. Introduce a new 'test()' function which creates the file file to test
and invokes '_do_test()'. Later this function will run '_do_test()' for
more than one file.
Change-Id: I252f396e45a780b1e20b024e5053d04c3ab11b0e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 10:13:36 +0000 (12:13 +0200)]
test_helpers: generate file of correct length
The 'create_random_sparse_file()' alwasy generates files of the length aligned
to block size, even when the requested size is unaligned. Fix this by adding a
truncation at the end.
Change-Id: Id3b7f8f0e6c1d9f11aa68d06d4883b8227991e50
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 08:48:13 +0000 (10:48 +0200)]
test_api_base: saner temprorary file creation
Instead of using the low-level mkstemp - use the NamedTemporaryFile function
instead. It removes the temporary files on close.
Change-Id: I8b8b0ee9db36d626d4d7d0a00c79adf2af186ddc
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 26 Nov 2012 07:52:57 +0000 (09:52 +0200)]
tests: create a separate module for helper functions
I am going to add a test for the Fiemap module, and it will need the
'create_random_sparse_file()' function from the base API test. Therefor, move
this function to a separate 'test_helpers.py' module which will be shared
across tests.
Additionally, remove the holes verification from 'create_random_sparse_file()'
- the test which I will add soon verifies holes separately.
Also change the egging to exclude the tests.
Change-Id: I2c81251c5e1884a4f644e89b417c8a18050aeada
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Sat, 24 Nov 2012 16:29:14 +0000 (18:29 +0200)]
Fiemap: introduce a separate helper for invoking FIEMAP
Introduce a '_invoke_fiemap' helper function which hides all the cruft related
to the FIEMAP ioctl invocation.
Change-Id: I8a45591f03f1d77ac24b661af96cd4cbe583216f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Sat, 24 Nov 2012 15:27:25 +0000 (17:27 +0200)]
Fiemap: introduce 'buf_size' parameter for __init__
We are going to use full power of FIEMAP. It requires an array where it will
place the block map on output. Generally, big arrays are good because fiemap
will be called less times, which is good for performance.
Make the default to be a 1MiB buffer, and allocate it in the constructur
for further use.
Change-Id: I4e271794bf747cc7d611d86a6a76afd554c8e09f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Sat, 24 Nov 2012 15:38:16 +0000 (17:38 +0200)]
Fiemap: more input arguments validation
Validate the input argument of 'block_is_mapped()'.
Change-Id: I31ec0e4b5ff6250ed80cf4b2ea1a095ecf517927
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Sat, 24 Nov 2012 15:34:18 +0000 (17:34 +0200)]
Fiemap: validate the input arguments
Validate the input arguments for 'get_mapped_ranges()' and
'get_unmapped_ranges()' - they should be positive and should not go beyond file
size.
Change-Id: Ib992e03c6d651bd4e00025bc1a75b69a90d95a31
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Sat, 24 Nov 2012 15:17:10 +0000 (17:17 +0200)]
Fiemap: fix error message
Fix a typo in one of the error messages: s/FIBMAP/FIEMAP/
Change-Id: Iec26e20aa014eb1b28c710698dd4a69e2133649d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Sat, 24 Nov 2012 15:15:19 +0000 (17:15 +0200)]
Fiemap: introduce constants
Introduce constants for various FIEMAP-related stuff like format strings for C
structures, their sizes, and the ioctl number. First of all, this is just
nicer. But more importantly, we'll soon introduce another function which will
use these constants as well.
Change-Id: Iac00bc93d8a590dc242b8ce7b20f2fe3d98a186d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 12:29:53 +0000 (14:29 +0200)]
BmapCopy.py: remove few useless initializations
We initialize these variables right after setting them to None, which is
probably an overkill, remove the None assignment.
Change-Id: Ibf419b2d138ab025e0aa60f1eb3ac6a477a92734
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 12:27:12 +0000 (14:27 +0200)]
BmapCopy: disable a couple of pylint recommendations
Change-Id: Ibae22f9aad139904f21fa0f63c5bdde78ba05a38
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 12:26:12 +0000 (14:26 +0200)]
BmapCopy: use "_" prefix for private variables
Change-Id: I91f3875aa2f12ae485f6b757ffe5f721aa3dc87d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 12:06:50 +0000 (14:06 +0200)]
test_api_base: improve the test
Not only compare that the destination file has the same contens as the source
file, but also make sure they have identical holes. This improves the test.
Change-Id: Idb53ed7c74eda2cd72608a845687c509c5adaa7e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 11:56:16 +0000 (13:56 +0200)]
Fiemap: improve the API
Improve the API and allow to specify the area of the file to generate ranges
for, so that it would be possible to generate ranges only for part of the file.
Change-Id: I1e8455be739d9835bc3f750709fbe7d8e9374ee7
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 11:26:51 +0000 (13:26 +0200)]
test_api_base: fix the test
The test apparently was half-broken because:
1. It did not check results of 'filecmp'.
2. It did not seek the bmap file descriptor so the bmap comparison did not
work.
Change-Id: I7d6e65acf7ee3c2524064a71faccdb6554f4ffe7
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 11:18:38 +0000 (13:18 +0200)]
BmapCopy: bugfix: make destination file to be of the same size
When we copy to a regular file using bmap, and the image has holes at the end,
the resulting copy does not contain these holes and its size is shorter.
Fix this bug by truncating the destination file to the image size at the end.
Change-Id: Ie9586a1972e8b1027f76e1973fe6e10fe939f035
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 10:24:55 +0000 (12:24 +0200)]
test_api_base: add a Fiemap module check
This patch improves tesst coverage by checking the sparse file. Namely,
when we create the random sparse file, we remember where the holes are. Then
we make sure that Fiemap reports absolutely the same holes.
Change-Id: I2cf97bcf3de456b09b4138cec9be3255c88ee22e
Artem Bityutskiy [Fri, 23 Nov 2012 08:32:27 +0000 (10:32 +0200)]
BmapCreate: separate out the FIEMAP functionality
Create a separate class for the FIEMAP ioctl API. I do this because
I am going to use full power of FIEMAP and the code will become a lot
more complex, so it is nicer to have it separate.
Besides, we need a stand-alone FIEMAP API for testing.
Change-Id: Ibc4f18f4f143e6bf878730a9777e2ef8dd3cdede
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 07:57:02 +0000 (09:57 +0200)]
Bmap API: remove useless 'bmap' prefix
The 'BmapCopy' and 'BmapCreate' modules provide a set of useful attributes like
block size, blocks count, mapped blocks count, etc. All these attributes start
with a 'bmap_' prefix, which is useless and makes the code less readable.
Remove the 'bmap_' prefix.
Change-Id: I9ec859c1bc26286c7328e0a7cf26c91c47fd43d9
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 07:25:17 +0000 (09:25 +0200)]
BmapCreate: handle flush() exception
We handle file.flush() exceptions for the image, but not for bmap. This is
inconsistent. Let's handle them for both.
Change-Id: Ie4e9e28b865d2dcb9da278d577443f3665a4a907
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 07:19:27 +0000 (09:19 +0200)]
BmapCreate: simplify the exception class
Once we removed the FIBMAP support, we do not need the 'errno' attribute in the
exception class - no one uses it. Let's kill it to have the code simpler and
more clear, as well as less error-prone.
Change-Id: I97d20cf1b6550359133fff5c81136b72bc77ec2f
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 23 Nov 2012 07:15:14 +0000 (09:15 +0200)]
BmapCreate.py: simplify the module by killing FIBMAP support
We do not need to support FIBMAP because FIEMAP is a lot better and does not
require root privileges. FIEMAP is supported since 2008's kernel version
2.6.28.
Thus, kill the FIBMAP ioctl support and remove a lot of cruft.
Change-Id: Iac0c8653291c937cb4ed3e20eef88dc3fc4647d6
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 14 Nov 2012 10:38:08 +0000 (12:38 +0200)]
Pre-release version 0.6
Also change the date in the debian changelog for the 0.5 release - I forgot
to update it.
Change-Id: Ie69aba677b2812b7405327c32142583cad610856
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 21 Nov 2012 14:38:21 +0000 (16:38 +0200)]
BmapCopy.py: fix a bug when copying without bmap
Fix a regression introduced in version 0.5: when copying without bmap the
script fails with the following error:
AttributeError: BmapCopy instance has no attribute '_f_bmap'
This is because we never define this attribute when there is no bmap, but still
use it. The fix is:
1. Define all attributes in the __init__ constructor.
2. Always check if _f_bmap is not None before using it.
While on it, do a minor space clean-up.
Change-Id: I3f729588d33be8e2a447fca817b994500840503a
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 21 Nov 2012 14:36:32 +0000 (16:36 +0200)]
tests: test_api_base: test copying without bmap as well
Improve the test a little by:
1. Calling the 'copy' function with the first 'sync' argument being 'True' as
well
2. Adding another pass which tests file copying without bmap.
This discovered a bug, actually, which will be fixed in the next commit.
Change-Id: I34441b1151588c31bb202d2cdfdeb1e874f4e0bb
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 14 Nov 2012 10:38:08 +0000 (12:38 +0200)]
Pre-release version 0.5
Change-Id: I304827a5204fd3aa4c807f236ead82d1113a3e3d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 07:36:57 +0000 (09:36 +0200)]
tests: add the base functionality test
Change-Id: I035f71eadda2d2da83a1c8776f035cc1d5ccde12
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 13:54:15 +0000 (15:54 +0200)]
BmapCreate: fix and silence pylint warnings
Fix several pylint warnings and silence a couple.
Change-Id: If714828815ac72c026e05e022fb03f6ffed2fa6c
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 13:46:08 +0000 (15:46 +0200)]
BmapCopy.py: fix and silence pylint warnings
Fix a couple of pylint warnings and silence one.
Change-Id: I392e95b6bf0c3805e3c07cb8d80e0d0aef1b7311
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 13:07:44 +0000 (15:07 +0200)]
BmapCopy.py: flush the destination file on exit
Always flush the destination file on exit to make sure that if the user
opens the same file by path - he/she sees all the data.
Change-Id: I493d0c92d4818e289216925e00111128cba88db6
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 13:06:21 +0000 (15:06 +0200)]
BmapCopy.py: open the destination file for writing
Open the destination file in write-only mode, because we only write to
this file, and never read from.
Change-Id: I5bdef84ecdb7faf38190a00bb34fb1b183fb9430
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 13:04:53 +0000 (15:04 +0200)]
bmaptool: open the destination file in binary mode
This is just a small correction - the destination file is a binary file,
so open it in binary mode.
Change-Id: I38ca25dabc38dabf65a18b6f50d8de066c8f18b1
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 11:52:08 +0000 (13:52 +0200)]
BmapCreate.py: support file-like objects
It is very useful give BmapCreate file-like objects instead of file paths, and
this is exactly what this patch implements.
Change-Id: I6d342ea64584b8b203f62a0df33e2a6df5eb9da2
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 11:19:33 +0000 (13:19 +0200)]
BmapCreate.py: stop using logger for the bmap output
It is an overkill to use a logger object to output the bmap, and it is also
rather difficult for the users. Instead, make 'BmapCreate' accept a file-like
object for the output, not a logger object.
Change-Id: I7cf83dc8e0c43f762449e7fcf200b1f8b66359c5
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 10:42:01 +0000 (12:42 +0200)]
BmapCopy.py: support file-like objects
It is very useful give BmapCopy file-like objects instead of file paths, and
this is exactly what this patch implements.
Change-Id: I41b40e8369743bf53e018f503b65719a1c823464
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 09:28:43 +0000 (11:28 +0200)]
TODO: add an entry about interruptions
Change-Id: Icd9f43eb8b9b9c48624cfaf4553b9fa2acf5f509
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 09:37:08 +0000 (11:37 +0200)]
BmapCreate: remove useless seek
We do not need to seek the image to the beginning because we do not care
about the file position - we do not read of write the file.
Change-Id: Ib5e9abd8560e77b1a095d1882594de15e97be4e7
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 20 Nov 2012 09:11:24 +0000 (11:11 +0200)]
BmapCopy.py: fix incorrect assignment
This patch fixes the following bug:
UnboundLocalError: local variable 'last' referenced before assignment
which happens when the range consists only of one block.
Change-Id: I4d91d2abdd45408b6d2a6131e3ddd7fbbe3492f9
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Mon, 19 Nov 2012 15:02:40 +0000 (17:02 +0200)]
BmapHelpers.py: introduce a 'get_block_size()' function
Move the piece of code which finds out block size from 'BmapCreate.py'
to 'BmapHelpers.py' - we need this code in tests as well.
Change-Id: I1682a0cffc0348132409bdf085c9fb4fce5b4473
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 16 Nov 2012 11:31:44 +0000 (13:31 +0200)]
BmapCopy.py: save and restore block device settings
We change block device settings to improve I/O speed (e.g., switch to the
'noop' I/O scheduler). This patch also teaches the BmapCopy module to restore
the settings when the copying is done.
Change-Id: I31caddf5d86510452a7b58be3268ed973d057354
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Fri, 16 Nov 2012 10:31:56 +0000 (12:31 +0200)]
BmapCopy.py: make blkdev optimization work for partitions as wall
If we are writing to something like /dev/sdc1 instead of /dev/sdc, we should
go one level up in the sysfs hierarchy to access the block device configuration
files like 'scheduler'.
Change-Id: I845a145197aa15513a288214f0b58c7a1da3b8cf
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 14 Nov 2012 10:38:08 +0000 (12:38 +0200)]
Pre-release version 0.4
Change-Id: I50a735e3dedcdfd3035d58adb7edcc355994689a
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Thu, 15 Nov 2012 09:28:10 +0000 (11:28 +0200)]
BmapCopy.py: write without bmap the same way as with bmap
Unify the code paths for the situations when we have bmap and when we do not
have it.
Change-Id: I7a8a47cf0a475077953590ea9ebbd4b905acb4ed
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Thu, 15 Nov 2012 09:00:34 +0000 (11:00 +0200)]
BmapCreate: generate more readable bmap
When the blocks rage consists of a single block (say 18282), write it
in form of <>18282</> instead of less readable <>18282-18282</> form.
'bmaptool copy' supports both.
Change-Id: I96cd97b1913227de878004fc29eb65b76f11058e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Thu, 15 Nov 2012 08:51:54 +0000 (10:51 +0200)]
BmapCopy.py: fix another pylint complaint
Change-Id: I526140d32b362cb7669f1ce10dbae793b8937229
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Thu, 15 Nov 2012 08:48:56 +0000 (10:48 +0200)]
BmapCreate.py: remove too long lines
Pylint complains a lot about long lines, which is fixed by this patch.
Change-Id: Ib615a44d7529ce35717cd00ac2714b6e36301e68
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 14 Nov 2012 09:59:38 +0000 (11:59 +0200)]
BmapCopy.py: handle exceptions from the reader thread
Use the batch queue to pass exceptions from the reader thread to the main
thread in case of errors, and raise the exceptions from the reader thread.
Change-Id: I02801278b76bb1b390b6e36372ee0d5b487e79c2
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 13 Nov 2012 13:43:50 +0000 (15:43 +0200)]
BmapCopy: implement threaded reader
Read the data from a separate thread. This does not change the performance
measurably in case of uncompressed images, but does improve writing speed in
case of compressed images - Tizen.bz2 image flashing drops from 2m13s to
1m43s.
Change-Id: I3300a569042d16357fcc0920c3f03f3fff08d29d
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 13 Nov 2012 12:52:26 +0000 (14:52 +0200)]
BmapCopy.py: implement an iterator for reading the image
Implement a '_get_batches()' iterator which reads batches of data from
the image file.
Change-Id: If914efc60ea6a259f8430ded8c2d4143b07adc3c
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 13 Nov 2012 11:57:56 +0000 (13:57 +0200)]
BmapCopy.py: introduce a helper iterator
The '_copy_data()' function is a bit ugly and large. Simplify it by introducing
an iterator which splits the entire blocks range on smaller batch ranges, and
we do actual I/O in these small batches.
Change-Id: I6c6e2cb23672a6abbdb679776c82046b5706b5c8
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 13 Nov 2012 11:37:03 +0000 (13:37 +0200)]
BmapCopy.py: move chunk_size to the class level
We use 'chunk_size' variable in several places and hard-code it to various
values. Clean this up by introducing class-level variables for size of the
I/O operations we do. Call them 'batch_blocks' and 'batch_bytes' for the
I/O batch size in blocks and bytes.
Change-Id: I0ec6058a20e89d685e917c13268c2f70f385c053
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 13 Nov 2012 10:18:12 +0000 (12:18 +0200)]
BmapCopy.py: introduce a iterator for blocks ranges
Add a 'get_block_ranges()' iterator function which parses the bmap
file and returns block ranges. This makes the code a bit nicer and
modular.
Change-Id: I09270f4bc3f74b676fcf3b3f6e87c57e222c709a
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 14 Nov 2012 13:25:05 +0000 (15:25 +0200)]
bmaptool: fix a print
We write to regular files too, not only to block devices, fix the
message correspondingly.
Change-Id: I169fc51bfd2833a01749fd30d936da4e6bd8b98e
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Wed, 14 Nov 2012 13:23:43 +0000 (15:23 +0200)]
Fix a number of pylint warnings
Change-Id: I18007ca6c50e3a733d5ee9e944636bfdaab97f45
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>
Artem Bityutskiy [Tue, 13 Nov 2012 09:00:05 +0000 (11:00 +0200)]
bmaptool: pre-release version 0.3
Change-Id: I1dacb92a73445600497a608917367ba4a2aff295
Signed-off-by: Artem Bityutskiy <artem.bityutskiy@intel.com>