From: Sehong Na Date: Sat, 31 May 2014 04:00:41 +0000 (+0900) Subject: Initialize Tizen 2.3 X-Git-Tag: 2.3a_release X-Git-Url: http://review.tizen.org/git/?a=commitdiff_plain;h=refs%2Ftags%2F2.3a_release;p=external%2Fsquashfs.git Initialize Tizen 2.3 --- ec5bd9231f2fc5861b1968905d8e455492fb1995 diff --git a/ACKNOWLEDGEMENTS b/ACKNOWLEDGEMENTS new file mode 100644 index 0000000..e346f7a --- /dev/null +++ b/ACKNOWLEDGEMENTS @@ -0,0 +1,160 @@ + ACKNOWLEDGEMENTS + +Thanks to everyone who have download squashfs. I appreciate people +using it, and any feedback you have. + +The following have provided useful feedback, which has guided +some of the extra features in squashfs. This is a randomly ordered +(roughly in chronological order) list, which is updated when +I remember... + + +Acknowledgements for Squashfs 4.2 +--------------------------------- + +Thanks to Lasse Collin (http://tukaani.org/xz/) for mainlining XZ +decompression support. + +Acknowledgements for Squashfs 4.1 +--------------------------------- + +Thanks to Chan Jeong and LG for the patches to support LZO +compression. + +Acknowledgements for Squashfs 4.0 +--------------------------------- + +Thanks to Tim Bird and CELF (Consumer Electronics Linux Forum) for helping +fund mainstreaming of Squashfs into the 2.6.29 kernel and the +changes to the Squashfs tools to support the new 4.0 file system layout. + +Acknowledgements for Squashfs-3.3 +------------------------------------ + +Peter Korsgaard and others sent patches updating Squashfs to changes in the +VFS interface for 2.6.22/2.6.23/2.6.24-rc1. Peter also sent some small patches +for the Squashfs kernel code. + +Vito Di Leo sent a patch extending Mksquashfs to support regex filters. +While his patched worked, it unfortunately made it easy to make Mksquashfs +perform unpredictably with poorly choosen regex expressions. It, however, +encouraged myself to add support for wildcard pattern matching and regex +filters in a different way. + +Acknowledgements for Squashfs-3.2-r2 +------------------------------------ + +Junjiro Okajima discovered a couple of SMP issues, thanks. + +Junjiro Okajima and Tomas Matejicek have produced some good LZMA patches +for Squashfs. + +Acknowledgements for Squashfs-3.2 +--------------------------------- + +Peter Korsgaard sent a patch updating Squashfs to changes in the VFS interface +in Linux 2.6.20. + +Acknowledgements for Squashfs-3.1 +--------------------------------- + +Kenneth Duda and Ed Swierk of Arastra Inc. identified numerous bugs with +Squashfs, and provided patches which were the basis for some of the +fixes. In particular they identified the fragment rounding bug, the +NFS bug, the initrd bug, and helped identify the 4K stack overflow bug. + +Scott James Remnant (Ubuntu) also identified the fragment rounding bug, +and he also provided a patch. + +Ming Zhang identified the Lseek bug in Mksquashfs. His tests on the +performance of Mksquashfs on SMP systems encouraged the rewrite of +Mksquashfs. + +Peter Korsgaard, Daniel Olivera and Zilvinas Valinskas noticed +Squashfs 3.0 didn't compile on Linux-2.6.18-rc[1-4] due to changes +in the Linux VFS interfaces, and provided patches. + +Tomas Matejicek (SLAX) suggested the -force option on Unsquashfs, and noticed +Unsquashfs didn't return the correct exit status. + +Yann Le Doare reported a kernel oops and provided a Qemu image that led +to the identification of the simultaneously accessing multiply mounted Squashfs +filesystems bug. + + +Older acknowledgements +---------------------- + +Mark Robson - pointed out early on that initrds didn't work + +Adam Warner - pointed out that greater than 2GB filesystems didn't work. + +John Sutton - raised the problem when archiving the entire filesystem +(/) there was no way to prevent /proc being archived. This prompted +exclude files. + +Martin Mueller (LinuxTV) - noticed that the filesystem length in the +superblock doesn't match the output filesystem length. This is due to +padding to a 4K boundary. This prompted the addition of the -nopad option. +He also reported a problem where 32K block filesystems hung when used as +initrds. + +Arkadiusz Patyk (Polish Linux Distribution - PLD) reported a problem where 32K +block filesystems hung when used as a root filesystem mounted as a loopback +device. + +Joe Blow emailed me that I'd forgotten to put anything in the README about +mounting the squashfs filesystem. + +David Fox (Lindows) noticed that the exit codes returned by Mksquashfs were +wrong. He also noticed that a lot of time was spent in the duplicate scan +routine. + +Cameron Rich complained that Squashfs did not support FIFOs or sockets. + +Steve Chadsey and Thomas Weissmuller noticed that files larger than the +available memory could not be compressed by Mksquashfs. + +"Ptwahyu" and "Hoan" (I have no full names and I don't like giving people's +email addresses), noticed that Mksquashfs 1.3 SEGV'd occasionally. Even though +I had already noticed this bug, it is useful to be informed by other people. + +Don Elwell, Murray Jensen and Cameron Rich, have all sent in patches. Thanks, +I have not had time to do anything about them yet... + +Drew Scott Daniels has been a good advocate for Squashfs. + +Erik Andersen has made some nice suggestions, unfortunately, I have +not had time to implement anything. + +Artemiy I. Pavlov has written a useful LDP mini-howto for Squashfs +(http://linuxdoc.artemio.net/squashfs). + +Yves Combe reported the Apple G5 bug, when using Squashfs for +his PPC Knoppix-mib livecd project. + +Jaco Greeff (mklivecd project, and maintainer of the Mandrake +squashfs-tools package) suggested the new mksquashfs -ef option, and the +standalone build for mksquashfs. + +Mike Schaudies made a donation. + +Arkadiusz Patyk from the Polish Linux Distribution reported that Squashfs +didn't work on amd64 machines. He gave me an account on a PLD amd64 machine +which allowed myself to track down these bugs. + +Miles Roper, Peter Kjellerstedt and Willy Tarreau reported that release 2.1 did +not compile with gcc < 3.x. + +Marcel J.E. Mol reported lack of kernel memory issues when using Squashfs +on small memory embedded systems. This prompted the addition of the embedded +system kernel configuration options. + +Era Scarecrow noticed that Mksquashfs had not been updated to reflect that +smaller than 4K blocks are no longer supported. + +Kenichi Shima reported the Kconfig file had not been updated to 2.2. + +Aaron Ten Clay made a donation! + +Tomas Matejicek (SLAX) made a donation! diff --git a/CHANGES b/CHANGES new file mode 100644 index 0000000..e088c71 --- /dev/null +++ b/CHANGES @@ -0,0 +1,510 @@ + SQUASHFS CHANGE LOG + +4.2 28 FEB 2011 XZ compression, and compression options support + + 1. Filesystem improvements: + + 1.1 Added XZ compression + 1.2 Added compression options support + + 2. Miscellaneous improvements/bug fixes + + 1.1 Add missing NO_XATTR filesystem flag to indicate no-xattrs + option was specified and no xattrs should be stored when + appending. + 1.2 Add suppport in Unquashfs -stat option for displaying + NO_XATTR flag. + 1.3 Remove checkdata entry from Unsquashfs -stat option if a 4.0 + filesystem - checkdata is no longer supported. + 1.4 Fix appending bug when appending to an empty filesystem - this + would be incorrectly treated as an error. + 1.5 Use glibc sys/xattr.h include rather than using attr/xattr.h + which isn't present by default on some distributions. + 1.6 Unsquashfs, fix block calculation error with regular files when + file size is between 2^32-block_size+1 and 2^32-1. + 1.7 Unsquashfs, fix sparse file writing when holes are larger than + 2^31-1. + 1.8 Add external CFLAGS and LDFLAGS support to Makefile, and allow + build options to be specified on command line. Also don't + over-write passed in CFLAGS definition. + + +4.1 19 SEPT 2010 Major filesystem and tools improvements + + 1. Filesystem improvements: + + 1.1 Extended attribute support + 1.2 New compression framework + 1.3 Support for LZO compression + 1.4 Support for LZMA compression (not yet in mainline) + + 2. Mksquashfs improvements: + + 1.1 Enhanced pseudo file support + 1.2 New options for choosing compression algorithm used + 1.3 New options for controlling extended attributes + 1.4 Fix misalignment issues with memcpy etc. seen on ARM + 1.5 Fix floating point error in progress_bar when max == 0 + 1.6 Removed use of get_nproc() call unavailable in ulibc + 1.7 Reorganised help text + + 3. Unsquashfs improvements: + + 1.1 New options for controlling extended attributes + 1.2 Fix misalignment issues with memcpy etc. seen on ARM + 1.3 Fix floating point error in progress_bar when max == 0 + 1.4 Removed use of get_nproc() call unavailable in ulibc + + +4.0 5 APR 2009 Major filesystems improvements + + 1. Kernel code improvements: + + 1.1 Fixed little endian layout adopted. All swapping macros + removed, and in-line swapping added for big-endian + architectures. + 1.2 Kernel code substantially improved and restructured. + 1.3 Kernel code split into separate files along functional lines. + 1.4 Vmalloc usage removed, and code changed to use separately + allocated 4K buffers + + 2. Unsquashfs improvements: + + 2.1 Support for 4.0 filesystems added. + 2.2 Swapping macros rewritten. + 2.3 Unsquashfs code restructured and split into separate files. + + 3. Mksquashfs improvements: + + 3.1 Swapping macros rewritten. Fixed little-endian layout allows + code to be optimised and only added at compile time for + big endian systems. + 3.2 Support for pseudo files added. + +3.4 26 AUG 2008 Performance improvements to Unsquashfs, Mksquashfs + and the kernel code. Plus many small bug fixes. + + 1. Kernel code improvements: + + 1.1 Internal Squashfs kernel metadata and fragment cache + implementations have been merged and optimised. Spinlocks are + now used, locks are held for smaller periods and wakeups have + been minimised. Small race condition fixed where if two or + more processes tried to read the same cache block + simultaneously they would both read and decompress it. 10-20%+ + speed improvement has been seen on tests. + 1.2 NFS export code rewritten following VFS changes in + linux-2.6.24. + 1.3 New patches for linux-2.6.25, linux-2.6.26, and linux-2.6.27. + Fixed patch for linux-2.6.24. + 1.4 Fixed small buffer_head leak in squashfs_read_data when + handling badly corrupted filesystems. + 1.5 Fixed bug in get_dir_index_using_offset. + + 2. Unsquashfs improvements: + + 2.1 Unsquashfs has been parallelised. Filesystem reading, writing + and decompression is now multi-threaded. Up to 40% speed + improvement seen on tests. + 2.2 Unsquashfs now has a progress bar. Use -no-progress to + disable it. + 2.3 Fixed small bug where unistd.h wasn't being included on + some distributions, leading to lseek being used rather than + lseek64 - which meant on these distributions Unsquashfs + couldn't unsquash filesystems larger than 4GB. + + 3. Mksquashfs improvements: + + 3.1 Removed some small remaining parallelisation bottlenecks. + Depending on source filesystem, up to 10%+ speed improvement. + 3.2 Progress bar improved, and moved to separate thread. + 3.3 Sparse file handling bug in Mksquashfs 3.3 fixed. + 3.4 Two rare appending restore bugs fixed (when ^C hit twice). + + +3.3 1 NOV 2007 Increase in block size, sparse file support, + Mksquashfs and Unsquashfs extended to use + pattern matching in exclude/extract files, plus + many more improvements and bug fixes. + + 1. Filesystem improvements: + + 1.1. Maximum block size has been increased to 1Mbyte, and the + default block size has been increased to 128 Kbytes. + This improves compression. + + 1.2. Sparse files are now supported. Sparse files are files + which have large areas of unallocated data commonly called + holes. These files are now detected by Squashfs and stored + more efficiently. This improves compression and read + performance for sparse files. + + 2. Mksquashfs improvements: + + 2.1. Exclude files have been extended to use wildcard pattern + matching and regular expressions. Support has also been + added for non-anchored excludes, which means it is + now possible to specify excludes which match anywhere + in the filesystem (i.e. leaf files), rather than always + having to specify exclude files starting from the root + directory (anchored excludes). + + 2.2. Recovery files are now created when appending to existing + Squashfs filesystems. This allows the original filesystem + to be recovered if Mksquashfs aborts unexpectedly + (i.e. power failure). + + 3. Unsquashfs improvements: + + 3.1. Multiple extract files can now be specified on the + command line, and the files/directories to be extracted can + now also be given in a file. + + 3.2. Extract files have been extended to use wildcard pattern + matching and regular expressions. + + 3.3. Filename printing has been enhanced and Unquashfs can + now display filenames with file attributes + ('ls -l' style output). + + 3.4. A -stat option has been added which displays the filesystem + superblock information. + + 3.5. Unsquashfs now supports 1.x filesystems. + + 4. Miscellaneous improvements/bug fixes: + + 4.1. Squashfs kernel code improved to use SetPageError in + squashfs_readpage() if I/O error occurs. + + 4.2. Fixed Squashfs kernel code bug preventing file + seeking beyond 2GB. + + 4.3. Mksquashfs now detects file size changes between + first phase directory scan and second phase filesystem create. + It also deals better with file I/O errors. + + +3.2-r2 15 JAN 2007 Kernel patch update and progress bar bug fix + + 1. Kernel patches 2.6.19/2.6.20 have been updated to use + const structures and mutexes rather than older semaphores. + 2. Minor SMP bug fixes. + 3. Progress bar broken on x86-64. Fixed. + +3.2 2 JAN 2007 NFS support, improvements to the Squashfs-tools, major + bug fixes, lots of small improvements/bug fixes, and new + kernel patches. + + Improvements: + + 1. Squashfs filesystems can now be exported via NFS. + 2. Unsquashfs now supports 2.x filesystems. + 3. Mksquashfs now displays a progress bar. + 4. Squashfs kernel code has been hardened against accidently or + maliciously corrupted Squashfs filesystems. + + Bug fixes: + + 5. Race condition occurring on S390 in readpage() fixed. + 6. Odd behaviour of MIPS memcpy in read_data() routine worked-around. + 7. Missing cache_flush in Squashfs symlink_readpage() added. + + +3.1-r2 30 AUG 2006 Mksquashfs -sort bug fix + + A code optimisation after testing unfortunately + broke sorting in Mksquashfs. This has been fixed. + +3.1 19 AUG 2006 This release has some major improvements to + the squashfs-tools, a couple of major bug + fixes, lots of small improvements/bug fixes, + and new kernel patches. + + + 1. Mksquashfs has been rewritten to be multi-threaded. It + has the following improvements + + 1.1. Parallel compression. By default as many compression and + fragment compression threads are created as there are available + processors. This significantly speeds up performance on SMP + systems. + 1.2. File input and filesystem output is peformed in parallel on + separate threads to maximise I/O performance. Even on single + processor systems this speeds up performance by at least 10%. + 1.3. Appending has been significantly improved, and files within the + filesystem being appended to are no longer scanned and + checksummed. This significantly improves append time for large + filesystems. + 1.4. File duplicate checking has been optimised, and split into two + separate phases. Only files which are considered possible + duplicates after the first phase are checksummed and cached in + memory. + 1.5 The use of swap memory was found to significantly impact + performance. The amount of memory used to cache files is now a + command line option, by default this is 512 Mbytes. + + 2. Unsquashfs has the following improvements + + 2.1 Unsquashfs now allows you to specify the filename or the + directory within the Squashfs filesystem that is to be + extracted, rather than always extracting the entire filesystem. + 2.2 A new -force option has been added which forces Unsquashfs to + output to the destination directory even if files and directories + already exist in the destination directory. This allows you to + update an already existing directory tree, or to Unsquashfs to + a partially filled directory tree. Without the -force option + Unsquashfs will refuse to output. + + 3. The following major bug fixes have been made + + 3.1 A fragment table rounding bug has been fixed in Mksquashfs. + Previously if the number of fragments in the filesystem + were a multiple of 512, Mksquashfs would generate an + incorrect filesystem. + 3.2 A rare SMP bug which occurred when simultaneously acccessing + multiply mounted Squashfs filesystems has been fixed. + + 4. Miscellaneous improvements/bug fixes + + 4.1 Kernel code stack usage has been reduced. This is to ensure + Squashfs works with 4K stacks. + 4.2 Readdir (Squashfs kernel code) has been fixed to always + return 0, rather than the number of directories read. Squashfs + should now interact better with NFS. + 4.3 Lseek bug in Mksquashfs when appending to larger than 4GB + filesystems fixed. + 4.4 Squashfs 2.x initrds can now been mounted. + 4.5 Unsquashfs exit status fixed. + 4.6 New patches for linux-2.6.18 and linux-2.4.33. + + +3.0 15 MAR 2006 Major filesystem improvements + + 1. Filesystems are no longer limited to 4 GB. In + theory 2^64 or 4 exabytes is now supported. + 2. Files are no longer limited to 4 GB. In theory the maximum + file size is 4 exabytes. + 3. Metadata (inode table and directory tables) are no longer + restricted to 16 Mbytes. + 4. Hardlinks are now suppported. + 5. Nlink counts are now supported. + 6. Readdir now returns '.' and '..' entries. + 7. Special support for files larger than 256 MB has been added to + the Squashfs kernel code for faster read access. + 8. Inode numbers are now stored within the inode rather than being + computed from inode location on disk (this is not so much an + improvement, but a change forced by the previously listed + improvements). + +2.2-r2 8 SEPT 2005 Second release of 2.2, this release fixes a couple + of small bugs, a couple of small documentation + mistakes, and adds a patch for kernel 2.6.13. + + 1. Mksquashfs now deletes the output filesystem image file if an + error occurs whilst generating the filesystem. Previously on + error the image file was left empty or partially written. + 2. Updated mksquashfs so that it doesn't allow you to generate + filesystems with block sizes smaller than 4K. Squashfs hasn't + supported block sizes less than 4K since 2.0-alpha. + 3. Mksquashfs now ignores missing files/directories in sort files. + This was the original behaviour before 2.2. + 4. Fixed small mistake in fs/Kconfig where the version was still + listed as 2.0. + 5. Updated ACKNOWLEDGEMENTS file. + + +2.2 3 JUL 2005 This release has some small improvements, bug fixes + and patches for new kernels. + + 1. Sort routine re-worked and debugged from release 2.1. It now allows + you to give Mkisofs style sort files and checks for filenames that + don't match anything. Sort priority has also been changed to + conform to Mkisofs usage, highest priority files are now placed + at the start of the filesystem (this means they will be on the + inside of a CD or DVD). + 2. New Configure options for embedded systems (memory constrained + systems). See INSTALL file for further details. + 3. Directory index bug fixed where chars were treated as signed on + some architectures. A file would not be found in the rare case + that the filename started with a chracter greater than 127. + 4. Bug introduced into the read_data() routine when sped up to use data + block queueing fixed. If the second or later block resulted in an + I/O error this was not checked. + 5. Append bug introduced in 2.1 fixed. The code to compute the new + compressed and uncompressed directory parts after appending was + wrong. + 6. Metadata block length read routine altered to not perform a + misaligned short read. This was to fix reading on an ARM7 running + uCLinux without a misaligned read interrupt handler. + 7. Checkdata bug introduced in 2.1 fixed. + + +2.1-r2 15 DEC 2004 Code changed so it can be compiled with gcc 2.x + + 1. In some of the code added for release 2.1 I unknowingly used some + gcc extensions only supported by 3.x compilers. I have received + a couple of reports that the 2.1 release doesn't build on 2.x and so + people are clearly still using gcc 2.x. The code has been + rewritten to remove these extensions. + +2.1 10 DEC 2004 Significantly improved directory handling plus numerous + other smaller improvements + + 1. Fast indexed directories implemented. These speed up directory + operations (ls, file lookup etc.) significantly for directories + larger than 8 KB. + 2. All directories are now sorted in alphabetical order. This again + speeds up directory operations, and in some cases it also results in + a small compression improvement (greater data similarity between + files with alphabetically similar names). + 3. Maximum directory size increased from 512 KB to 128 MB. + 4. Duplicate fragment checking and appending optimised in mksquashfs, + depending on filesystem, this is now up to 25% faster. + 5. Mksquashfs help information reformatted and reorganised. + 6. The Squashfs version and release date is now printed at kernel + boot-time or module insertion. This addition will hopefully help + to reduce the growing problem where the Squashfs version supported + by a kernel is unknown and the kernel source is unavailable. + 7. New PERFORMANCE.README file. + 8. New -2.0 mksquashfs option. + 9. CHANGES file reorganised. + 10. README file reorganised, clarified and updated to include the 2.0 + mksquashfs options. + 11. New patch for Linux 2.6.9. + 12. New patch for Linux 2.4.28. + +2.0r2 29 AUG 2004 Workaround for kernel bug in kernels 2.6.8 and newer + added + + 1. New patch for kernel 2.6.8.1. This includes a workaround for a + kernel bug introduced in 2.6.7bk14, which is present in all later + versions of the kernel. + + If you're using a 2.6.8 kernel or later then you must use this + 2.6.8.1 patch. If you've experienced hangs or oopses using Squashfs + with a 2.6.8 or later kernel then you've hit this bug, and this + patch will fix it. + + It is worth mentioning that this kernel bug potentially affects + other filesystems. If you receive odd results with other + filesystems you may be experiencing this bug with that filesystem. + I submitted a patch but this has not yet gone into the + kernel, hopefully the bug will be fixed in later kernels. + +2.0 13 JULY 2004 A couple of new options, and some bug fixes + + 1. New mksquashfs -all-root, -root-owned, -force-uid, and -force-gid + options. These allow the uids/gids of files in the generated + filesystem to be specified, overriding the uids/gids in the + source filesystem. + 2. Initrds are now supported for kernels 2.6.x. + 3. amd64 bug fixes. If you use an amd64, please read the README-AMD64 + file. + 4. Check-data and gid bug fixes. With 2.0-alpha when mounting 1.x + filesystems in certain cases file gids were corrupted. + 5. New patch for Linux 2.6.7. + +2.0-ALPHA 21 MAY 2004 Filesystem changes and compression improvements + + 1. Squashfs 2.0 has added the concept of fragment blocks. + Files smaller than the file block size and optionally the + remainder of files that do not fit fully into a block (i.e. the + last 32K in a 96K file) are packed into shared fragments and + compressed together. This achieves on average 5 - 20% better + compression than Squashfs 1.x. + 2. The maximum block size has been increased to 64K (in the ALPHA + version of Squashfs 2.0). + 3. The maximum number of UIDs has been increased to 256 (from 48 in + 1.x). + 4. The maximum number of GIDs has been increased to 256 (from 15 in + 1.x). + 5. Removal of sleep_on() function call in 2.6.x patch, to allow Squashfs + to work on the Fedora rc2 kernel. + 6. Numerous small bug fixes have been made. + +1.3r3 18 JAN 2004 Third release of 1.3, this adds a new mksquashfs option, + some bug fixes, and extra patches for new kernels + + 1. New mksquashfs -ef exclude option. This option reads the exclude + dirs/files from an exclude file, one exclude dir/file per line. This + avoids the command line size limit when using the -e exclude option, + 2. When appending to existing filesystems, if mksquashfs experiences a + fatal error (e.g. out of space when adding to the destination), the + original filesystem is restored, + 3. Mksquashfs now builds standalone, without the kernel needing to be + patched. + 4. Bug fix in the kernel squashfs filesystem, where the pages being + filled were not kmapped. This seems to only have caused problems + on an Apple G5, + 5. New patch for Linux 2.4.24, + + 6. New patch for Linux 2.6.1, this replaces the patch for 2.6.0-test7. + +1.3r2 14 OCT 2003 Second release of 1.3, bug fixes and extra patches for + new kernels + + 1. Bug fix in routine that adds files to the filesystem being + generated in mksquashfs. This bug was introduced in 1.3 + (not enough testing...) when I rewrote it to handle files larger + than available memory. This bug caused a SEGV, so if you've ever + got that, it is now fixed, + 2. Long running bug where ls -s and du reported wrong block size + fixed. I'm pretty sure this used to work many kernel versions ago + (2.4.7) but it broke somewhere along the line since then, + 3. New patch for Linux 2.4.22, + 4. New patch for 2.6.0-test7, this replaces the patch for 2.6.0-test1. + +1.3 29 JUL 2003 FIFO/Socket support added plus optimisations and + improvements + + 1. FIFOs and Socket inodes are now supported, + 2. Mksquashfs can now compress files larger than available + memory, + 3. File duplicate check routine optimised, + 4. Exit codes fixed in Mksquashfs, + 5. Patch for Linux 2.4.21, + 6. Patch for Linux 2.6.0-test1. Hopefully, this will work for + the next few releases of 2.6.0-testx, otherwise, I'll be + releasing a lot of updates to the 2.6.0 patch... + +1.2 13 MAR 2003 Append feature and new mksquashfs options added + + Mksquashfs can now add to existing squashfs filesystems. Three extra + options "-noappend", "-keep-as-directory", and "root-becomes" + have been added. + + The append option with file duplicate detection, means squashfs can be + used as a simple versioning archiving filesystem. A squashfs + filesystem can be created with for example the linux-2.4.19 source. + Appending the linux-2.4.20 source will create a filesystem with the + two source trees, but only the changed files will take extra room, + the unchanged files will be detected as duplicates. + + See the README file for usage changes. + +1.1b 16 JAN 2003 Bug fix release + + Fixed readpage deadlock bug. This was a rare deadlock bug that + happened when pushing pages into the page cache when using greater + than 4K blocks. I never got this bug when I tested the filesystem, + but two people emailed me on the same day about the problem! + I fixed it by using a page cache function that wasn't there when + I originally did the work, which was nice :-) + +1.1 8 JAN 2003 Added features + + 1. Kernel squashfs can now mount different byte order filesystems. + 2. Additional features added to mksquashfs. Mksquashfs now supports + exclude files and multiple source files/directories can be + specified. A nopad option has also been added, which + informs mksquashfs not to pad filesystems to a multiple of 4K. + See README for mksquashfs usage changes. + 3. Greater than 2GB filesystems bug fix. Filesystems greater than 2GB + can now be created. + +1.0c 14 NOV 2002 Bug fix release + + Fixed bugs with initrds and device nodes + +1.0 23 OCT 2002 Initial release diff --git a/COPYING b/COPYING new file mode 100644 index 0000000..60549be --- /dev/null +++ b/COPYING @@ -0,0 +1,340 @@ + GNU GENERAL PUBLIC LICENSE + Version 2, June 1991 + + Copyright (C) 1989, 1991 Free Software Foundation, Inc. + 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The licenses for most software are designed to take away your +freedom to share and change it. By contrast, the GNU General Public +License is intended to guarantee your freedom to share and change free +software--to make sure the software is free for all its users. This +General Public License applies to most of the Free Software +Foundation's software and to any other program whose authors commit to +using it. (Some other Free Software Foundation software is covered by +the GNU Library General Public License instead.) You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +this service if you wish), that you receive source code or can get it +if you want it, that you can change the software or use pieces of it +in new free programs; and that you know you can do these things. + + To protect your rights, we need to make restrictions that forbid +anyone to deny you these rights or to ask you to surrender the rights. +These restrictions translate to certain responsibilities for you if you +distribute copies of the software, or if you modify it. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must give the recipients all the rights that +you have. You must make sure that they, too, receive or can get the +source code. And you must show them these terms so they know their +rights. + + We protect your rights with two steps: (1) copyright the software, and +(2) offer you this license which gives you legal permission to copy, +distribute and/or modify the software. + + Also, for each author's protection and ours, we want to make certain +that everyone understands that there is no warranty for this free +software. If the software is modified by someone else and passed on, we +want its recipients to know that what they have is not the original, so +that any problems introduced by others will not reflect on the original +authors' reputations. + + Finally, any free program is threatened constantly by software +patents. We wish to avoid the danger that redistributors of a free +program will individually obtain patent licenses, in effect making the +program proprietary. To prevent this, we have made it clear that any +patent must be licensed for everyone's free use or not licensed at all. + + The precise terms and conditions for copying, distribution and +modification follow. + + GNU GENERAL PUBLIC LICENSE + TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION + + 0. This License applies to any program or other work which contains +a notice placed by the copyright holder saying it may be distributed +under the terms of this General Public License. The "Program", below, +refers to any such program or work, and a "work based on the Program" +means either the Program or any derivative work under copyright law: +that is to say, a work containing the Program or a portion of it, +either verbatim or with modifications and/or translated into another +language. (Hereinafter, translation is included without limitation in +the term "modification".) Each licensee is addressed as "you". + +Activities other than copying, distribution and modification are not +covered by this License; they are outside its scope. The act of +running the Program is not restricted, and the output from the Program +is covered only if its contents constitute a work based on the +Program (independent of having been made by running the Program). +Whether that is true depends on what the Program does. + + 1. You may copy and distribute verbatim copies of the Program's +source code as you receive it, in any medium, provided that you +conspicuously and appropriately publish on each copy an appropriate +copyright notice and disclaimer of warranty; keep intact all the +notices that refer to this License and to the absence of any warranty; +and give any other recipients of the Program a copy of this License +along with the Program. + +You may charge a fee for the physical act of transferring a copy, and +you may at your option offer warranty protection in exchange for a fee. + + 2. You may modify your copy or copies of the Program or any portion +of it, thus forming a work based on the Program, and copy and +distribute such modifications or work under the terms of Section 1 +above, provided that you also meet all of these conditions: + + a) You must cause the modified files to carry prominent notices + stating that you changed the files and the date of any change. + + b) You must cause any work that you distribute or publish, that in + whole or in part contains or is derived from the Program or any + part thereof, to be licensed as a whole at no charge to all third + parties under the terms of this License. + + c) If the modified program normally reads commands interactively + when run, you must cause it, when started running for such + interactive use in the most ordinary way, to print or display an + announcement including an appropriate copyright notice and a + notice that there is no warranty (or else, saying that you provide + a warranty) and that users may redistribute the program under + these conditions, and telling the user how to view a copy of this + License. (Exception: if the Program itself is interactive but + does not normally print such an announcement, your work based on + the Program is not required to print an announcement.) + +These requirements apply to the modified work as a whole. If +identifiable sections of that work are not derived from the Program, +and can be reasonably considered independent and separate works in +themselves, then this License, and its terms, do not apply to those +sections when you distribute them as separate works. But when you +distribute the same sections as part of a whole which is a work based +on the Program, the distribution of the whole must be on the terms of +this License, whose permissions for other licensees extend to the +entire whole, and thus to each and every part regardless of who wrote it. + +Thus, it is not the intent of this section to claim rights or contest +your rights to work written entirely by you; rather, the intent is to +exercise the right to control the distribution of derivative or +collective works based on the Program. + +In addition, mere aggregation of another work not based on the Program +with the Program (or with a work based on the Program) on a volume of +a storage or distribution medium does not bring the other work under +the scope of this License. + + 3. You may copy and distribute the Program (or a work based on it, +under Section 2) in object code or executable form under the terms of +Sections 1 and 2 above provided that you also do one of the following: + + a) Accompany it with the complete corresponding machine-readable + source code, which must be distributed under the terms of Sections + 1 and 2 above on a medium customarily used for software interchange; or, + + b) Accompany it with a written offer, valid for at least three + years, to give any third party, for a charge no more than your + cost of physically performing source distribution, a complete + machine-readable copy of the corresponding source code, to be + distributed under the terms of Sections 1 and 2 above on a medium + customarily used for software interchange; or, + + c) Accompany it with the information you received as to the offer + to distribute corresponding source code. (This alternative is + allowed only for noncommercial distribution and only if you + received the program in object code or executable form with such + an offer, in accord with Subsection b above.) + +The source code for a work means the preferred form of the work for +making modifications to it. For an executable work, complete source +code means all the source code for all modules it contains, plus any +associated interface definition files, plus the scripts used to +control compilation and installation of the executable. However, as a +special exception, the source code distributed need not include +anything that is normally distributed (in either source or binary +form) with the major components (compiler, kernel, and so on) of the +operating system on which the executable runs, unless that component +itself accompanies the executable. + +If distribution of executable or object code is made by offering +access to copy from a designated place, then offering equivalent +access to copy the source code from the same place counts as +distribution of the source code, even though third parties are not +compelled to copy the source along with the object code. + + 4. You may not copy, modify, sublicense, or distribute the Program +except as expressly provided under this License. Any attempt +otherwise to copy, modify, sublicense or distribute the Program is +void, and will automatically terminate your rights under this License. +However, parties who have received copies, or rights, from you under +this License will not have their licenses terminated so long as such +parties remain in full compliance. + + 5. You are not required to accept this License, since you have not +signed it. However, nothing else grants you permission to modify or +distribute the Program or its derivative works. These actions are +prohibited by law if you do not accept this License. Therefore, by +modifying or distributing the Program (or any work based on the +Program), you indicate your acceptance of this License to do so, and +all its terms and conditions for copying, distributing or modifying +the Program or works based on it. + + 6. Each time you redistribute the Program (or any work based on the +Program), the recipient automatically receives a license from the +original licensor to copy, distribute or modify the Program subject to +these terms and conditions. You may not impose any further +restrictions on the recipients' exercise of the rights granted herein. +You are not responsible for enforcing compliance by third parties to +this License. + + 7. If, as a consequence of a court judgment or allegation of patent +infringement or for any other reason (not limited to patent issues), +conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot +distribute so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you +may not distribute the Program at all. For example, if a patent +license would not permit royalty-free redistribution of the Program by +all those who receive copies directly or indirectly through you, then +the only way you could satisfy both it and this License would be to +refrain entirely from distribution of the Program. + +If any portion of this section is held invalid or unenforceable under +any particular circumstance, the balance of the section is intended to +apply and the section as a whole is intended to apply in other +circumstances. + +It is not the purpose of this section to induce you to infringe any +patents or other property right claims or to contest validity of any +such claims; this section has the sole purpose of protecting the +integrity of the free software distribution system, which is +implemented by public license practices. Many people have made +generous contributions to the wide range of software distributed +through that system in reliance on consistent application of that +system; it is up to the author/donor to decide if he or she is willing +to distribute software through any other system and a licensee cannot +impose that choice. + +This section is intended to make thoroughly clear what is believed to +be a consequence of the rest of this License. + + 8. If the distribution and/or use of the Program is restricted in +certain countries either by patents or by copyrighted interfaces, the +original copyright holder who places the Program under this License +may add an explicit geographical distribution limitation excluding +those countries, so that distribution is permitted only in or among +countries not thus excluded. In such case, this License incorporates +the limitation as if written in the body of this License. + + 9. The Free Software Foundation may publish revised and/or new versions +of the General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies a version number of this License which applies to it and "any +later version", you have the option of following the terms and conditions +either of that version or of any later version published by the Free +Software Foundation. If the Program does not specify a version number of +this License, you may choose any version ever published by the Free Software +Foundation. + + 10. If you wish to incorporate parts of the Program into other free +programs whose distribution conditions are different, write to the author +to ask for permission. For software which is copyrighted by the Free +Software Foundation, write to the Free Software Foundation; we sometimes +make exceptions for this. Our decision will be guided by the two goals +of preserving the free status of all derivatives of our free software and +of promoting the sharing and reuse of software generally. + + NO WARRANTY + + 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY +FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN +OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES +PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED +OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS +TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE +PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, +REPAIR OR CORRECTION. + + 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR +REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING +OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED +TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY +YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER +PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGES. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +convey the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) 19yy + + This program is free software; you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 2 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program; if not, write to the Free Software + Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +Also add information on how to contact you by electronic and paper mail. + +If the program is interactive, make it output a short notice like this +when it starts in an interactive mode: + + Gnomovision version 69, Copyright (C) 19yy name of author + Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, the commands you use may +be called something other than `show w' and `show c'; they could even be +mouse-clicks or menu items--whatever suits your program. + +You should also get your employer (if you work as a programmer) or your +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. Here is a sample; alter the names: + + Yoyodyne, Inc., hereby disclaims all copyright interest in the program + `Gnomovision' (which makes passes at compilers) written by James Hacker. + + , 1 April 1989 + Ty Coon, President of Vice + +This General Public License does not permit incorporating your program into +proprietary programs. If your program is a subroutine library, you may +consider it more useful to permit linking proprietary applications with the +library. If this is what you want to do, use the GNU Library General +Public License instead of this License. diff --git a/DONATIONS b/DONATIONS new file mode 100644 index 0000000..9df2f09 --- /dev/null +++ b/DONATIONS @@ -0,0 +1,19 @@ +Help sponsor Squashfs development! + +Maintaining and improving Squashfs is a lot of work, but Squashfs is one of +the only widely used Linux file systems that has no company backing. Squashfs +development is funded soley by the author, partially supported by donations +from companies and individuals that want to improve Squashfs for themselves +and others. + +Mainlining of Squashfs only became possible when CELF (Consumer Electronics +Linux Forum) offered to contribute to the costs, which allowed the author +to work full-time on the project. + +There's lots of exciting new improvements to Squashfs in the pipeline, and +if your company is a serious user of Squashfs, please consider accelerating +development of Squashfs by donating. + +Donatations can be made from the Squashfs sourceforge homepage, or if you +prefer by contacting the author privately. + diff --git a/INSTALL b/INSTALL new file mode 100644 index 0000000..b0505aa --- /dev/null +++ b/INSTALL @@ -0,0 +1,28 @@ + INSTALLING SQUASHFS + +The squashfs4.2.tar.gz file contains the squashfs-tools directory containing +mksquashfs and unsquashfs. + +1. Kernel support +----------------- + +This release is for 2.6.29 and newer kernels. Kernel patching is not necessary. + +Extended attribute support requires 2.6.35 or newer. File systems with +extended attributes can be mounted on 2.6.29 and newer kernels (the +extended attributes will be ignored with a warning). + +LZO compression support requires 2.6.36 or newer kernels. + +XZ compression support requires 2.6.38 or newer kernels. + +2. Building squashfs tools +-------------------------- + +The squashfs-tools directory contains the mksquashfs and unsquashfs programs. +These can be made by typing make (or make install to install in /usr/local/bin). + +By default the tools are built with GZIP compression and extended attribute +support. Read the Makefile in squashfs-tools/ for instructions on building +LZO and XZ compression support, and for instructions on disabling GZIP +and extended attribute support if desired. diff --git a/OLD-READMEs/README-2.0 b/OLD-READMEs/README-2.0 new file mode 100644 index 0000000..41931d8 --- /dev/null +++ b/OLD-READMEs/README-2.0 @@ -0,0 +1,161 @@ +NOTE: This the original README for version 2.0. It is retained as it +contains information about the fragment design. A description of the new 2.0 +mksquashfs options has been added to the main README file, and that +file should now be consulted for these. + + SQUASHFS 2.0 - A squashed read-only filesystem for Linux + + Copyright 2004 Phillip Lougher (plougher@users.sourceforge.net) + + Released under the GPL licence (version 2 or later). + +Welcome to the final release of Squashfs version 2.0! A lot of changes to the +filesystem have been made under the bonnet (hood). Squashfs 2.0 uses fragment +blocks and larger blocks (64K) to improve compression ratio by about 5 - 20% +over Squashfs 1.0 depending on the files being compressed. Using fragment +blocks allows Squashfs 2.0 to achieve better compression than cloop and similar +compression to tgz files while retaining the I/O efficiency of a compressed +filesystem. + +Detailed changes: + +1. Squashfs 2.0 has added the concept of fragment blocks (see later discussion). + Files smaller than the file block size (64K in Squashfs 2.0) and optionally + the remainder of files that do not fit fully into a block (i.e. the last 32K + in a 96K file) are packed into shared fragments and compressed together. + This achieves on average 5 - 20% better compression than Squashfs 1.x. + +2. The maximum block size has been increased to 64K. + +3. The maximum number of UIDs has been increased to 256 (from 48 in 1.x). + +4. The maximum number of GIDs has been increased to 256 (from 15 in 1.x). + +5. New mksquashfs -all-root, -root-owned, -force-uid, and -force-gid + options. These allow the uids/gids of files in the generated + filesystem to be specified, overriding the uids/gids in the + source filesystem. + +6. Initrds are now supported for kernels 2.6.x. + +7. Removal of sleep_on() function call in 2.6.x patch, to allow Squashfs + to work on the Fedora rc2 kernel. + +8. AMD64, check-data and gid bug fixes. + +9. Numerous small bug fixes have been made. + +10. New patch for Linux 2.6.7. + + +New Squashfs 2.0 options +------------------------ + +-noF or -noFragmentCompression + + Do not compress the fragments. Added for compatibility with noI and + noD, probably not that useful. + +-no-fragments + + Do not use fragment blocks, and rather generate a filesystem + similar to a Squashfs 1.x filesystem. It will of course still + be a Squashfs 2.0 filesystem but without fragments, and so + it won't be mountable on a Squashfs 1.x system. + +-always-use-fragments + + By default only small files less than the block size are packed into + fragment blocks. The ends of files which do not fit fully into a block, + are NOT by default packed into fragments. To illustrate this, a + 100K file has an initial 64K block and a 36K remainder. This + 36K remainder is not packed into a fragment by default. This is + because to do so leads to a 10 - 20% drop in sequential I/O + performance, as a disk head seek is needed to seek to the initial + file data and another disk seek is need to seek to the fragment + block. + + Specify this option if you want file remainders to be packed into + fragment blocks. Doing so may increase the compression obtained + BUT at the expense of I/O speed. + +-no-duplicates + + Do not detect duplicate files. + +-all-root +-root-owned + + These options (both do exactly the same thing), force all file + uids/gids in the generated Squashfs filesystem to be root. + This allows root owned filesystems to be built without root access + on the host machine. + +-force-uid uid + + This option forces all files in the generated Squashfs filesystem to + be owned by the specified uid. The uid can be specified either by + name (i.e. "root") or by number. + +-force-gid gid + + This option forces all files in the generated Squashfs filesystem to + be group owned by the specified gid. The gid can be specified either by + name (i.e. "root") or by number. + + +Compression improvements example +-------------------------------- + +The following is the compression results obtained compressing the 2.6.6 +linux kernel source using CRAMFS, Cloop (with iso filesystem), Squashfs 1.3 and +Squashfs 2.0 (results generated using big-endian filesystems). + +In decreasing order of size: + + CRAMFS 62791680 bytes (59.9M) + Squashfs 1.x 51351552 bytes (48.9M) + Cloop 46118681 bytes (44.0M) + Squashfs 2.0 45604854 bytes (43.5M) + + +The Squashfs 1.x filesystem is 12.6% larger than the new 2.0 filesystem. +The cloop filesystem is 1.1% larger than the Squashfs 2.0 filesystem. + + +Fragment blocks in Squashfs 2.0 +------------------------------- + +Squashfs like all other compressed filesystems compresses files individually +on a block by block basis. This is performed to allow mounting and +de-compression of files on a block by block basis without requiring the entire +filesystem to be decompressed. This is in contrast to data-based compression +schemes which compress without understanding the underlying filesystem (i.e. +cloop and tgz files) and which, therefore, do not compress files individually. +Each approach has advantages and disadvantages, data-based systems have better +compression because compression is always performed at the maximum block size +(64K in cloop) irrespective of the size of each file (which could be less than +the block size). Compressed filesystems tend to be faster at I/O because +they understand the filesystem and therefore employ better caching stategies +and read less un-needed data from the filesystem. + +Fragment blocks in Squashfs 2.0 solves this problem by packing files (and +optionally the ends of files) which are smaller than the block size into +shared blocks, which are compressed together. For example five files each of +10K will be packed into one shared fragment of 50K and compressed together, +rather than being compressed in five 10K blocks. + +This scheme produces a hybrid filesystem, retaining the I/O efficiency +of a compressed filesystem, while obtaining the compression efficiency +of data-based schemes by compressing small files together. + + +Squashfs 1.x and Squashfs 2.0 compatibility +------------------------------------------- + +Appending to Squashfs 1.x filesystems is not supported. If you wish to append +to 1.x filesystems, then either use the original mksquashfs, or convert them +to Squashfs 2.0 by mounting the filesystem and running the 2.0 mksquashfs +on the mounted filesystem. + +Mounting Squashfs 1.x filesystems IS supported by the 2.0 kernel patch. diff --git a/OLD-READMEs/README-2.1 b/OLD-READMEs/README-2.1 new file mode 100644 index 0000000..e70167e --- /dev/null +++ b/OLD-READMEs/README-2.1 @@ -0,0 +1,87 @@ + SQUASHFS 2.1 - A squashed read-only filesystem for Linux + + Copyright 2004 Phillip Lougher (plougher@users.sourceforge.net) + + Released under the GPL licence (version 2 or later). + +Welcome to Squashfs version 2.1-r2. Squashfs 2.1 introduces indexed +directories which considerably speed up directory lookup (ls, find etc.) for +directories which are greater than 8K in size. All directories are now also +sorted alphabetically which further speeds up directory lookup. Many smaller +improvements have also been made to this release, please see the CHANGES file +entry for detailed changes. + +1. DIRECTORY SPEED IMPROVEMENT EXAMPLES +--------------------------------------- + +To give an indication of the directory speed improvements a number of test +results are shown here. There is in addition a new PERFORMANCE.README file +which gives details of I/O and lookup performance for Squashfs 2.1 against +the Zisofs, Cloop and CRAMFS filesystems. + +example 1: + +Filesystems generated from a single directory of 72,784 files (2.6 MB +directory size). Each file is 10 bytes in size (the test is directory +lookup and so the file size isn't an issue). The ext3 uncompressed +directory size is 288 MB (presumably because of one file per block). + +Zisofs compressed size 153.50 MB +Cloop (isofs) compressed size 1.74 MB +Squashfs2.1 compressed size 612 KB (0.60 MB) + +Time taken to perform "ls -lR --color=always | cat > /dev/null" on +filesystems mounted on hard disk. + +Zisofs 35 minutes 7.895 seconds (User 7.868 secs, Sys 34 mins 5.621 secs) +Cloop 35 minutes 12.765 seconds (User 7.771 secs, Sys 34 mins 3.869 secs) +Squashfs2.1 19 seconds (User 5.119 secs, Sys 14.547 secs) + +example 2: + +Filesystems were generated from the Ubuntu Warty livecd (original uncompressed +size on ext3 is 1.4 GB). + +Zisofs compressed size 589.81 MB +Cloop (isofs) compressed size 471.19 MB +Squashfs2.0 compressed size 448.58 MB +Squashfs2.1 compressed size 448.58 MB + +Time taken to perform "ls -lR --color=always | cat > /dev/null" on +filesystems mounted on hard disk. + +Zisofs 49.875 seconds (User time 2.589 secs, Sys 11.194 secs) +Cloop 20.797 seconds (User time 2.706 secs, Sys 13.496 secs) +Squashfs2.0 16.556 seconds (User time 2.424 secs, Sys 10.371 secs) +Squashfs2.1 10.143 seconds (User time 2.475 secs, Sys 4.440 secs) + + +NOTE: the usual warnings apply to these results, they are provided for +illustrative purposes only, and due to different hardware and/or file data, you +may obtain different results. As such the results are provided "as is" without +any warranty (either express or implied) and you assume all risks as to their +quality and accuracy. + +2. NEW MKSQUASHFS OPTIONS +------------------------- + +There is only one extra option "-2.0". This tells mksquashfs to generate +a filesystem which is mountable with Squashfs version 2.0. + +3. APPENDING AND MOUNTING SQUASHFS 2.0 FILESYSTEMS +-------------------------------------------------- + +Mounting 2.0 filesystems is supported by Squashfs 2.1. In addition +mksquashfs v2.1 can append to 2.0 filesystems, although the generated +filesystem will still be a 2.0 filesystem. + +4. DONATIONS +------------ + +If you find Squashfs useful then please consider making a donation, +particularly if you use Squashfs in a commercial product. Please consider +giving something back especially if you're making money from it. + +Off the Squashfs subject somewhat I'm currently looking for another +job doing Linux kernel or filesystems work. If you know of any such +work that can be performed from the UK then please get in touch. Thanks. diff --git a/OLD-READMEs/README-3.0 b/OLD-READMEs/README-3.0 new file mode 100644 index 0000000..fd16dd3 --- /dev/null +++ b/OLD-READMEs/README-3.0 @@ -0,0 +1,60 @@ + SQUASHFS 3.0 - A squashed read-only filesystem for Linux + + Copyright 2002-2006 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to the first release of Squashfs version 3.0. Squashfs 3.0 has the +the following improvements to 2.x. + + 1. Filesystems are no longer limited to 4 GB. In + theory 2^64 or 4 exabytes is now supported. + + 2. Files are no longer limited to 4 GB. In theory the maximum + file size is 4 exabytes. + + 3. Metadata (inode table and directory tables) are no longer + restricted to 16 Mbytes. + + 4. Hardlinks are now suppported. + + 5. Nlink counts are now supported. + + 6. Readdir now returns '.' and '..' entries. + + 7. Special support for files larger than 256 MB has been added to + the Squashfs kernel code for faster read access. + + 8. Inode numbers are now stored within the inode rather than being + computed from inode location on disk (this is not so much an + improvement, but a change forced by the previously listed + improvements). + +There is a new Unsquashfs utility (in squashfs-tools) than can be used to +decompress a filesystem without mounting it. + +Squashfs 3.0 supports 2.x filesystems. Support for 1.x filesystems +will be added in the future. + +1. UNSQUASHFS +------------- + +Unsquashfs has the following options: + +SYNTAX: unsquashfs [-ls | -dest] filesystem + -version print version, licence and copyright information + -info print files as they are unsquashed + -ls list filesystem only + -dest unsquash to , default "squashfs-root" + +The "-ls" option can be used to list the contents of a filesystem without +decompressing the filesystem data itself. + +The "-info" option forces Unsquashfs to print each file as it is decompressed. + +The "-dest" option specifies the directory that is used to decompress +the filesystem data. If this option is not given then the filesystem is +decompressed to the directory "squashfs-root" in the current working directory. + +Unsquashfs can decompress 3.0 filesystems. Support for 2.x and 1.x +filesystems will be added in the future. diff --git a/OLD-READMEs/README-3.1 b/OLD-READMEs/README-3.1 new file mode 100644 index 0000000..0e1ee79 --- /dev/null +++ b/OLD-READMEs/README-3.1 @@ -0,0 +1,158 @@ + SQUASHFS 3.1 - A squashed read-only filesystem for Linux + + Copyright 2002-2006 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to Squashfs version 3.1-r2. Squashfs 3.1 has major improvements to +the Squashfs tools (Mksquashfs and Unsquashfs), some major bug fixes, new +kernel patches, and various other smaller improvements and bug fixes. +Please see the CHANGES file for a detailed list. + +1. MKSQUASHFS +------------- + +Mksquashfs has been rewritten and it is now multi-threaded. It offers +the following improvements: + +1. Parallel compression. By default as many compression and fragment +compression threads are created as there are available processors. +This significantly speeds up performance on SMP systems. + +2. File input and filesystem output is peformed in parallel on separate +threads to maximise I/O performance. Even on single processor systems +this speeds up performance by at least 10%. + +3. Appending has been significantly improved, and files within the +filesystem being appended to are no longer scanned and checksummed. This +significantly improves append time for large filesystems. + +4. File duplicate checking has been optimised, and split into two separate +phases. Only files which are considered possible duplicates after the +first phase are checksummed and cached in memory. + +5. The use of swap memory was found to significantly impact performance. The +amount of memory used to cache the file is now a command line option, by default +this is 512 Mbytes. + +1.1 NEW COMMAND LINE OPTIONS +---------------------------- + +The new Mksquashfs program has a couple of extra command line options +which can be used to control the new features: + +-processors + +This specifies the number of processors used by Mksquashfs. +By default this is the number of available processors. + +-read_queue + +This specifies the size of the file input queue used by the reader thread. +This defaults to 64 Mbytes. + +-write_queue + +This specifies the size of the filesystem output queue used by the +writer thread. It also specifies the maximum cache used in file +duplicate detection (the output queue is shared between these tasks). +This defaults to 512 Mbytes. + +1.2 PERFORMANCE RESULTS +----------------------- + +The following results give an indication of the speed improvements. Two +example filesystems were tested, a liveCD filesystem (about 1.8 Gbytes +uncompressed), and my home directory consisting largely of text files +(about 1.3 Gbytes uncompressed). Tests were run on a single core +and a dual core system. + +Dual Core (AMDx2 3800+) system: +Source directories on ext3. + +LiveCD, old mksquashfs: + +real 11m48.401s +user 9m27.056s +sys 0m15.281s + +LiveCD, new par_mksquashfs: + +real 4m8.736s +user 7m11.771s +sys 0m27.749s + +"Home", old mksquashfs: + +real 4m34.360s +user 3m54.007s +sys 0m32.155s + +"Home", new par_mksquashfs: + +real 1m27.381s +user 2m7.304s +sys 0m17.234s + +Single Core PowerBook (PowerPC G4 1.5 GHz Ubuntu Linux) +Source directories on ext3. + +LiveCD, old mksquashs: + +real 11m38.472s +user 9m6.137s +sys 0m23.799s + +LiveCD, par_mksquashfs: + +real 10m5.572s +user 8m59.921s +sys 0m16.145s + +"Home", old mksquashfs: + +real 3m42.298s +user 2m49.478s +sys 0m13.675s + +"Home", new par_mksquashfs: + +real 3m9.178s +user 2m50.699s +sys 0m9.069s + +I'll be interested in any performance results obtained, especially from SMP +machines larger than my dual-core AMD box, as this will give an indication of +the scalability of the code. Obviously, I'm also interested in any problems, +deadlocks, low performance etc. + +2. UNSQUASHFS +------------- + +Unsquashfs now allows you to specify the filename or directory that is to be +extracted from the Squashfs filesystem, rather than always extracting the +entire filesystem. It also has a new "-force" option, and all options can be +specified in a short form (-i rather than -info). + +The Unsquashfs usage info is now: + +SYNTAX: ./unsquashfs [options] filesystem [directory or file to extract] + -v[ersion] print version, licence and copyright information + -i[nfo] print files as they are unsquashed + -l[s] list filesystem only + -d[est] unsquash to , default "squashfs-root" + -f[orce] if file already exists then overwrite + +To extract a subset of the filesystem, the filename or directory +tree that is to be extracted can now be specified on the command line. The +file/directory should be specified using the full path to the file/directory +as it appears within the Squashfs filesystem. The file/directory will also be +extracted to that position within the specified destination directory. + +The new "-force" option forces Unsquashfs to output to the destination +directory even if files or directories already exist. This allows you +to update an existing directory tree, or to Unsquashfs to a partially +filled directory. Without the "-force" option, Unsquashfs will +refuse to overwrite any existing files, or to create any directories if they +already exist. This is done to protect data in case of mistakes, and +so the "-force" option should be used with caution. diff --git a/OLD-READMEs/README-3.2 b/OLD-READMEs/README-3.2 new file mode 100644 index 0000000..e38286f --- /dev/null +++ b/OLD-READMEs/README-3.2 @@ -0,0 +1,33 @@ + SQUASHFS 3.2 - A squashed read-only filesystem for Linux + + Copyright 2002-2007 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to Squashfs version 3.2. Squashfs 3.2 has support for NFS exporting, +some improvements to the Squashfs tools (Mksquashfs and Unsquashfs), some +major bug fixes, new kernel patches, and various other smaller improvements +and bug fixes. Please see the CHANGES file for a detailed list. + +1. MKSQUASHFS +------------- + +New command line options: + +-no-exports + + Squashfs now supports NFS exports. By default the additional + information necessary is added to the filesystem by Mksquashfs. If you + do not wish this extra information, then this option can be specified. + This will save a couple of bytes per file, and the filesystem + will be identical to Squashfs 3.1. + +-no-progress + + Mksquashfs by default now displays a progress bar. This option disables + it. + +2. UNSQUASHFS +------------- + + Unsquashfs now supports Squashfs 2.x filesystems. diff --git a/OLD-READMEs/README-3.3 b/OLD-READMEs/README-3.3 new file mode 100644 index 0000000..a38a39e --- /dev/null +++ b/OLD-READMEs/README-3.3 @@ -0,0 +1,169 @@ + SQUASHFS 3.3 - A squashed read-only filesystem for Linux + + Copyright 2002-2007 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to another release of Squashfs. This is the 22nd release in just +over five years of work. Squashfs 3.3 has lots of nice improvements, +both to the filesystem itself (bigger blocks, and sparse files), but +also to the Squashfs-tools Mksquashfs and Unsquashfs. As usual the +CHANGES file has a detailed list of all the improvements. + +Following is a description of the changes to the Squashfs tools, usage +guides to the new options, and a summary of the new options. + +1. MKSQUASHFS - EXTENDED EXCLUDE FILE HANDLING +---------------------------------------------- + +1. Extended wildcard pattern matching now supported in exclude files + + Enabled by specifying -wildcards option + + Supports both anchored and non-anchored exclude files. + +1.1 Anchored excludes + + Similar to existing exclude files except with wildcards. Exclude + file matches from root of source directories. + + Examples: + + 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz' + + Exclude all files matching "*.gz" in the top level directory "test". + + 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*' + + Exclude all files beginning with "example" inside directories called + "Test" or "test", that occur inside any top level directory. + + Using extended wildcards, negative matching is also possible. + + 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz' + + Exclude all files matching "*.gz" in top level directory "test", + except those with "data" in the name. + +1.2 Non-anchored excludes + + By default excludes match from the top level directory, but it is + often useful to exclude a file matching anywhere in the source directories. + For this non-anchored excludes can be used, specified by pre-fixing the + exclude with "...". + + Examples: + + 1. mksquashfs example image.sqsh -wildcards -e '... *.gz' + + Exclude files matching "*.gz" anywhere in the source directories. + For example this will match "example.gz", "test/example.gz", and + "test/test/example.gz". + + 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz' + + Exclude files matching "*.gz" inside directories called "Test" or + "test" that occur anywhere in the source directories. + + Again, using extended wildcards, negative matching is also possible. + + 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz' + + Exclude all files matching "*.gz" anywhere in the source directories, + except those with "data" in the name. + +2. Regular expression pattern matching now supported in exclude files + + Enabled by specifying -regex option. Identical behaviour to wild +card pattern matching, except patterns are considered to be regular +expressions. + + Supports both anchored and non-anchored exclude files. + + +2. MKSQUASHFS - NEW RECOVERY FILE FEATURE +----------------------------------------- + +Recovery files are now created when appending to existing Squashfs +filesystems. This allows the original filesystem to be recovered +if Mksquashfs aborts unexpectedly (i.e. power failure). + +The recovery files are called squashfs_recovery_xxx_yyy, where +"xxx" is the name of the filesystem being appended to, and "yyy" is a +number to guarantee filename uniqueness (the PID of the parent Mksquashfs +process). + +Normally if Mksquashfs exits correctly the recovery file is deleted to +avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover" +option can be used to recover the filesystem, giving the previously +created recovery file as a parameter, i.e. + +mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234 + +The writing of the recovery file can be disabled by specifying the +"-no-recovery" option. + + +3. UNSQUASHFS - EXTENDED EXTRACT FILE HANDLING +---------------------------------------------- + +1. Multiple extract files can now be specified on the command line, and the +files/directories to be extracted can now also be given in a file. + +To specify a file containing the extract files use the "-e[f]" option. + +2. Extended wildcard pattern matching now supported in extract files + + Enabled by default. Similar to existing extract files except with +wildcards. + + Examples: + + 1. unsquashfs image.sqsh 'test/*.gz' + + Extract all files matching "*.gz" in the top level directory "test". + + 2. unsquashfs image.sqsh '[Tt]est/example*' + + Extract all files beginning with "example" inside top level directories + called "Test" or "test". + + Using extended wildcards, negative matching is also possible. + + 3. unsquashfs image.sqsh 'test/!(*data*).gz' + + Extract all files matching "*.gz" in top level directory "test", + except those with "data" in the name. + +3. Regular expression pattern matching now supported in extract files + + Enabled by specifying -r[egex] option. Identical behaviour to wild +card pattern matching, except patterns are considered to be regular +expressions. + +4. UNSQUASHFS - EXTENDED FILENAME PRINTING +------------------------------------------ + +Filename printing has been enhanced and Unquashfs can now display filenames +with file attributes ('ls -l' style output). + +New options: + + -ll[s] + + list filesystem with file attributes, but don't unsquash + + -li[nfo] + + print files as they are unsquashed with file attributes + + +5. UNSQUASHFS - MISCELLANEOUS OPTIONS +------------------------------------- + + -s[tat] + + Display the filesystem superblock information. This is useful to + discover the filesystem version, byte ordering, whether it has an + NFS export table, and what options were used to compress + the filesystem. diff --git a/OLD-READMEs/README-4.0 b/OLD-READMEs/README-4.0 new file mode 100644 index 0000000..8cc9514 --- /dev/null +++ b/OLD-READMEs/README-4.0 @@ -0,0 +1,48 @@ + SQUASHFS 4.0 - A squashed read-only filesystem for Linux + + Copyright 2002-2009 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to Squashfs 4.0. This is an initial tools only release to +support users of the 2.6.29 kernel, following the mainlining of Squashfs +earlier this year. + +Later releases will probably contain kernel patches supporting 4.0 +layouts for earlier kernels. + +New Mksquashfs options +---------------------- + +Mksquashfs now supports pseudo files, these allow fake directories, character +and block devices to be specified and added to the Squashfs filesystem being +built, rather than requiring them to be present in the source directories. +This, for example, allows device nodes to be added to the filesystem without +requiring root access. + +Two options are supported, -p allows one pseudo file to be specified on the +command line, and -pf allows a pseudo file to be specified containing a +list of pseduo definitions, one per line. + +Pseudo device nodes are specified using 7 arguments + +Filename type mode uid gid major minor + +Where type is either + b - for block devices, and + c - for character devices + +mode is the octal mode specifier, similar to that expected by chmod. + +Uid and gid can be either specified as a decimal number, or by name. + +For example: + +/dev/chr_dev c 666 root root 100 1 +/dev/blk_dev b 444 0 0 200 200 + +Directories are specified using 5 arguments + +Filename type mode uid gid + +Where type is d. diff --git a/OLD-READMEs/README-4.1 b/OLD-READMEs/README-4.1 new file mode 100644 index 0000000..d2712f9 --- /dev/null +++ b/OLD-READMEs/README-4.1 @@ -0,0 +1,265 @@ + SQUASHFS 4.1 - A squashed read-only filesystem for Linux + + Copyright 2002-2010 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to Squashfs 4.1. This is a tools only release, support for Squashfs +file systems is in mainline (2.6.29 and later). + +New features in Squashfs-tools 4.1 +---------------------------------- + + 1. Support for extended attributes + 2. Support for LZMA and LZO compression + 3. New pseudo file features + +Compatiblity +------------ + +Mksquashfs 4.1 generates 4.0 filesystems. These filesystems are fully +compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are +mountable on 2.6.29 and later kernels. + +Extended attributes (xattrs) +---------------------------- + +Squashfs file systems now have extended attribute support. The +extended attribute implementation has the following features: + +1. Layout can store up to 2^48 bytes of compressed xattr data. +2. Number of xattrs per inode unlimited. +3. Total size of xattr data per inode 2^48 bytes of compressed data. +4. Up to 4 Gbytes of data per xattr value. +5. Inline and out-of-line xattr values supported for higher performance + in xattr scanning (listxattr & getxattr), and to allow xattr value + de-duplication. +6. Both whole inode xattr duplicate detection and individual xattr value + duplicate detection supported. These can obviously nest, file C's + xattrs can be a complete duplicate of file B, and file B's xattrs + can be a partial duplicate of file A. +7. Xattr name prefix types stored, allowing the redundant "user.", "trusted." + etc. characters to be eliminated and more concisely stored. +8. Support for files, directories, symbolic links, device nodes, fifos + and sockets. + +Extended attribute support is in 2.6.35 and later kernels. File systems +with extended attributes can be mounted on 2.6.29 and later kernels, the +extended attributes will be ignored with a warning. + +LZMA and LZO compression +------------------------ + +Squashfs now supports LZMA and LZO compression. + +LZO support is in 2.6.36 and newer kernels. LZMA is not yet in mainline. + +New Mksquashfs options +---------------------- + +-comp + + Select compression. + + The compression algorithms supported by the build of Mksquashfs can be + found by typing mksquashfs without any arguments. The compressors available + are displayed at the end of the help message, e.g. + + Compressors available: + gzip (default) + lzma + lzo + + The default compression used when -comp isn't specified on the command line + is indicated by "(default)". + +-no-xattrs + Don't store extended attributes + +-xattrs + Store extended attributes + + The default behaviour of Mksquashfs with respect to extended attribute + storage is build time selectable. The Mksquashfs help message indicates + whether extended attributes are stored or not, e.g. + + -no-xattrs don't store extended attributes + -xattrs store extended attributes (default) + + shows that extended attributes are stored by default, and can be disabled + by the -no-xattrs option. + + -no-xattrs don't store extended attributes (default) + -xattrs store extended attributes + + shows that extended attributes are not stored by default, storage can be + enabled by the -xattrs option. + + +-noX +-noXattrCompression + Don't compress extended attributes + + +New Unsquashfs options +---------------------- + +-n[o-xattrs] + Don't extract xattrs in filesystem + +-x[attrs] + Extract xattrs in filesystem + + The default behaviour of Unsquashfs with respect to extended attributes + is build time selectable. The Unsquashfs help message indicates whether + extended attributes are stored or not, e.g. + + -no[-xattrs] don't extract xattrs in file system + -x[attrs] extract xattrs in file system (default) + + shows that xattrs are extracted by default. + + -no[-xattrs] don't extract xattrs in file system (default) + -x[attrs] extract xattrs in file system + + shows that xattrs are not extracted by default. + + +New pseudo file support +----------------------- + +Mksquashfs supports pseudo files, these allow fake files, directories, character +and block devices to be specified and added to the Squashfs filesystem being +built, rather than requiring them to be present in the source directories. +This, for example, allows device nodes to be added to the filesystem without +requiring root access. + +Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation. +Dynamic pseudo files allow files to be dynamically created when Mksquashfs +is run, their contents being the result of running a command or piece of +shell script. The modifiy operation allows the mode/uid/gid of an existing +file in the source filesystem to be modified. + +Two Mksquashfs options are supported, -p allows one pseudo file to be specified +on the command line, and -pf allows a pseudo file to be specified containing a +list of pseduo definitions, one per line. + +Pseudo operations +----------------- + +1. Creating a dynamic file +-------------------------- + +Pseudo definition + +Filename f mode uid gid command + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +command can be an executable or a piece of shell script, and it is executed +by running "/bin/sh -c command". The stdout becomes the contents of +"Filename". + +Examples: + +Running a basic command +----------------------- + +/somedir/dmesg f 444 root root dmesg + +creates a file "/somedir/dmesg" containing the output from dmesg. + +Executing shell script +---------------------- + +RELEASE f 444 root root \ + if [ ! -e /tmp/ver ]; then \ + echo 0 > /tmp/ver; \ + fi; \ + ver=`cat /tmp/ver`; \ + ver=$((ver +1)); \ + echo $ver > /tmp/ver; \ + echo -n `cat /tmp/release`; \ + echo "-dev #"$ver `date` "Build host" `hostname` + +Creates a file RELEASE containing the release name, date, build host, and +an incrementing version number. The incrementing version is a side-effect +of executing the shell script, and ensures every time Mksquashfs is run a +new version number is used without requiring any other shell scripting. + +The above example also shows that commands can be split across multiple lines +using "\". Obviously as the script will be presented to the shell as a single +line, a semicolon is need to separate individual shell commands within the +shell script. + +Reading from a device (or fifo/named socket) +-------------------------------------------- + +input f 444 root root dd if=/dev/sda1 bs=1024 count=10 + +Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs +given a device, fifo, or named socket will place that special file within the +Squashfs filesystem, the above allows input from these special files to be +captured and placed in the Squashfs filesystem. + +2. Creating a block or character device +--------------------------------------- + +Pseudo definition + +Filename type mode uid gid major minor + +Where type is either + b - for block devices, and + c - for character devices + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +For example: + +/dev/chr_dev c 666 root root 100 1 +/dev/blk_dev b 666 0 0 200 200 + +creates a character device "/dev/chr_dev" with major:minor 100:1 and +a block device "/dev/blk_dev" with major:minor 200:200, both with root +uid/gid and a mode of rw-rw-rw. + +3. Creating a directory +----------------------- + +Pseudo definition + +Filename d mode uid gid + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +For example: + +/pseudo_dir d 666 root root + +creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw. + +4. Modifying attributes of an existing file +------------------------------------------- + +Pseudo definition + +Filename m mode uid gid + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +For example: + +dmesg m 666 root root + +Changes the attributes of the file "dmesg" in the filesystem to have +root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained +from the source filesystem. diff --git a/OLD-READMEs/README-AMD64 b/OLD-READMEs/README-AMD64 new file mode 100644 index 0000000..c32af85 --- /dev/null +++ b/OLD-READMEs/README-AMD64 @@ -0,0 +1,18 @@ +Information for amd64 users +--------------------------- + +All releases of Squashfs prior to 2.0 generate incorrect +filesystems on amd64 machines. Filesystems created on amd64 machines work +correctly on amd64 machines, but cannot be mounted on non-amd64 machines. +Likewise, filesystems created on non-amd64 machines cannot be mounted on amd64 +machines. This bug is caused by the different size of the "time_t" definition +used in SquashFS filesystem structures. + +This bug is fixed in releases 2.0 and newer. However, all amd64 filesystems +generated by previous releases will not be mountable on amd64 machines +with newer releases. If you have amd64 filesystems generated with mksquashfs +version 2.0-alpha or older, it is important that you recreate the filesystem. +This can be performed by mounting the filesystem using a kernel with the +original patch (i.e. a 2.0-alpha or older patch) and running the NEW (i.e. this +release) mksquashfs tool to create a new SquashFS filesystem. + diff --git a/PERFORMANCE.README b/PERFORMANCE.README new file mode 100644 index 0000000..efb98f2 --- /dev/null +++ b/PERFORMANCE.README @@ -0,0 +1,171 @@ +GENERAL INFORMATION ON PERFORMANCE TESTS +---------------------------------------- + +The following performance tests were based on two file sets: the +liveCD filesystem from the Ubuntu liveCD (Warty release), and the +liveCD filesystem from the Damn Small Linux liveCD (release 0.8.4). +The Ubuntu liveCD filesystem was used to test filesystem performance +from CDROM and hard disk for Zisofs, Cloop, Squashfs 2.0 and Squashfs2.1. +CRAMFS filesystem performance could not be tested for this filesystem +bacause it exceeds the maximum supported size of CRAMFS. To test +CRAMFS performance against Squashfs, the liveCD filesystem from +Damn Small Linux was used. + +NOTE: the usual warnings apply to these results, they are provided for +illustrative purposes only, and due to different hardware and/or file data, you +may obtain different results. As such the results are provided "as is" without +any warranty (either express or implied) and you assume all risks as to their +quality and accuracy. + +1. Ubuntu liveCD performance tests + + ext3 uncompressed size 1.4 GB + Zisofs compressed size 589.81 MB + Cloop compressed size 471.89 MB + Squashfs2.0 compressed size 448.58 MB + Squashfs2.1 compressed size 448.58 MB + +1.1 Performance tests from CDROM + +1.1.1 Directory Lookup performance + + Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem + mounted from CDROM + + Zisofs 49.88 seconds (User 2.60 secs, Sys 11.19 secs) + Cloop 20.80 seconds (User 2.71 secs, Sys 13.50 secs) + Squashfs2.0 16.56 seconds (User 2.42 secs, Sys 10.37 secs) + Squashfs2.1 10.14 seconds (User 2.48 secs, Sys 4.44 secs) + +1.1.2 Sequential I/O performance + + Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted + from CDROM + + Zisofs 27 minutes 28.54 seconds (User 3.00 secs, Sys 1 min 4.80 secs) + Cloop 5 minutes 55.72 seconds (User 2.90 secs, Sys 3 min 37.90 secs) + Squashfs2.0 5 minutes 20.87 seconds (User 2.33 secs, Sys 56.98 secs) + Squashfs2.1 5 minutes 15.46 seconds (user 2.28 secs, Sys 51.12 secs) + +1.1.3 Random I/O performance + + Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort + -g | awk '{ printf $2 }' > /tmp/sort + + Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" + on filesystem mounted from CDROM + + Zisofs 101 minutes 29.65 seconds (User 5.33 secs, Sys 1 min 17.20 secs) + Cloop 35 minutes 27.51 seconds (user 5.93 secs, Sys 4 mins 30.23 secs) + Squashfs2.0 21 minutes 53.05 seconds (user 5.71 secs, Sys 2 mins 36.59 secs) + Squashfs2.1 21 minutes 46.99 seconds (User 5.80 secs, Sys 2 mins 31.88 secs) + + +1.2 Performance tests from Hard disk + +1.2.1 Directory Lookup performance + + Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem + mounted from Hard disk + + Zisofs 17.29 seconds (User 2.62 secs, Sys 11.08 secs) + Cloop 16.46 seconds (User 2.63 secs, Sys 13.41 secs) + Squashfs2.0 13.75 seconds (User 2.44 secs, Sys 11.00 secs) + Squashfs2.1 6.94 seconds (User 2.44 secs, Sys 4.48 secs) + +1.2.2 Sequential I/O performance + + Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted + from Hard disk + + Zisofs 1 minute 21.47 seconds (User 2.73 secs, Sys 54.44 secs) + Cloop 1 minute 34.06 seconds (user 2.85 secs, Sys 1 min 12.13 secs) + Squashfs2.0 1 minute 21.22 seconds (User 2.42 secs, Sys 56.21 secs) + Squashfs2.1 1 minute 15.46 seconds (User 2.36 secs, Sys 49.78 secs) + +1.2.3 Random I/O performance + + Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort + -g | awk '{ printf $2 }' > /tmp/sort + + Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" + on filesystem mounted from Hard disk + + Zisofs 11 minutes 13.64 seconds (User 5.08 secs, Sys 52.62 secs) + Cloop 5 minutes 37.93 seconds (user 6 secs, Sys 2 mins 22.38 secs) + Squashfs2.0 5 minutes 7.11 seconds (user 5.63 secs, Sys 2 mins 35.23 secs) + Squashfs2.1 5 minutes 1.87 seconds (User 5.71 secs, Sys 2 mins 29.98 secs) + + +2. Damn Small Linux liveCD performance tests + + ext3 uncompressed size 126 MB + CRAMFS compressed size 52.19 MB + Squashfs2.0 compressed size 46.52 MB + Squashfs2.1 compressed size 46.52 MB + +2.1 Performance tests from CDROM + +2.1.1 Directory Lookup performance + + Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem + mounted from CDROM + + CRAMFS 10.85 seconds (User 0.39 secs, Sys 0.98 secs) + Squashfs2.0 2.97 seconds (User 0.36 secs, Sys 2.15 secs) + Squashfs2.1 2.43 seconds (User 0.40 secs, Sys 1.42 secs) + +2.1.2 Sequential I/O performance + + Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted + from CDROM + + CRAMFS 55.38 seconds (User 0.34 secs, Sys 6.98 secs) + Squashfs2.0 35.99 seconds (User 0.30 secs, Sys 6.35 secs) + Squashfs2.1 33.83 seconds (User 0.26 secs, Sys 5.56 secs) + +2.1.3 Random I/O performance + + Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort + -g | awk '{ printf $2 }' > /tmp/sort + + Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" + on filesystem mounted from CDROM + + + CRAMFS 3 minutes 1.68 seconds (User 0.54 secs, Sys 9.51 secs) + Squashfs2.0 1 minute 39.45 seconds (User 0.57 secs, Sys 13.14 secs) + Squashfs2.1 1 minute 38.41 seconds (User 0.58 secs, Sys 13.08 secs) + +2.2 Performance tests from Hard disk + +2.2.1 Directory Lookup performance + + Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem + mounted from Hard disk + + CRAMFS 1.77 seconds (User 0.53 secs, Sys 1.21 secs) + Squashfs2.0 2.67 seconds (User 0.41 secs, Sys 2.25 secs) + Squashfs2.1 1.87 seconds (User 0.41 secs, Sys 1.46 secs) + +2.2.2 Sequential I/O performance + + Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted + from Hard disk + + CRAMFS 6.80 seconds (User 0.36 secs, Sys 6.02 secs) + Squashfs2.0 7.23 seconds (User 0.29 secs, Sys 6.62 secs) + Squashfs2.1 6.53 seconds (User 0.31 secs, Sys 5.82 secs) + +2.2.3 Random I/O performance + + Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort + -g | awk '{ printf $2 }' > /tmp/sort + + Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" + on filesystem mounted from Hard disk + + + CRAMFS 28.55 seconds (User 0.49 secs, Sys 6.49 secs) + Squashfs2.0 25.44 seconds (User 0.58 secs, Sys 13.17 secs) + Squashfs2.1 24.72 seconds (User 0.56 secs, Sys 13.15 secs) diff --git a/README b/README new file mode 100644 index 0000000..351f61d --- /dev/null +++ b/README @@ -0,0 +1,937 @@ + SQUASHFS 4.2 - A squashed read-only filesystem for Linux + + Copyright 2002-2011 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to Squashfs version 4.2. Please read the README-4.2 and CHANGES files +for details of changes. + +Squashfs is a highly compressed read-only filesystem for Linux. +It uses zlib compression to compress both files, inodes and directories. +Inodes in the system are very small and all blocks are packed to minimise +data overhead. Block sizes greater than 4K are supported up to a maximum +of 1Mbytes (default block size 128K). + +Squashfs is intended for general read-only filesystem use, for archival +use (i.e. in cases where a .tar.gz file may be used), and in constrained +block device/memory systems (e.g. embedded systems) where low overhead is +needed. + +1. SQUASHFS OVERVIEW +-------------------- + +1. Data, inodes and directories are compressed. + +2. Squashfs stores full uid/gids (32 bits), and file creation time. + +3. In theory files up to 2^64 bytes are supported. In theory filesystems can + be up to 2^64 bytes. + +4. Inode and directory data are highly compacted, and packed on byte + boundaries. Each compressed inode is on average 8 bytes in length + (the exact length varies on file type, i.e. regular file, directory, + symbolic link, and block/char device inodes have different sizes). + +5. Squashfs can use block sizes up to 1Mbyte (the default size is 128K). + Using 128K blocks achieves greater compression ratios than the normal + 4K block size. + +6. File duplicates are detected and removed. + +1.1 Extended attributes (xattrs) +-------------------------------- + +Squashfs filesystems now have extended attribute support. The +extended attribute implementation has the following features: + +1. Layout can store up to 2^48 bytes of compressed xattr data. +2. Number of xattrs per inode unlimited. +3. Total size of xattr data per inode 2^48 bytes of compressed data. +4. Up to 4 Gbytes of data per xattr value. +5. Inline and out-of-line xattr values supported for higher performance + in xattr scanning (listxattr & getxattr), and to allow xattr value + de-duplication. +6. Both whole inode xattr duplicate detection and individual xattr value + duplicate detection supported. These can obviously nest, file C's + xattrs can be a complete duplicate of file B, and file B's xattrs + can be a partial duplicate of file A. +7. Xattr name prefix types stored, allowing the redundant "user.", "trusted." + etc. characters to be eliminated and more concisely stored. +8. Support for files, directories, symbolic links, device nodes, fifos + and sockets. + +Extended attribute support is in 2.6.35 and later kernels. Filesystems +with extended attributes can be mounted on 2.6.29 and later kernels, the +extended attributes will be ignored with a warning. + +2. USING SQUASHFS +----------------- + +Squashfs filesystems should be mounted with 'mount' with the filesystem type +'squashfs'. If the filesystem is on a block device, the filesystem can be +mounted directly, e.g. + +%mount -t squashfs /dev/sda1 /mnt + +Will mount the squashfs filesystem on "/dev/sda1" under the directory "/mnt". + +If the squashfs filesystem has been written to a file, the loopback device +can be used to mount it (loopback support must be in the kernel), e.g. + +%mount -t squashfs image /mnt -o loop + +Will mount the squashfs filesystem in the file "image" under +the directory "/mnt". + + +3. MKSQUASHFS +------------- + +3.1 Mksquashfs options and overview. +------------------------------------ + +As squashfs is a read-only filesystem, the mksquashfs program must be used to +create populated squashfs filesystems. + +SYNTAX:./mksquashfs source1 source2 ... dest [options] [-e list of exclude +dirs/files] + +Filesystem build options: +-comp select compression + Compressors available: + gzip (default) + lzo + xz +-b set data block to . Default 131072 bytes +-no-exports don't make the filesystem exportable via NFS +-no-sparse don't detect sparse files +-no-xattrs don't store extended attributes +-xattrs store extended attributes (default) +-noI do not compress inode table +-noD do not compress data blocks +-noF do not compress fragment blocks +-noX do not compress extended attributes +-no-fragments do not use fragments +-always-use-fragments use fragment blocks for files larger than block size +-no-duplicates do not perform duplicate checking +-all-root make all files owned by root +-force-uid uid set all file uids to uid +-force-gid gid set all file gids to gid +-nopad do not pad filesystem to a multiple of 4K +-keep-as-directory if one source directory is specified, create a root + directory containing that directory, rather than the + contents of the directory + +Filesystem filter options: +-p Add pseudo file definition +-pf Add list of pseudo file definitions +-sort sort files according to priorities in . One + file or dir with priority per line. Priority -32768 to + 32767, default priority 0 +-ef list of exclude dirs/files. One per line +-wildcards Allow extended shell wildcards (globbing) to be used in + exclude dirs/files +-regex Allow POSIX regular expressions to be used in exclude + dirs/files + +Filesystem append options: +-noappend do not append to existing filesystem +-root-becomes when appending source files/directories, make the + original root become a subdirectory in the new root + called , rather than adding the new source items + to the original root + +Mksquashfs runtime options: +-version print version, licence and copyright message +-recover recover filesystem data using recovery file +-no-recovery don't generate a recovery file +-info print files written to filesystem +-no-progress don't display the progress bar +-processors Use processors. By default will use number of + processors available +-read-queue Set input queue to Mbytes. Default 64 Mbytes +-write-queue Set output queue to Mbytes. Default 512 Mbytes +-fragment-queue Set fragment queue to Mbytes. Default 64 Mbytes + +Miscellaneous options: +-root-owned alternative name for -all-root +-noInodeCompression alternative name for -noI +-noDataCompression alternative name for -noD +-noFragmentCompression alternative name for -noF +-noXattrCompression alternative name for -noX + +Compressors available and compressor specific options: + gzip (no options) (default) + lzo (no options) + xz + -Xbcj filter1,filter2,...,filterN + Compress using filter1,filter2,...,filterN in turn + (in addition to no filter), and choose the best compression. + Available filters: x86, arm, armthumb, powerpc, sparc, ia64 + -Xdict-size + Use as the XZ dictionary size. The dictionary size + can be specified as a percentage of the block size, or as an + absolute value. The dictionary size must be less than or equal + to the block size and 8192 bytes or larger. It must also be + storable in the xz header as either 2^n or as 2^n+2^(n+1). + Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K + etc. + + +Source1 source2 ... are the source directories/files containing the +files/directories that will form the squashfs filesystem. If a single +directory is specified (i.e. mksquashfs source output_fs) the squashfs +filesystem will consist of that directory, with the top-level root +directory corresponding to the source directory. + +If multiple source directories or files are specified, mksquashfs will merge +the specified sources into a single filesystem, with the root directory +containing each of the source files/directories. The name of each directory +entry will be the basename of the source path. If more than one source +entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where +xxx is the original name. + +To make this clear, take two example directories. Source directory +"/home/phillip/test" contains "file1", "file2" and "dir1". +Source directory "goodies" contains "goodies1", "goodies2" and "goodies3". + +usage example 1: + +%mksquashfs /home/phillip/test output_fs + +This will generate a squashfs filesystem with root entries +"file1", "file2" and "dir1". + +example 2: + +%mksquashfs /home/phillip/test goodies output_fs + +This will create a squashfs filesystem with the root containing +entries "test" and "goodies" corresponding to the source +directories "/home/phillip/test" and "goodies". + +example 3: + +%mksquashfs /home/phillip/test goodies test output_fs + +This is the same as the previous example, except a third +source directory "test" has been specified. This conflicts +with the first directory named "test" and will be renamed "test_1". + +Multiple sources allow filesystems to be generated without needing to +copy all source files into a common directory. This simplifies creating +filesystems. + +The -keep-as-directory option can be used when only one source directory +is specified, and you wish the root to contain that directory, rather than +the contents of the directory. For example: + +example 4: + +%mksquashfs /home/phillip/test output_fs -keep-as-directory + +This is the same as example 1, except for -keep-as-directory. +This will generate a root directory containing directory "test", +rather than the "test" directory contents "file1", "file2" and "dir1". + +The Dest argument is the destination where the squashfs filesystem will be +written. This can either be a conventional file or a block device. If the file +doesn't exist it will be created, if it does exist and a squashfs +filesystem exists on it, mksquashfs will append. The -noappend option will +write a new filesystem irrespective of whether an existing filesystem is +present. + +3.2 Changing compression algorithm and compression specific options +------------------------------------------------------------------- + +By default Mksquashfs will compress using the gzip compression +algorithm. This algorithm offers a good trade-off between compression +ratio, and memory and time taken to decompress. + +Squashfs also supports LZO and XZ (LZMA2) compression. LZO offers worse +compression ratio than gzip, but is faster to decompress. XZ offers better +compression ratio than gzip, but at the expense of greater memory and time +to decompress (and significantly more time to compress). + +If you're not building the squashfs-tools and kernel from source, then +the tools and kernel may or may not have been built with support for LZO or +XZ compression. The compression algorithms supported by the build of +Mksquashfs can be found by typing mksquashfs without any arguments. The +compressors available are displayed at the end of the help message, e.g. + +Compressors available and compressor specific options: + gzip (no options) (default) + lzo (no options) + xz + -Xbcj filter1,filter2,...,filterN + Compress using filter1,filter2,...,filterN in turn + (in addition to no filter), and choose the best compression. + Available filters: x86, arm, armthumb, powerpc, sparc, ia64 + -Xdict-size + Use as the XZ dictionary size. The dictionary size + can be specified as a percentage of the block size, or as an + absolute value. The dictionary size must be less than or equal + to the block size and 8192 bytes or larger. It must also be + storable in the xz header as either 2^n or as 2^n+2^(n+1). + Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K + etc. + +If the compressor offers compression specific options then these +options are also displayed (.i.e. in the above XZ is shown with two +compression specific options). + +3.3 Changing global compression defaults used in mksquashfs +----------------------------------------------------------- + +There are a large number of options that can be used to control the +compression in mksquashfs. By and large the defaults are the most +optimum settings and should only be changed in exceptional circumstances! + +The -noI, -noD and -noF options (also -noInodeCompression, -noDataCompression +and -noFragmentCompression) can be used to force mksquashfs to not compress +inodes/directories, data and fragments respectively. Giving all options +generates an uncompressed filesystem. + +The -no-fragments tells mksquashfs to not generate fragment blocks, and rather +generate a filesystem similar to a Squashfs 1.x filesystem. It will of course +still be a Squashfs 3.1 filesystem but without fragments, and so it won't be +mountable on a Squashfs 1.x system. + +The -always-use-fragments option tells mksquashfs to always generate +fragments for files irrespective of the file length. By default only small +files less than the block size are packed into fragment blocks. The ends of +files which do not fit fully into a block, are NOT by default packed into +fragments. To illustrate this, a 100K file has an initial 64K block and a 36K +remainder. This 36K remainder is not packed into a fragment by default. This +is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a +disk head seek is needed to seek to the initial file data and another disk seek +is need to seek to the fragment block. Specify this option if you want file +remainders to be packed into fragment blocks. Doing so may increase the +compression obtained BUT at the expense of I/O speed. + +The -no-duplicates option tells mksquashfs to not check the files being +added to the filesystem for duplicates. This can result in quicker filesystem +generation and appending although obviously compression will suffer badly if +there is a lot of duplicate files. + +The -b option allows the block size to be selected, both "K" and "M" postfixes +are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or +1M bytes. + +3.4 Specifying the UIDs/GIDs used in the filesystem +--------------------------------------------------- + +By default files in the generated filesystem inherit the UID and GID ownership +of the original file. However, mksquashfs provides a number of options which +can be used to override the ownership. + +The options -all-root and -root-owned (both do exactly the same thing) force all +file uids/gids in the generated Squashfs filesystem to be root. This allows +root owned filesystems to be built without root access on the host machine. + +The "-force-uid uid" option forces all files in the generated Squashfs +filesystem to be owned by the specified uid. The uid can be specified either by +name (i.e. "root") or by number. + +The "-force-gid gid" option forces all files in the generated Squashfs +filesystem to be group owned by the specified gid. The gid can be specified +either by name (i.e. "root") or by number. + +3.5 Excluding files from the filesystem +--------------------------------------- + +The -e and -ef options allow files/directories to be specified which are +excluded from the output filesystem. The -e option takes the exclude +files/directories from the command line, the -ef option takes the +exlude files/directories from the specified exclude file, one file/directory +per line. + +Two styles of exclude file matching are supported: basic exclude matching, and +extended wildcard matching. Basic exclude matching is a legacy feature +retained for backwards compatibility with earlier versions of Mksquashfs. +Extended wildcard matching should be used in preference. + +3.5.1 Basic exclude matching + +Each exclude file is treated as an exact match of a file/directory in +the source directories. If an exclude file/directory is absolute (i.e. +prefixed with /, ../, or ./) the entry is treated as absolute, however, if an +exclude file/directory is relative, it is treated as being relative to each of +the sources in turn, i.e. + +%mksquashfs /tmp/source1 source2 output_fs -e ex1 /tmp/source1/ex2 out/ex3 + +Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1, +/tmp/source1/out/ex3 and source2/out/ex3. + +3.5.2 Extended exclude file handling + +Extended exclude file matching treats each exclude file as a wildcard or +regex expression. To enable wildcard matching specify the -wildcards +option, and to enable regex matching specify the -regex option. In most +cases the -wildcards option should be used rather than -regex because wildcard +matching behaviour is significantly easier to understand! + +In addition to wildcards/regex expressions, exclude files can be "anchored" or +"non-anchored". An anchored exclude is one which matches from the root of the +directory and nowhere else, a non-anchored exclude matches anywhere. For +example given the directory hierarchy "a/b/c/a/b", the anchored exclude +"a/b" will match "a/b" at the root of the directory hierarchy, but +it will not match the "/a/b" sub-directory within directory "c", whereas a +non-anchored exclude would. + +A couple of examples should make this clearer. + +Anchored excludes + + 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz' + + Exclude all files matching "*.gz" in the top level directory "test". + + 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*' + + Exclude all files beginning with "example" inside directories called + "Test" or "test", that occur inside any top level directory. + + Using extended wildcards, negative matching is also possible. + + 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz' + + Exclude all files matching "*.gz" in top level directory "test", + except those with "data" in the name. + +Non-anchored excludes + + By default excludes match from the top level directory, but it is + often useful to exclude a file matching anywhere in the source directories. + For this non-anchored excludes can be used, specified by pre-fixing the + exclude with "...". + + Examples: + + 1. mksquashfs example image.sqsh -wildcards -e '... *.gz' + + Exclude files matching "*.gz" anywhere in the source directories. + For example this will match "example.gz", "test/example.gz", and + "test/test/example.gz". + + 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz' + + Exclude files matching "*.gz" inside directories called "Test" or + "test" that occur anywhere in the source directories. + + Again, using extended wildcards, negative matching is also possible. + + 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz' + + Exclude all files matching "*.gz" anywhere in the source directories, + except those with "data" in the name. + +3.5.3 Exclude files summary + +The -e and -ef exclude options are usefully used in archiving the entire +filesystem, where it is wished to avoid archiving /proc, and the filesystem +being generated, i.e. + +%mksquashfs / /tmp/root.sqsh -e proc /tmp/root.sqsh + +Multiple -ef options can be specified on the command line, and the -ef +option can be used in conjuction with the -e option. + +3.6 Appending to squashfs filesystems +------------------------------------- + +Running squashfs with the destination directory containing an existing +filesystem will add the source items to the existing filesystem. By default, +the source items are added to the existing root directory. + +To make this clear... An existing filesystem "image" contains root entries +"old1", and "old2". Source directory "/home/phillip/test" contains "file1", +"file2" and "dir1". + +example 1: + +%mksquashfs /home/phillip/test image + +Will create a new "image" with root entries "old1", "old2", "file1", "file2" and +"dir1" + +example 2: + +%mksquashfs /home/phillip/test image -keep-as-directory + +Will create a new "image" with root entries "old1", "old2", and "test". +As shown in the previous section, for single source directories +'-keep-as-directory' adds the source directory rather than the +contents of the directory. + +example 3: + +%mksquashfs /home/phillip/test image -keep-as-directory -root-becomes +original-root + +Will create a new "image" with root entries "original-root", and "test". The +'-root-becomes' option specifies that the original root becomes a subdirectory +in the new root, with the specified name. + +The append option with file duplicate detection, means squashfs can be +used as a simple versioning archiving filesystem. A squashfs filesystem can +be created with for example the linux-2.4.19 source. Appending the linux-2.4.20 +source will create a filesystem with the two source trees, but only the +changed files will take extra room, the unchanged files will be detected as +duplicates. + +3.7 Appending recovery file feature +----------------------------------- + +Recovery files are created when appending to existing Squashfs +filesystems. This allows the original filesystem to be recovered +if Mksquashfs aborts unexpectedly (i.e. power failure). + +The recovery files are called squashfs_recovery_xxx_yyy, where +"xxx" is the name of the filesystem being appended to, and "yyy" is a +number to guarantee filename uniqueness (the PID of the parent Mksquashfs +process). + +Normally if Mksquashfs exits correctly the recovery file is deleted to +avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover" +option can be used to recover the filesystem, giving the previously +created recovery file as a parameter, i.e. + +mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234 + +The writing of the recovery file can be disabled by specifying the +"-no-recovery" option. + +3.8 Pseudo file support +----------------------- + +Mksquashfs supports pseudo files, these allow fake files, directories, character +and block devices to be specified and added to the Squashfs filesystem being +built, rather than requiring them to be present in the source directories. +This, for example, allows device nodes to be added to the filesystem without +requiring root access. + +Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation. +Dynamic pseudo files allow files to be dynamically created when Mksquashfs +is run, their contents being the result of running a command or piece of +shell script. The modifiy operation allows the mode/uid/gid of an existing +file in the source filesystem to be modified. + +Two Mksquashfs options are supported, -p allows one pseudo file to be specified +on the command line, and -pf allows a pseudo file to be specified containing a +list of pseduo definitions, one per line. + +Pseudo operations +----------------- + +3.8.1. Creating a dynamic file +-------------------------- + +Pseudo definition + +Filename f mode uid gid command + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +command can be an executable or a piece of shell script, and it is executed +by running "/bin/sh -c command". The stdout becomes the contents of +"Filename". + +Examples: + +Running a basic command +----------------------- + +/somedir/dmesg f 444 root root dmesg + +creates a file "/somedir/dmesg" containing the output from dmesg. + +Executing shell script +---------------------- + +RELEASE f 444 root root \ + if [ ! -e /tmp/ver ]; then \ + echo 0 > /tmp/ver; \ + fi; \ + ver=`cat /tmp/ver`; \ + ver=$((ver +1)); \ + echo $ver > /tmp/ver; \ + echo -n `cat /tmp/release`; \ + echo "-dev #"$ver `date` "Build host" `hostname` + +Creates a file RELEASE containing the release name, date, build host, and +an incrementing version number. The incrementing version is a side-effect +of executing the shell script, and ensures every time Mksquashfs is run a +new version number is used without requiring any other shell scripting. + +The above example also shows that commands can be split across multiple lines +using "\". Obviously as the script will be presented to the shell as a single +line, a semicolon is need to separate individual shell commands within the +shell script. + +Reading from a device (or fifo/named socket) +-------------------------------------------- + +input f 444 root root dd if=/dev/sda1 bs=1024 count=10 + +Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs +given a device, fifo, or named socket will place that special file within the +Squashfs filesystem, the above allows input from these special files to be +captured and placed in the Squashfs filesystem. + +3.8.2. Creating a block or character device +--------------------------------------- + +Pseudo definition + +Filename type mode uid gid major minor + +Where type is either + b - for block devices, and + c - for character devices + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +For example: + +/dev/chr_dev c 666 root root 100 1 +/dev/blk_dev b 666 0 0 200 200 + +creates a character device "/dev/chr_dev" with major:minor 100:1 and +a block device "/dev/blk_dev" with major:minor 200:200, both with root +uid/gid and a mode of rw-rw-rw. + +3.8.3. Creating a directory +----------------------- + +Pseudo definition + +Filename d mode uid gid + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +For example: + +/pseudo_dir d 666 root root + +creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw. + +4.8.4. Modifying attributes of an existing file +------------------------------------------- + +Pseudo definition + +Filename m mode uid gid + +mode is the octal mode specifier, similar to that expected by chmod. + +uid and gid can be either specified as a decimal number, or by name. + +For example: + +dmesg m 666 root root + +Changes the attributes of the file "dmesg" in the filesystem to have +root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained +from the source filesystem. + +3.9 Miscellaneous options +------------------------- + +The -info option displays the files/directories as they are compressed and +added to the filesystem. The original uncompressed size of each file +is printed, along with DUPLICATE if the file is a duplicate of a +file in the filesystem. + +The -nopad option informs mksquashfs to not pad the filesystem to a 4K multiple. +This is performed by default to enable the output filesystem file to be mounted +by loopback, which requires files to be a 4K multiple. If the filesystem is +being written to a block device, or is to be stored in a bootimage, the extra +pad bytes are not needed. + +4. UNSQUASHFS +------------- + +Unsquashfs allows you to decompress and extract a Squashfs filesystem without +mounting it. It can extract the entire filesystem, or a specific +file or directory. + +The Unsquashfs usage info is: + +SYNTAX: unsquashfs [options] filesystem [directories or files to extract] + -v[ersion] print version, licence and copyright information + -d[est] unsquash to , default "squashfs-root" + -n[o-progress] don't display the progress bar + -no[-xattrs] don't extract xattrs in file system + -x[attrs] extract xattrs in file system (default) + -p[rocessors] use processors. By default will use + number of processors available + -i[nfo] print files as they are unsquashed + -li[nfo] print files as they are unsquashed with file + attributes (like ls -l output) + -l[s] list filesystem, but don't unsquash + -ll[s] list filesystem with file attributes (like + ls -l output), but don't unsquash + -f[orce] if file already exists then overwrite + -s[tat] display filesystem superblock information + -e[f] list of directories or files to extract. + One per line + -da[ta-queue] Set data queue to Mbytes. Default 256 + Mbytes + -fr[ag-queue] Set fagment queue to Mbytes. Default 256 + Mbytes + -r[egex] treat extract names as POSIX regular expressions + rather than use the default shell wildcard + expansion (globbing) + +Decompressors available: + gzip + lzma + xz + + +To extract a subset of the filesystem, the filenames or directory +trees that are to be extracted can be specified on the command line. The +files/directories should be specified using the full path to the +files/directories as they appear within the Squashfs filesystem. The +files/directories will also be extracted to those positions within the specified +destination directory. + +The extract files can also be given in a file using the "-e[f]" option. + +Similarly to Mksquashfs, wildcard matching is performed on the extract +files. Wildcard matching is enabled by default. + +Examples: + + 1. unsquashfs image.sqsh 'test/*.gz' + + Extract all files matching "*.gz" in the top level directory "test". + + 2. unsquashfs image.sqsh '[Tt]est/example*' + + Extract all files beginning with "example" inside top level directories + called "Test" or "test". + + Using extended wildcards, negative matching is also possible. + + 3. unsquashfs image.sqsh 'test/!(*data*).gz' + + Extract all files matching "*.gz" in top level directory "test", + except those with "data" in the name. + + +4.1 Unsquashfs options +---------------------- + +The "-ls" option can be used to list the contents of a filesystem without +decompressing the filesystem data itself. The "-lls" option is similar +but it also displays file attributes (ls -l style output). + +The "-info" option forces Unsquashfs to print each file as it is decompressed. +The -"linfo" is similar but it also displays file attributes. + +The "-dest" option specifies the directory that is used to decompress +the filesystem data. If this option is not given then the filesystem is +decompressed to the directory "squashfs-root" in the current working directory. + +The "-force" option forces Unsquashfs to output to the destination +directory even if files or directories already exist. This allows you +to update an existing directory tree, or to Unsquashfs to a partially +filled directory. Without the "-force" option, Unsquashfs will +refuse to overwrite any existing files, or to create any directories if they +already exist. This is done to protect data in case of mistakes, and +so the "-force" option should be used with caution. + +The "-stat" option displays filesystem superblock information. This is +useful to discover the filesystem version, byte ordering, whether it has a NFS +export table, and what options were used to compress the filesystem, etc. + +Unsquashfs can decompress all Squashfs filesystem versions, 1.x, 2.x and 3.x +filesystems. + +5. FILESYSTEM LAYOUT +-------------------- + +A squashfs filesystem consists of a maximum of nine parts, packed together on a +byte alignment: + + --------------- + | superblock | + |---------------| + | compression | + | options | + |---------------| + | datablocks | + | & fragments | + |---------------| + | inode table | + |---------------| + | directory | + | table | + |---------------| + | fragment | + | table | + |---------------| + | export | + | table | + |---------------| + | uid/gid | + | lookup table | + |---------------| + | xattr | + | table | + --------------- + +Compressed data blocks are written to the filesystem as files are read from +the source directory, and checked for duplicates. Once all file data has been +written the completed inode, directory, fragment, export and uid/gid lookup +tables are written. + +5.1 Compression options +----------------------- + +Compressors can optionally support compression specific options (e.g. +dictionary size). If non-default compression options have been used, then +these are stored here. + +5.2 Inodes +---------- + +Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each +compressed block is prefixed by a two byte length, the top bit is set if the +block is uncompressed. A block will be uncompressed if the -noI option is set, +or if the compressed block was larger than the uncompressed block. + +Inodes are packed into the metadata blocks, and are not aligned to block +boundaries, therefore inodes overlap compressed blocks. Inodes are identified +by a 48-bit number which encodes the location of the compressed metadata block +containing the inode, and the byte offset into that block where the inode is +placed (). + +To maximise compression there are different inodes for each file type +(regular file, directory, device, etc.), the inode contents and length +varying with the type. + +To further maximise compression, two types of regular file inode and +directory inode are defined: inodes optimised for frequently occurring +regular files and directories, and extended types where extra +information has to be stored. + +5.3 Directories +--------------- + +Like inodes, directories are packed into compressed metadata blocks, stored +in a directory table. Directories are accessed using the start address of +the metablock containing the directory and the offset into the +decompressed block (). + +Directories are organised in a slightly complex way, and are not simply +a list of file names. The organisation takes advantage of the +fact that (in most cases) the inodes of the files will be in the same +compressed metadata block, and therefore, can share the start block. +Directories are therefore organised in a two level list, a directory +header containing the shared start block value, and a sequence of directory +entries, each of which share the shared start block. A new directory header +is written once/if the inode start block changes. The directory +header/directory entry list is repeated as many times as necessary. + +Directories are sorted, and can contain a directory index to speed up +file lookup. Directory indexes store one entry per metablock, each entry +storing the index/filename mapping to the first directory header +in each metadata block. Directories are sorted in alphabetical order, +and at lookup the index is scanned linearly looking for the first filename +alphabetically larger than the filename being looked up. At this point the +location of the metadata block the filename is in has been found. +The general idea of the index is ensure only one metadata block needs to be +decompressed to do a lookup irrespective of the length of the directory. +This scheme has the advantage that it doesn't require extra memory overhead +and doesn't require much extra storage on disk. + +5.4 File data +------------- + +Regular files consist of a sequence of contiguous compressed blocks, and/or a +compressed fragment block (tail-end packed block). The compressed size +of each datablock is stored in a block list contained within the +file inode. + +To speed up access to datablocks when reading 'large' files (256 Mbytes or +larger), the code implements an index cache that caches the mapping from +block index to datablock location on disk. + +The index cache allows Squashfs to handle large files (up to 1.75 TiB) while +retaining a simple and space-efficient block list on disk. The cache +is split into slots, caching up to eight 224 GiB files (128 KiB blocks). +Larger files use multiple slots, with 1.75 TiB files using all 8 slots. +The index cache is designed to be memory efficient, and by default uses +16 KiB. + +5.5 Fragment lookup table +------------------------- + +Regular files can contain a fragment index which is mapped to a fragment +location on disk and compressed size using a fragment lookup table. This +fragment lookup table is itself stored compressed into metadata blocks. +A second index table is used to locate these. This second index table for +speed of access (and because it is small) is read at mount time and cached +in memory. + +5.6 Uid/gid lookup table +------------------------ + +For space efficiency regular files store uid and gid indexes, which are +converted to 32-bit uids/gids using an id look up table. This table is +stored compressed into metadata blocks. A second index table is used to +locate these. This second index table for speed of access (and because it +is small) is read at mount time and cached in memory. + +5.7 Export table +---------------- + +To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems +can optionally (disabled with the -no-exports Mksquashfs option) contain +an inode number to inode disk location lookup table. This is required to +enable Squashfs to map inode numbers passed in filehandles to the inode +location on disk, which is necessary when the export code reinstantiates +expired/flushed inodes. + +This table is stored compressed into metadata blocks. A second index table is +used to locate these. This second index table for speed of access (and because +it is small) is read at mount time and cached in memory. + +5.8 Xattr table +--------------- + +The xattr table contains extended attributes for each inode. The xattrs +for each inode are stored in a list, each list entry containing a type, +name and value field. The type field encodes the xattr prefix +("user.", "trusted." etc) and it also encodes how the name/value fields +should be interpreted. Currently the type indicates whether the value +is stored inline (in which case the value field contains the xattr value), +or if it is stored out of line (in which case the value field stores a +reference to where the actual value is stored). This allows large values +to be stored out of line improving scanning and lookup performance and it +also allows values to be de-duplicated, the value being stored once, and +all other occurences holding an out of line reference to that value. + +The xattr lists are packed into compressed 8K metadata blocks. +To reduce overhead in inodes, rather than storing the on-disk +location of the xattr list inside each inode, a 32-bit xattr id +is stored. This xattr id is mapped into the location of the xattr +list using a second xattr id lookup table. + +6. AUTHOR INFO +-------------- + +Squashfs was written by Phillip Lougher, email phillip@lougher.demon.co.uk, +in Chepstow, Wales, UK. If you like the program, or have any problems, +then please email me, as it's nice to get feedback! diff --git a/README-4.2 b/README-4.2 new file mode 100644 index 0000000..db28f53 --- /dev/null +++ b/README-4.2 @@ -0,0 +1,57 @@ + SQUASHFS 4.2 - A squashed read-only filesystem for Linux + + Copyright 2002-2011 Phillip Lougher + + Released under the GPL licence (version 2 or later). + +Welcome to Squashfs 4.2. This is a tools only release, support for Squashfs +filesystems is in mainline (2.6.29 and later). + +New features in Squashfs-tools 4.2 +---------------------------------- + + 1. Support for XZ compression + 2. Support for compressor specific options + +Compatiblity +------------ + +Mksquashfs 4.2 generates 4.0 filesystems. These filesystems are fully +compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are +mountable on 2.6.29 and later kernels. + +XZ compression +-------------- + +Squashfs now supports XZ compression. + +XZ support is in 2.6.38 and newer kernels. + +New Mksquashfs options +---------------------- + +-X + + Compression algorithms can now support compression specific options. These +options are prefixed by -X, and are passed to the compressor for handling. + + The compression specific options supported by each compressor can be +found by typing mksquashfs without any arguments. They are displayed at the +end of the help message, e.g. + +Compressors available and compressor specific options: + gzip (no options) (default) + lzo (no options) + xz + -Xbcj filter1,filter2,...,filterN + Compress using filter1,filter2,...,filterN in turn + (in addition to no filter), and choose the best compression. + Available filters: x86, arm, armthumb, powerpc, sparc, ia64 + -Xdict-size + Use as the XZ dictionary size. The dictionary size + can be specified as a percentage of the block size, or as an + absolute value. The dictionary size must be less than or equal + to the block size and 8192 bytes or larger. It must also be + storable in the xz header as either 2^n or as 2^n+2^(n+1). + Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K + etc. diff --git a/packaging/squashfs-4.2-makefile_config.patch b/packaging/squashfs-4.2-makefile_config.patch new file mode 100644 index 0000000..24fbbfd --- /dev/null +++ b/packaging/squashfs-4.2-makefile_config.patch @@ -0,0 +1,9 @@ +--- squashfs-4.2/squashfs-tools/Makefile.org 2011-03-01 05:04:14.000000000 +0900 ++++ squashfs-4.2/squashfs-tools/Makefile 2013-01-10 20:20:52.000000000 +0900 +@@ -37,7 +37,7 @@ + # LZO_SUPPORT line below. If needed, uncomment and set LZO_DIR to the + # installation prefix. + # +-#LZO_SUPPORT = 1 ++LZO_SUPPORT = 1 + #LZO_DIR = /usr/local diff --git a/packaging/squashfs.spec b/packaging/squashfs.spec new file mode 100644 index 0000000..3675c1a --- /dev/null +++ b/packaging/squashfs.spec @@ -0,0 +1,67 @@ +Name: squashfs +Version: 4.2 +Release: 1 +Summary: Tools for squashfs, a compressed read-only filesystem for Linux +Group: System/Tools +Source0: %{name}-%{version}.tar.gz +Patch0: squashfs-4.2-makefile_config.patch +License: GPL +URL: http://squashfs.sourceforge.net/ +BuildRequires: eglibc-devel +BuildRequires: lzo-devel +BuildRequires: zlib-devel +Requires: lzo + +%description +Squashfs is a compressed read-only filesystem for Linux. +Squashfs is intended for general read-only filesystem use, +for archival use (i.e. in cases where a .tar.gz file may be used), +and in constrained block device/memory systems (e.g. embedded systems) where low overhead is needed. +The filesystem is currently stable, and has been tested on PowerPC, i586, Sparc and ARM architectures. + +%prep +%setup -q +%patch0 -p1 + +%build +cd squashfs-tools +make %{?jobs:-j%jobs} + +%install +rm -rf %{buildroot} +cd squashfs-tools +mkdir -p %{buildroot}%{_bindir} +install mksquashfs %{buildroot}%{_bindir}/mksquashfs +install unsquashfs %{buildroot}%{_bindir}/unsquashfs + +mkdir -p %{buildroot}/usr/share/license +cp -f ../COPYING %{buildroot}/usr/share/license/%{name} + +%files +%defattr(-,root,root) +%attr(755,root,root) %{_bindir}/* +%doc CHANGES COPYING README +/usr/share/license/%{name} + +%changelog +* Wed Mar 02 2011 Silvan Calarco 4.2-1mamba +- update to 4.2 + +* Mon Feb 08 2010 Silvan Calarco 4.0-2mamba +- added patch for lzma support and rebuilt using lzma sdk + +* Tue Apr 07 2009 Silvan Calarco 4.0-1mamba +- update to 4.0 + +* Sun Sep 14 2008 Silvan Calarco 3.4-1mamba +- update to 3.4 + +* Fri Jul 04 2008 Silvan Calarco 3.3-2mamba +- applied patch to fix hang problem while creating livegames dvd + +* Tue Jan 15 2008 Silvan Calarco 3.3-1mamba +- update to 3.3 + +* Fri Jun 30 2006 Silvan Calarco 3.0-1qilnx +- package created by autospec + diff --git a/pseudo-file.example b/pseudo-file.example new file mode 100644 index 0000000..f866d90 --- /dev/null +++ b/pseudo-file.example @@ -0,0 +1,74 @@ +# Pseudo file example + +# Mksquashfs supports pseudo files, these allow fake files, directories, +# character and block devices to be specified and added to the Squashfs +# filesystem being built, rather than requiring them to be present in the +# source directories. +# +# This, for example, allows device nodes to be added to the filesystem without +# requiring root access. + +# Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation. +# Dynamic pseudo files allow files to be dynamically created when Mksquashfs +# is run, their contents being the result of running a command or piece of +# shell script. The modifiy operation allows the mode/uid/gid of an existing +# file in the source filesystem to be modified. + +# Two Mksquashfs options are supported, -p allows one pseudo file to be +# specified #on the command line, and -pf allows a pseudo file to be specified +# containing a list of pseduo definitions, one per line. + +# Pseudo file examples +# Run mkquashfs . /tmp/img -pf pseudo-file.examples +# to see their effect + +# Creating dynamic file examples + +# Create a file "dmesg" containing the output from dmesg. +dmesg f 444 root root dmesg + + +# Create a file RELEASE containing the release name, date, build host, and +# an incrementing version number. The incrementing version is a side-effect +# of executing the shell script, and ensures every time Mksquashfs is run a +# new version number is used without requiring any other shell scripting. +RELEASE f 444 root root \ + if [ ! -e /tmp/ver ]; then \ + echo 0 > /tmp/ver; \ + fi; \ + ver=`cat /tmp/ver`; \ + ver=$((ver +1)); \ + echo $ver > /tmp/ver; \ + echo -n "release x.x"; \ + echo "-dev #"$ver `date` "Build host" `hostname` + + +# Copy 10K from the device /dev/sda1 into the file input. Ordinarily +# Mksquashfs given a device, fifo, or named socket will place that special file +# within the Squashfs filesystem, this allows input from these special +# files to be captured and placed in the Squashfs filesystem. +input f 444 root root dd if=/dev/sda1 bs=1024 count=10 + + +# Creating a block or character device examples + +# Create a character device "chr_dev" with major:minor 100:1 and +# a block device "blk_dev" with major:minor 200:200, both with root +# uid/gid and a mode of rw-rw-rw. +chr_dev c 666 root root 100 1 +blk_dev b 666 0 0 200 200 + + +# Creating a directory example + +# create a directory "pseudo_dir" with root uid/gid and mode of r--r--r--. +pseudo_dir d 444 root root + + +# Modifying attributes of an existing file exmaple + +# Change the attributes of the file "INSTALL" in the filesystem to have +# root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained +# from the source filesystem. +INSTALL m 666 root root + diff --git a/squashfs-tools/Makefile b/squashfs-tools/Makefile new file mode 100644 index 0000000..769ec3f --- /dev/null +++ b/squashfs-tools/Makefile @@ -0,0 +1,258 @@ +############################################### +# Compression build options # +############################################### +# +# +############# Building gzip support ########### +# +# Gzip support is by default enabled, and the compression type default +# (COMP_DEFAULT) is gzip. +# +# If you don't want/need gzip support then comment out the GZIP SUPPORT line +# below, and change COMP_DEFAULT to one of the compression types you have +# selected. +# +# Obviously, you must select at least one of the available gzip, lzma, lzo +# compression types. +# +GZIP_SUPPORT = 1 + +########### Building XZ support ############# +# +# LZMA2 compression. +# +# XZ Utils liblzma (http://tukaani.org/xz/) is supported +# +# To build using XZ Utils liblzma - install the library and uncomment +# the XZ_SUPPORT line below. +# +#XZ_SUPPORT = 1 + + +############ Building LZO support ############## +# +# The LZO library (http://www.oberhumer.com/opensource/lzo/) is supported. +# +# To build using the LZO library - install the library and uncomment the +# LZO_SUPPORT line below. If needed, uncomment and set LZO_DIR to the +# installation prefix. +# +#LZO_SUPPORT = 1 +#LZO_DIR = /usr/local + +########### Building LZMA support ############# +# +# LZMA1 compression. +# +# LZMA1 compression is deprecated, and the newer and better XZ (LZMA2) +# compression should be used in preference. +# +# Both XZ Utils liblzma (http://tukaani.org/xz/) and LZMA SDK +# (http://www.7-zip.org/sdk.html) are supported +# +# To build using XZ Utils liblzma - install the library and uncomment +# the LZMA_XZ_SUPPORT line below. +# +# To build using the LZMA SDK (4.65 used in development, other versions may +# work) - download and unpack it, uncomment and set LZMA_DIR to unpacked source, +# and uncomment the LZMA_SUPPORT line below. +# +#LZMA_XZ_SUPPORT = 1 +#LZMA_SUPPORT = 1 +#LZMA_DIR = ../../../../LZMA/lzma465 + +######## Specifying default compression ######## +# +# The next line specifies which compression algorithm is used by default +# in Mksquashfs. Obviously the compression algorithm must have been +# selected to be built +# +COMP_DEFAULT = gzip + +############################################### +# Extended attribute (XATTRs) build options # +############################################### +# +# Building XATTR support for Mksquashfs and Unsquashfs +# +# If your C library or build/target environment doesn't support XATTRs then +# comment out the next line to build Mksquashfs and Unsquashfs without XATTR +# support +XATTR_SUPPORT = 1 + +# Select whether you wish xattrs to be stored by Mksquashfs and extracted +# by Unsquashfs by default. If selected users can disable xattr support by +# using the -no-xattrs option +# +# If unselected, Mksquashfs/Unsquashfs won't store and extract xattrs by +# default. Users can enable xattrs by using the -xattrs option. +XATTR_DEFAULT = 1 + + +############################################### +# End of BUILD options section # +############################################### + +INCLUDEDIR = -I. +INSTALL_DIR = /usr/local/bin + +MKSQUASHFS_OBJS = mksquashfs.o read_fs.o sort.o swap.o pseudo.o compressor.o + +UNSQUASHFS_OBJS = unsquashfs.o unsquash-1.o unsquash-2.o unsquash-3.o \ + unsquash-4.o swap.o compressor.o + +CFLAGS ?= -O2 +CFLAGS += $(EXTRA_CFLAGS) $(INCLUDEDIR) -D_FILE_OFFSET_BITS=64 \ + -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"$(COMP_DEFAULT)\" \ + -Wall + +LIBS = -lpthread -lm +ifeq ($(GZIP_SUPPORT),1) +CFLAGS += -DGZIP_SUPPORT +MKSQUASHFS_OBJS += gzip_wrapper.o +UNSQUASHFS_OBJS += gzip_wrapper.o +LIBS += -lz +COMPRESSORS += gzip +endif + +ifeq ($(LZMA_SUPPORT),1) +LZMA_OBJS = $(LZMA_DIR)/C/Alloc.o $(LZMA_DIR)/C/LzFind.o \ + $(LZMA_DIR)/C/LzmaDec.o $(LZMA_DIR)/C/LzmaEnc.o $(LZMA_DIR)/C/LzmaLib.o +INCLUDEDIR += -I$(LZMA_DIR)/C +CFLAGS += -DLZMA_SUPPORT +MKSQUASHFS_OBJS += lzma_wrapper.o $(LZMA_OBJS) +UNSQUASHFS_OBJS += lzma_wrapper.o $(LZMA_OBJS) +COMPRESSORS += lzma +endif + +ifeq ($(LZMA_XZ_SUPPORT),1) +CFLAGS += -DLZMA_SUPPORT +MKSQUASHFS_OBJS += lzma_xz_wrapper.o +UNSQUASHFS_OBJS += lzma_xz_wrapper.o +LIBS += -llzma +COMPRESSORS += lzma +endif + +ifeq ($(XZ_SUPPORT),1) +CFLAGS += -DXZ_SUPPORT +MKSQUASHFS_OBJS += xz_wrapper.o +UNSQUASHFS_OBJS += xz_wrapper.o +LIBS += -llzma +COMPRESSORS += xz +endif + +ifeq ($(LZO_SUPPORT),1) +CFLAGS += -DLZO_SUPPORT +ifdef LZO_DIR +INCLUDEDIR += -I$(LZO_DIR)/include +LZO_LIBDIR = -L$(LZO_DIR)/lib +endif +MKSQUASHFS_OBJS += lzo_wrapper.o +UNSQUASHFS_OBJS += lzo_wrapper.o +LIBS += $(LZO_LIBDIR) -llzo2 +COMPRESSORS += lzo +endif + +ifeq ($(XATTR_SUPPORT),1) +ifeq ($(XATTR_DEFAULT),1) +CFLAGS += -DXATTR_SUPPORT -DXATTR_DEFAULT +else +CFLAGS += -DXATTR_SUPPORT +endif +MKSQUASHFS_OBJS += xattr.o read_xattrs.o +UNSQUASHFS_OBJS += read_xattrs.o unsquashfs_xattr.o +endif + +# +# If LZMA_SUPPORT is specified then LZMA_DIR must be specified too +# +ifeq ($(LZMA_SUPPORT),1) +ifndef LZMA_DIR +$(error "LZMA_SUPPORT requires LZMA_DIR to be also defined") +endif +endif + +# +# Both LZMA_XZ_SUPPORT and LZMA_SUPPORT cannot be specified +# +ifeq ($(LZMA_XZ_SUPPORT),1) +ifeq ($(LZMA_SUPPORT),1) +$(error "Both LZMA_XZ_SUPPORT and LZMA_SUPPORT cannot be specified") +endif +endif + +# +# At least one compressor must have been selected +# +ifndef COMPRESSORS +$(error "No compressor selected! Select one or more of GZIP, LZMA, XZ or LZO!") +endif + +# +# COMP_DEFAULT must be a selected compressor +# +ifeq (, $(findstring $(COMP_DEFAULT), $(COMPRESSORS))) +$(error "COMP_DEFAULT isn't selected to be built!") +endif + +.PHONY: all +all: mksquashfs unsquashfs + +mksquashfs: $(MKSQUASHFS_OBJS) + $(CC) $(LDFLAGS) $(EXTRA_LDFLAGS) $(MKSQUASHFS_OBJS) $(LIBS) -o $@ + +mksquashfs.o: mksquashfs.c squashfs_fs.h mksquashfs.h sort.h squashfs_swap.h \ + xattr.h pseudo.h compressor.h + +read_fs.o: read_fs.c squashfs_fs.h read_fs.h squashfs_swap.h compressor.h \ + xattr.h + +sort.o: sort.c squashfs_fs.h sort.h mksquashfs.h + +swap.o: swap.c + +pseudo.o: pseudo.c pseudo.h + +compressor.o: compressor.c compressor.h squashfs_fs.h + +xattr.o: xattr.c xattr.h squashfs_fs.h squashfs_swap.h mksquashfs.h + +read_xattrs.o: read_xattrs.c xattr.h squashfs_fs.h squashfs_swap.h read_fs.h + +gzip_wrapper.o: gzip_wrapper.c compressor.h squashfs_fs.h + +lzma_wrapper.o: lzma_wrapper.c compressor.h squashfs_fs.h + +lzma_xz_wrapper.o: lzma_xz_wrapper.c compressor.h squashfs_fs.h + +lzo_wrapper.o: lzo_wrapper.c compressor.h squashfs_fs.h + +xz_wrapper.o: xz_wrapper.c compressor.h squashfs_fs.h + +unsquashfs: $(UNSQUASHFS_OBJS) + $(CC) $(LDFLAGS) $(EXTRA_LDFLAGS) $(UNSQUASHFS_OBJS) $(LIBS) -o $@ + +unsquashfs.o: unsquashfs.h unsquashfs.c squashfs_fs.h squashfs_swap.h \ + squashfs_compat.h xattr.h read_fs.h compressor.h + +unsquash-1.o: unsquashfs.h unsquash-1.c squashfs_fs.h squashfs_compat.h + +unsquash-2.o: unsquashfs.h unsquash-2.c squashfs_fs.h squashfs_compat.h + +unsquash-3.o: unsquashfs.h unsquash-3.c squashfs_fs.h squashfs_compat.h + +unsquash-4.o: unsquashfs.h unsquash-4.c squashfs_fs.h squashfs_swap.h \ + read_fs.h + +unsquashfs_xattr.o: unsquashfs_xattr.c unsquashfs.h squashfs_fs.h xattr.h + + +.PHONY: clean +clean: + -rm -f *.o mksquashfs unsquashfs + +.PHONY: install +install: mksquashfs unsquashfs + mkdir -p $(INSTALL_DIR) + cp mksquashfs $(INSTALL_DIR) + cp unsquashfs $(INSTALL_DIR) diff --git a/squashfs-tools/compressor.c b/squashfs-tools/compressor.c new file mode 100644 index 0000000..48b0a99 --- /dev/null +++ b/squashfs-tools/compressor.c @@ -0,0 +1,130 @@ +/* + * + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * compressor.c + */ + +#include +#include +#include "compressor.h" +#include "squashfs_fs.h" + +#ifndef GZIP_SUPPORT +static struct compressor gzip_comp_ops = { + NULL, NULL, NULL, NULL, NULL, NULL, ZLIB_COMPRESSION, "gzip", 0 +}; +#else +extern struct compressor gzip_comp_ops; +#endif + +#ifndef LZMA_SUPPORT +static struct compressor lzma_comp_ops = { + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, LZMA_COMPRESSION, + "lzma", 0 +}; +#else +extern struct compressor lzma_comp_ops; +#endif + +#ifndef LZO_SUPPORT +static struct compressor lzo_comp_ops = { + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, LZO_COMPRESSION, "lzo", + 0 +}; +#else +extern struct compressor lzo_comp_ops; +#endif + +#ifndef XZ_SUPPORT +static struct compressor xz_comp_ops = { + NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, XZ_COMPRESSION, "xz", 0 +}; +#else +extern struct compressor xz_comp_ops; +#endif + + +static struct compressor unknown_comp_ops = { + NULL, NULL, NULL , NULL, NULL, NULL, NULL, NULL, 0, "unknown", 0 +}; + + +struct compressor *compressor[] = { + &gzip_comp_ops, + &lzma_comp_ops, + &lzo_comp_ops, + &xz_comp_ops, + &unknown_comp_ops +}; + + +struct compressor *lookup_compressor(char *name) +{ + int i; + + for(i = 0; compressor[i]->id; i++) + if(strcmp(compressor[i]->name, name) == 0) + break; + + return compressor[i]; +} + + +struct compressor *lookup_compressor_id(int id) +{ + int i; + + for(i = 0; compressor[i]->id; i++) + if(id == compressor[i]->id) + break; + + return compressor[i]; +} + + +void display_compressors(char *indent, char *def_comp) +{ + int i; + + for(i = 0; compressor[i]->id; i++) + if(compressor[i]->supported) + fprintf(stderr, "%s\t%s%s\n", indent, + compressor[i]->name, + strcmp(compressor[i]->name, def_comp) == 0 ? + " (default)" : ""); +} + + +void display_compressor_usage(char *def_comp) +{ + int i; + + for(i = 0; compressor[i]->id; i++) + if(compressor[i]->supported) { + char *str = strcmp(compressor[i]->name, def_comp) == 0 ? + " (default)" : ""; + if(compressor[i]->usage) { + fprintf(stderr, "\t%s%s\n", + compressor[i]->name, str); + compressor[i]->usage(); + } else + fprintf(stderr, "\t%s (no options)%s\n", + compressor[i]->name, str); + } +} diff --git a/squashfs-tools/compressor.h b/squashfs-tools/compressor.h new file mode 100644 index 0000000..9c19416 --- /dev/null +++ b/squashfs-tools/compressor.h @@ -0,0 +1,98 @@ +/* + * + * Copyright (c) 2009, 2010, 2011 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * compressor.h + */ + +struct compressor { + int (*init)(void **, int, int); + int (*compress)(void *, void *, void *, int, int, int *); + int (*uncompress)(void *, void *, int, int, int *); + int (*options)(char **, int); + int (*options_post)(int); + void *(*dump_options)(int, int *); + int (*extract_options)(int, void *, int); + void (*usage)(); + int id; + char *name; + int supported; +}; + +extern struct compressor *lookup_compressor(char *); +extern struct compressor *lookup_compressor_id(int); +extern void display_compressors(char *, char *); +extern void display_compressor_usage(char *); + +static inline int compressor_init(struct compressor *comp, void **stream, + int block_size, int datablock) +{ + if(comp->init == NULL) + return 0; + return comp->init(stream, block_size, datablock); +} + + +static inline int compressor_compress(struct compressor *comp, void *strm, + void *dest, void *src, int size, int block_size, int *error) +{ + return comp->compress(strm, dest, src, size, block_size, error); +} + + +static inline int compressor_uncompress(struct compressor *comp, void *dest, + void *src, int size, int block_size, int *error) +{ + return comp->uncompress(dest, src, size, block_size, error); +} + + +static inline int compressor_options(struct compressor *comp, char *argv[], + int argc) +{ + if(comp->options == NULL) + return -1; + + return comp->options(argv, argc); +} + + +static inline int compressor_options_post(struct compressor *comp, int block_size) +{ + if(comp->options_post == NULL) + return 0; + return comp->options_post(block_size); +} + + +static inline void *compressor_dump_options(struct compressor *comp, + int block_size, int *size) +{ + if(comp->dump_options == NULL) + return NULL; + return comp->dump_options(block_size, size); +} + + +static inline int compressor_extract_options(struct compressor *comp, + int block_size, void *buffer, int size) +{ + if(comp->extract_options == NULL) + return size ? -1 : 0; + return comp->extract_options(block_size, buffer, size); +} diff --git a/squashfs-tools/gzip_wrapper.c b/squashfs-tools/gzip_wrapper.c new file mode 100644 index 0000000..1b2bb3a --- /dev/null +++ b/squashfs-tools/gzip_wrapper.c @@ -0,0 +1,112 @@ +/* + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * gzip_wrapper.c + */ + +#include +#include + +#include "squashfs_fs.h" +#include "compressor.h" + +static int gzip_init(void **strm, int block_size, int flags) +{ + int res; + z_stream *stream; + + stream = *strm = malloc(sizeof(z_stream)); + if(stream == NULL) + goto failed; + + stream->zalloc = Z_NULL; + stream->zfree = Z_NULL; + stream->opaque = 0; + + res = deflateInit(stream, 9); + if(res != Z_OK) + goto failed2; + + return 0; + +failed2: + free(stream); +failed: + return -1; +} + + +static int gzip_compress(void *strm, void *d, void *s, int size, int block_size, + int *error) +{ + int res; + z_stream *stream = strm; + + res = deflateReset(stream); + if(res != Z_OK) + goto failed; + + stream->next_in = s; + stream->avail_in = size; + stream->next_out = d; + stream->avail_out = block_size; + + res = deflate(stream, Z_FINISH); + if(res == Z_STREAM_END) + /* + * Success, return the compressed size. + */ + return (int) stream->total_out; + if(res == Z_OK) + /* + * Output buffer overflow. Return out of buffer space + */ + return 0; +failed: + /* + * All other errors return failure, with the compressor + * specific error code in *error + */ + *error = res; + return -1; +} + + +static int gzip_uncompress(void *d, void *s, int size, int block_size, int *error) +{ + int res; + unsigned long bytes = block_size; + + res = uncompress(d, &bytes, s, size); + + *error = res; + return res == Z_OK ? (int) bytes : -1; +} + + +struct compressor gzip_comp_ops = { + .init = gzip_init, + .compress = gzip_compress, + .uncompress = gzip_uncompress, + .options = NULL, + .usage = NULL, + .id = ZLIB_COMPRESSION, + .name = "gzip", + .supported = 1 +}; + diff --git a/squashfs-tools/lzma_wrapper.c b/squashfs-tools/lzma_wrapper.c new file mode 100644 index 0000000..af01442 --- /dev/null +++ b/squashfs-tools/lzma_wrapper.c @@ -0,0 +1,112 @@ +/* + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * lzma_wrapper.c + * + * Support for LZMA1 compression using LZMA SDK (4.65 used in + * development, other versions may work) http://www.7-zip.org/sdk.html + */ + +#include + +#include "squashfs_fs.h" +#include "compressor.h" + +#define LZMA_HEADER_SIZE (LZMA_PROPS_SIZE + 8) + +static int lzma_compress(void *strm, void *dest, void *src, int size, int block_size, + int *error) +{ + unsigned char *d = dest; + size_t props_size = LZMA_PROPS_SIZE, + outlen = block_size - LZMA_HEADER_SIZE; + int res; + + res = LzmaCompress(dest + LZMA_HEADER_SIZE, &outlen, src, size, dest, + &props_size, 5, block_size, 3, 0, 2, 32, 1); + + if(res == SZ_ERROR_OUTPUT_EOF) { + /* + * Output buffer overflow. Return out of buffer space error + */ + return 0; + } + + if(res != SZ_OK) { + /* + * All other errors return failure, with the compressor + * specific error code in *error + */ + *error = res; + return -1; + } + + /* + * Fill in the 8 byte little endian uncompressed size field in the + * LZMA header. 8 bytes is excessively large for squashfs but + * this is the standard LZMA header and which is expected by the kernel + * code + */ + d[LZMA_PROPS_SIZE] = size & 255; + d[LZMA_PROPS_SIZE + 1] = (size >> 8) & 255; + d[LZMA_PROPS_SIZE + 2] = (size >> 16) & 255; + d[LZMA_PROPS_SIZE + 3] = (size >> 24) & 255; + d[LZMA_PROPS_SIZE + 4] = 0; + d[LZMA_PROPS_SIZE + 5] = 0; + d[LZMA_PROPS_SIZE + 6] = 0; + d[LZMA_PROPS_SIZE + 7] = 0; + + /* + * Success, return the compressed size. Outlen returned by the LZMA + * compressor does not include the LZMA header space + */ + return outlen + LZMA_HEADER_SIZE; +} + + +static int lzma_uncompress(void *dest, void *src, int size, int block_size, + int *error) +{ + unsigned char *s = src; + size_t outlen, inlen = size - LZMA_HEADER_SIZE; + int res; + + outlen = s[LZMA_PROPS_SIZE] | + (s[LZMA_PROPS_SIZE + 1] << 8) | + (s[LZMA_PROPS_SIZE + 2] << 16) | + (s[LZMA_PROPS_SIZE + 3] << 24); + + res = LzmaUncompress(dest, &outlen, src + LZMA_HEADER_SIZE, &inlen, src, + LZMA_PROPS_SIZE); + + *error = res; + return res == SZ_OK ? outlen : -1; +} + + +struct compressor lzma_comp_ops = { + .init = NULL, + .compress = lzma_compress, + .uncompress = lzma_uncompress, + .options = NULL, + .usage = NULL, + .id = LZMA_COMPRESSION, + .name = "lzma", + .supported = 1 +}; + diff --git a/squashfs-tools/lzma_xz_wrapper.c b/squashfs-tools/lzma_xz_wrapper.c new file mode 100644 index 0000000..080d910 --- /dev/null +++ b/squashfs-tools/lzma_xz_wrapper.c @@ -0,0 +1,157 @@ +/* + * Copyright (c) 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * lzma_xz_wrapper.c + * + * Support for LZMA1 compression using XZ Utils liblzma http://tukaani.org/xz/ + */ + +#include +#include +#include + +#include "squashfs_fs.h" +#include "compressor.h" + +#define LZMA_PROPS_SIZE 5 +#define LZMA_UNCOMP_SIZE 8 +#define LZMA_HEADER_SIZE (LZMA_PROPS_SIZE + LZMA_UNCOMP_SIZE) + +#define LZMA_OPTIONS 5 +#define MEMLIMIT (32 * 1024 * 1024) + +static int lzma_compress(void *dummy, void *dest, void *src, int size, + int block_size, int *error) +{ + unsigned char *d = (unsigned char *) dest; + lzma_options_lzma opt; + lzma_stream strm = LZMA_STREAM_INIT; + int res; + + lzma_lzma_preset(&opt, LZMA_OPTIONS); + opt.dict_size = block_size; + + res = lzma_alone_encoder(&strm, &opt); + if(res != LZMA_OK) { + lzma_end(&strm); + goto failed; + } + + strm.next_out = dest; + strm.avail_out = block_size; + strm.next_in = src; + strm.avail_in = size; + + res = lzma_code(&strm, LZMA_FINISH); + lzma_end(&strm); + + if(res == LZMA_STREAM_END) { + /* + * Fill in the 8 byte little endian uncompressed size field in + * the LZMA header. 8 bytes is excessively large for squashfs + * but this is the standard LZMA header and which is expected by + * the kernel code + */ + + d[LZMA_PROPS_SIZE] = size & 255; + d[LZMA_PROPS_SIZE + 1] = (size >> 8) & 255; + d[LZMA_PROPS_SIZE + 2] = (size >> 16) & 255; + d[LZMA_PROPS_SIZE + 3] = (size >> 24) & 255; + d[LZMA_PROPS_SIZE + 4] = 0; + d[LZMA_PROPS_SIZE + 5] = 0; + d[LZMA_PROPS_SIZE + 6] = 0; + d[LZMA_PROPS_SIZE + 7] = 0; + + return (int) strm.total_out; + } + + if(res == LZMA_OK) + /* + * Output buffer overflow. Return out of buffer space + */ + return 0; + +failed: + /* + * All other errors return failure, with the compressor + * specific error code in *error + */ + *error = res; + return -1; +} + + +static int lzma_uncompress(void *dest, void *src, int size, int block_size, + int *error) +{ + lzma_stream strm = LZMA_STREAM_INIT; + int uncompressed_size = 0, res; + unsigned char lzma_header[LZMA_HEADER_SIZE]; + + res = lzma_alone_decoder(&strm, MEMLIMIT); + if(res != LZMA_OK) { + lzma_end(&strm); + goto failed; + } + + memcpy(lzma_header, src, LZMA_HEADER_SIZE); + uncompressed_size = lzma_header[LZMA_PROPS_SIZE] | + (lzma_header[LZMA_PROPS_SIZE + 1] << 8) | + (lzma_header[LZMA_PROPS_SIZE + 2] << 16) | + (lzma_header[LZMA_PROPS_SIZE + 3] << 24); + memset(lzma_header + LZMA_PROPS_SIZE, 255, LZMA_UNCOMP_SIZE); + + strm.next_out = dest; + strm.avail_out = block_size; + strm.next_in = lzma_header; + strm.avail_in = LZMA_HEADER_SIZE; + + res = lzma_code(&strm, LZMA_RUN); + + if(res != LZMA_OK || strm.avail_in != 0) { + lzma_end(&strm); + goto failed; + } + + strm.next_in = src + LZMA_HEADER_SIZE; + strm.avail_in = size - LZMA_HEADER_SIZE; + + res = lzma_code(&strm, LZMA_FINISH); + lzma_end(&strm); + + if(res == LZMA_STREAM_END || (res == LZMA_OK && + strm.total_out >= uncompressed_size && strm.avail_in == 0)) + return uncompressed_size; + +failed: + *error = res; + return -1; +} + + +struct compressor lzma_comp_ops = { + .init = NULL, + .compress = lzma_compress, + .uncompress = lzma_uncompress, + .options = NULL, + .usage = NULL, + .id = LZMA_COMPRESSION, + .name = "lzma", + .supported = 1 +}; + diff --git a/squashfs-tools/lzo_wrapper.c b/squashfs-tools/lzo_wrapper.c new file mode 100644 index 0000000..f76260d --- /dev/null +++ b/squashfs-tools/lzo_wrapper.c @@ -0,0 +1,122 @@ +/* + * Copyright (c) 2010 LG Electronics + * Chan Jeong + * + * All modifications Copyright (c) 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * lzo_wrapper.c + */ + +#include +#include + +#include +#include + +#include "squashfs_fs.h" +#include "compressor.h" + +/* worst-case expansion calculation during compression, + see LZO FAQ for more information */ +#define LZO_OUTPUT_BUFFER_SIZE(size) (size + (size/16) + 64 + 3) + +struct lzo_stream { + lzo_voidp wrkmem; + lzo_bytep out; +}; + + +static int squashfs_lzo_init(void **strm, int block_size, int flags) +{ + struct lzo_stream *stream; + + if((stream = *strm = malloc(sizeof(struct lzo_stream))) == NULL) + goto failed; + /* work memory for compression */ + if((stream->wrkmem = malloc(LZO1X_999_MEM_COMPRESS)) == NULL) + goto failed2; + /* temporal output buffer */ + if((stream->out = malloc(LZO_OUTPUT_BUFFER_SIZE(block_size))) == NULL) + goto failed3; + + return 0; + +failed3: + free(stream->wrkmem); +failed2: + free(stream); +failed: + return -1; +} + + +static int lzo_compress(void *strm, void *d, void *s, int size, int block_size, + int *error) +{ + int res; + lzo_uint outlen; + struct lzo_stream *stream = strm; + + res = lzo1x_999_compress(s, size, stream->out, &outlen, stream->wrkmem); + if(res != LZO_E_OK) + goto failed; + if(outlen >= size) + /* + * Output buffer overflow. Return out of buffer space + */ + return 0; + + /* + * Success, return the compressed size. + */ + memcpy(d, stream->out, outlen); + return outlen; + +failed: + /* + * All other errors return failure, with the compressor + * specific error code in *error + */ + *error = res; + return -1; +} + + +static int lzo_uncompress(void *d, void *s, int size, int block_size, int *error) +{ + int res; + lzo_uint bytes = block_size; + + res = lzo1x_decompress_safe(s, size, d, &bytes, NULL); + + *error = res; + return res == LZO_E_OK ? bytes : -1; +} + + +struct compressor lzo_comp_ops = { + .init = squashfs_lzo_init, + .compress = lzo_compress, + .uncompress = lzo_uncompress, + .options = NULL, + .usage = NULL, + .id = LZO_COMPRESSION, + .name = "lzo", + .supported = 1 +}; + diff --git a/squashfs-tools/mksquashfs.c b/squashfs-tools/mksquashfs.c new file mode 100644 index 0000000..58168f3 --- /dev/null +++ b/squashfs-tools/mksquashfs.c @@ -0,0 +1,5424 @@ +/* + * Create a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * mksquashfs.c + */ + +#define FALSE 0 +#define TRUE 1 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef linux +#define __BYTE_ORDER BYTE_ORDER +#define __BIG_ENDIAN BIG_ENDIAN +#define __LITTLE_ENDIAN LITTLE_ENDIAN +#include +#else +#include +#include +#endif + +#ifdef SQUASHFS_TRACE +#define TRACE(s, args...) \ + do { \ + if(progress_enabled) \ + printf("\n"); \ + printf("mksquashfs: "s, ## args); \ + } while(0) +#else +#define TRACE(s, args...) +#endif + +#define INFO(s, args...) \ + do {\ + if(!silent)\ + printf("mksquashfs: "s, ## args);\ + } while(0) + +#define ERROR(s, args...) \ + do {\ + pthread_mutex_lock(&progress_mutex); \ + if(progress_enabled) \ + fprintf(stderr, "\n"); \ + fprintf(stderr, s, ## args);\ + pthread_mutex_unlock(&progress_mutex); \ + } while(0) + +#define EXIT_MKSQUASHFS() \ + do {\ + if(restore)\ + restorefs();\ + if(delete && destination_file && !block_device)\ + unlink(destination_file);\ + exit(1);\ + } while(0) + +#define BAD_ERROR(s, args...) \ + do {\ + pthread_mutex_lock(&progress_mutex); \ + if(progress_enabled) \ + fprintf(stderr, "\n"); \ + fprintf(stderr, "FATAL ERROR:" s, ##args);\ + pthread_mutex_unlock(&progress_mutex); \ + EXIT_MKSQUASHFS();\ + } while(0) + +#include "squashfs_fs.h" +#include "squashfs_swap.h" +#include "mksquashfs.h" +#include "sort.h" +#include "pseudo.h" +#include "compressor.h" +#include "xattr.h" + +int delete = FALSE; +int fd; +int cur_uncompressed = 0, estimated_uncompressed = 0; +int columns; + +/* filesystem flags for building */ +int comp_opts = FALSE; +int no_xattrs = XATTR_DEF, noX = 0; +int duplicate_checking = 1, noF = 0, no_fragments = 0, always_use_fragments = 0; +int noI = 0, noD = 0; +int silent = TRUE; +long long global_uid = -1, global_gid = -1; +int exportable = TRUE; +int progress = TRUE; +int progress_enabled = FALSE; +int sparse_files = TRUE; +int old_exclude = TRUE; +int use_regex = FALSE; +int first_freelist = TRUE; + +/* superblock attributes */ +int block_size = SQUASHFS_FILE_SIZE, block_log; +unsigned int id_count = 0; +int file_count = 0, sym_count = 0, dev_count = 0, dir_count = 0, fifo_count = 0, + sock_count = 0; + +/* write position within data section */ +long long bytes = 0, total_bytes = 0; + +/* in memory directory table - possibly compressed */ +char *directory_table = NULL; +unsigned int directory_bytes = 0, directory_size = 0, total_directory_bytes = 0; + +/* cached directory table */ +char *directory_data_cache = NULL; +unsigned int directory_cache_bytes = 0, directory_cache_size = 0; + +/* in memory inode table - possibly compressed */ +char *inode_table = NULL; +unsigned int inode_bytes = 0, inode_size = 0, total_inode_bytes = 0; + +/* cached inode table */ +char *data_cache = NULL; +unsigned int cache_bytes = 0, cache_size = 0, inode_count = 0; + +/* inode lookup table */ +squashfs_inode *inode_lookup_table = NULL; + +/* in memory directory data */ +#define I_COUNT_SIZE 128 +#define DIR_ENTRIES 32 +#define INODE_HASH_SIZE 65536 +#define INODE_HASH_MASK (INODE_HASH_SIZE - 1) +#define INODE_HASH(dev, ino) (ino & INODE_HASH_MASK) + +struct cached_dir_index { + struct squashfs_dir_index index; + char *name; +}; + +struct directory { + unsigned int start_block; + unsigned int size; + unsigned char *buff; + unsigned char *p; + unsigned int entry_count; + unsigned char *entry_count_p; + unsigned int i_count; + unsigned int i_size; + struct cached_dir_index *index; + unsigned char *index_count_p; + unsigned int inode_number; +}; + +struct inode_info *inode_info[INODE_HASH_SIZE]; + +/* hash tables used to do fast duplicate searches in duplicate check */ +struct file_info *dupl[65536]; +int dup_files = 0; + +/* exclude file handling */ +/* list of exclude dirs/files */ +struct exclude_info { + dev_t st_dev; + ino_t st_ino; +}; + +#define EXCLUDE_SIZE 8192 +int exclude = 0; +struct exclude_info *exclude_paths = NULL; +int old_excluded(char *filename, struct stat *buf); + +struct path_entry { + char *name; + regex_t *preg; + struct pathname *paths; +}; + +struct pathname { + int names; + struct path_entry *name; +}; + +struct pathnames { + int count; + struct pathname *path[0]; +}; +#define PATHS_ALLOC_SIZE 10 + +struct pathnames *paths = NULL; +struct pathname *path = NULL; +struct pathname *stickypath = NULL; +int excluded(struct pathnames *paths, char *name, struct pathnames **new); + +/* fragment block data structures */ +int fragments = 0; +struct file_buffer *fragment_data = NULL; +int fragment_size = 0; + +struct fragment { + unsigned int index; + int offset; + int size; +}; + +#define FRAG_SIZE 32768 +#define FRAG_INDEX (1LL << 32) + +struct squashfs_fragment_entry *fragment_table = NULL; +int fragments_outstanding = 0; + +/* current inode number for directories and non directories */ +unsigned int dir_inode_no = 1; +unsigned int inode_no = 0; +unsigned int root_inode_number = 0; + +/* list of source dirs/files */ +int source = 0; +char **source_path; + +/* list of root directory entries read from original filesystem */ +int old_root_entries = 0; +struct old_root_entry_info { + char *name; + struct inode_info inode; +}; +struct old_root_entry_info *old_root_entry; + +/* in memory file info */ +struct file_info { + long long file_size; + long long bytes; + unsigned short checksum; + unsigned short fragment_checksum; + long long start; + unsigned int *block_list; + struct file_info *next; + struct fragment *fragment; + char checksum_flag; +}; + +/* count of how many times SIGINT or SIGQUIT has been sent */ +int interrupted = 0; + +/* flag if we're restoring existing filesystem */ +int restoring = 0; + +/* restore orignal filesystem state if appending to existing filesystem is + * cancelled */ +jmp_buf env; +char *sdata_cache, *sdirectory_data_cache, *sdirectory_compressed; + +long long sbytes, stotal_bytes; + +unsigned int sinode_bytes, scache_bytes, sdirectory_bytes, + sdirectory_cache_bytes, sdirectory_compressed_bytes, + stotal_inode_bytes, stotal_directory_bytes, + sinode_count = 0, sfile_count, ssym_count, sdev_count, + sdir_count, sfifo_count, ssock_count, sdup_files; +int sfragments; +int restore = 0; +int threads; + +/* flag whether destination file is a block device */ +int block_device = 0; + +/* flag indicating whether files are sorted using sort list(s) */ +int sorted = 0; + +/* save destination file name for deleting on error */ +char *destination_file = NULL; + +/* recovery file for abnormal exit on appending */ +char recovery_file[1024] = ""; +int recover = TRUE; + +/* struct describing a cache entry passed between threads */ +struct file_buffer { + struct cache *cache; + int keep; + long long file_size; + long long index; + long long block; + long long sequence; + int size; + int c_byte; + int used; + int fragment; + int error; + struct file_buffer *hash_next; + struct file_buffer *hash_prev; + struct file_buffer *free_next; + struct file_buffer *free_prev; + struct file_buffer *next; + char data[0]; +}; + + +/* struct describing queues used to pass data between threads */ +struct queue { + int size; + int readp; + int writep; + pthread_mutex_t mutex; + pthread_cond_t empty; + pthread_cond_t full; + void **data; +}; + + +/* in memory uid tables */ +#define ID_ENTRIES 256 +#define ID_HASH(id) (id & (ID_ENTRIES - 1)) +#define ISA_UID 1 +#define ISA_GID 2 +struct id { + unsigned int id; + int index; + char flags; + struct id *next; +}; +struct id *id_hash_table[ID_ENTRIES]; +struct id *id_table[SQUASHFS_IDS], *sid_table[SQUASHFS_IDS]; +unsigned int uid_count = 0, guid_count = 0; +unsigned int sid_count = 0, suid_count = 0, sguid_count = 0; + +struct cache *reader_buffer, *writer_buffer, *fragment_buffer; +struct queue *to_reader, *from_reader, *to_writer, *from_writer, *from_deflate, + *to_frag; +pthread_t *thread, *deflator_thread, *frag_deflator_thread, progress_thread; +pthread_mutex_t fragment_mutex; +pthread_cond_t fragment_waiting; +pthread_mutex_t pos_mutex; +pthread_mutex_t progress_mutex; +pthread_cond_t progress_wait; +int rotate = 0; +struct pseudo *pseudo = NULL; + +/* user options that control parallelisation */ +int processors = -1; +/* default size of output buffer in Mbytes */ +#define WRITER_BUFFER_DEFAULT 512 +/* default size of input buffer in Mbytes */ +#define READER_BUFFER_DEFAULT 64 +/* default size of fragment buffer in Mbytes */ +#define FRAGMENT_BUFFER_DEFAULT 64 +int writer_buffer_size; + +/* compression operations */ +static struct compressor *comp; +int compressor_opts_parsed = 0; +void *stream = NULL; + +/* xattr stats */ +unsigned int xattr_bytes = 0, total_xattr_bytes = 0; + +char *read_from_disk(long long start, unsigned int avail_bytes); +void add_old_root_entry(char *name, squashfs_inode inode, int inode_number, + int type); +extern struct compressor *read_super(int fd, struct squashfs_super_block *sBlk, + char *source); +extern long long read_filesystem(char *root_name, int fd, + struct squashfs_super_block *sBlk, char **cinode_table, char **data_cache, + char **cdirectory_table, char **directory_data_cache, + unsigned int *last_directory_block, unsigned int *inode_dir_offset, + unsigned int *inode_dir_file_size, unsigned int *root_inode_size, + unsigned int *inode_dir_start_block, int *file_count, int *sym_count, + int *dev_count, int *dir_count, int *fifo_count, int *sock_count, + long long *uncompressed_file, unsigned int *uncompressed_inode, + unsigned int *uncompressed_directory, + unsigned int *inode_dir_inode_number, + unsigned int *inode_dir_parent_inode, + void (push_directory_entry)(char *, squashfs_inode, int, int), + struct squashfs_fragment_entry **fragment_table, + squashfs_inode **inode_lookup_table); +extern int read_sort_file(char *filename, int source, char *source_path[]); +extern void sort_files_and_write(struct dir_info *dir); +struct file_info *duplicate(long long file_size, long long bytes, + unsigned int **block_list, long long *start, struct fragment **fragment, + struct file_buffer *file_buffer, int blocks, unsigned short checksum, + unsigned short fragment_checksum, int checksum_flag); +struct dir_info *dir_scan1(char *, struct pathnames *, int (_readdir)(char *, + char *, struct dir_info *)); +struct dir_info *dir_scan2(struct dir_info *dir, struct pseudo *pseudo); +void dir_scan3(squashfs_inode *inode, struct dir_info *dir_info); +struct file_info *add_non_dup(long long file_size, long long bytes, + unsigned int *block_list, long long start, struct fragment *fragment, + unsigned short checksum, unsigned short fragment_checksum, + int checksum_flag); +extern int generate_file_priorities(struct dir_info *dir, int priority, + struct stat *buf); +extern struct priority_entry *priority_list[65536]; +void progress_bar(long long current, long long max, int columns); +long long generic_write_table(int, void *, int, void *, int); +void restorefs(); + + +struct queue *queue_init(int size) +{ + struct queue *queue = malloc(sizeof(struct queue)); + + if(queue == NULL) + goto failed; + + queue->data = malloc(sizeof(void *) * (size + 1)); + if(queue->data == NULL) { + free(queue); + goto failed; + } + + queue->size = size + 1; + queue->readp = queue->writep = 0; + pthread_mutex_init(&queue->mutex, NULL); + pthread_cond_init(&queue->empty, NULL); + pthread_cond_init(&queue->full, NULL); + + return queue; + +failed: + BAD_ERROR("Out of memory in queue_init\n"); +} + + +void queue_put(struct queue *queue, void *data) +{ + int nextp; + + pthread_mutex_lock(&queue->mutex); + + while((nextp = (queue->writep + 1) % queue->size) == queue->readp) + pthread_cond_wait(&queue->full, &queue->mutex); + + queue->data[queue->writep] = data; + queue->writep = nextp; + pthread_cond_signal(&queue->empty); + pthread_mutex_unlock(&queue->mutex); +} + + +void *queue_get(struct queue *queue) +{ + void *data; + pthread_mutex_lock(&queue->mutex); + + while(queue->readp == queue->writep) + pthread_cond_wait(&queue->empty, &queue->mutex); + + data = queue->data[queue->readp]; + queue->readp = (queue->readp + 1) % queue->size; + pthread_cond_signal(&queue->full); + pthread_mutex_unlock(&queue->mutex); + + return data; +} + + +/* Cache status struct. Caches are used to keep + track of memory buffers passed between different threads */ +struct cache { + int max_buffers; + int count; + int buffer_size; + pthread_mutex_t mutex; + pthread_cond_t wait_for_free; + struct file_buffer *free_list; + struct file_buffer *hash_table[65536]; +}; + + +#define INSERT_LIST(NAME, TYPE) \ +void insert_##NAME##_list(TYPE **list, TYPE *entry) { \ + if(*list) { \ + entry->NAME##_next = *list; \ + entry->NAME##_prev = (*list)->NAME##_prev; \ + (*list)->NAME##_prev->NAME##_next = entry; \ + (*list)->NAME##_prev = entry; \ + } else { \ + *list = entry; \ + entry->NAME##_prev = entry->NAME##_next = entry; \ + } \ +} + + +#define REMOVE_LIST(NAME, TYPE) \ +void remove_##NAME##_list(TYPE **list, TYPE *entry) { \ + if(entry->NAME##_prev == entry && entry->NAME##_next == entry) { \ + /* only this entry in the list */ \ + *list = NULL; \ + } else if(entry->NAME##_prev != NULL && entry->NAME##_next != NULL) { \ + /* more than one entry in the list */ \ + entry->NAME##_next->NAME##_prev = entry->NAME##_prev; \ + entry->NAME##_prev->NAME##_next = entry->NAME##_next; \ + if(*list == entry) \ + *list = entry->NAME##_next; \ + } \ + entry->NAME##_prev = entry->NAME##_next = NULL; \ +} + + +#define CALCULATE_HASH(start) (start & 0xffff) \ + + +/* Called with the cache mutex held */ +void insert_hash_table(struct cache *cache, struct file_buffer *entry) +{ + int hash = CALCULATE_HASH(entry->index); + + entry->hash_next = cache->hash_table[hash]; + cache->hash_table[hash] = entry; + entry->hash_prev = NULL; + if(entry->hash_next) + entry->hash_next->hash_prev = entry; +} + + +/* Called with the cache mutex held */ +void remove_hash_table(struct cache *cache, struct file_buffer *entry) +{ + if(entry->hash_prev) + entry->hash_prev->hash_next = entry->hash_next; + else + cache->hash_table[CALCULATE_HASH(entry->index)] = + entry->hash_next; + if(entry->hash_next) + entry->hash_next->hash_prev = entry->hash_prev; + + entry->hash_prev = entry->hash_next = NULL; +} + + +/* Called with the cache mutex held */ +INSERT_LIST(free, struct file_buffer) + +/* Called with the cache mutex held */ +REMOVE_LIST(free, struct file_buffer) + + +struct cache *cache_init(int buffer_size, int max_buffers) +{ + struct cache *cache = malloc(sizeof(struct cache)); + + if(cache == NULL) + BAD_ERROR("Out of memory in cache_init\n"); + + cache->max_buffers = max_buffers; + cache->buffer_size = buffer_size; + cache->count = 0; + cache->free_list = NULL; + memset(cache->hash_table, 0, sizeof(struct file_buffer *) * 65536); + pthread_mutex_init(&cache->mutex, NULL); + pthread_cond_init(&cache->wait_for_free, NULL); + + return cache; +} + + +struct file_buffer *cache_lookup(struct cache *cache, long long index) +{ + /* Lookup block in the cache, if found return with usage count + * incremented, if not found return NULL */ + int hash = CALCULATE_HASH(index); + struct file_buffer *entry; + + pthread_mutex_lock(&cache->mutex); + + for(entry = cache->hash_table[hash]; entry; entry = entry->hash_next) + if(entry->index == index) + break; + + if(entry) { + /* found the block in the cache, increment used count and + * if necessary remove from free list so it won't disappear + */ + entry->used ++; + remove_free_list(&cache->free_list, entry); + } + + pthread_mutex_unlock(&cache->mutex); + + return entry; +} + + +struct file_buffer *cache_get(struct cache *cache, long long index, int keep) +{ + /* Get a free block out of the cache indexed on index. */ + struct file_buffer *entry; + + pthread_mutex_lock(&cache->mutex); + + while(1) { + /* first try to get a block from the free list */ + if(first_freelist && cache->free_list) { + /* a block on the free_list is a "keep" block */ + entry = cache->free_list; + remove_free_list(&cache->free_list, entry); + remove_hash_table(cache, entry); + break; + } else if(cache->count < cache->max_buffers) { + /* next try to allocate new block */ + entry = malloc(sizeof(struct file_buffer) + + cache->buffer_size); + if(entry == NULL) + goto failed; + entry->cache = cache; + entry->free_prev = entry->free_next = NULL; + cache->count ++; + break; + } else if(!first_freelist && cache->free_list) { + /* a block on the free_list is a "keep" block */ + entry = cache->free_list; + remove_free_list(&cache->free_list, entry); + remove_hash_table(cache, entry); + break; + } else + /* wait for a block */ + pthread_cond_wait(&cache->wait_for_free, &cache->mutex); + } + + /* initialise block and if a keep block insert into the hash table */ + entry->used = 1; + entry->error = FALSE; + entry->keep = keep; + if(keep) { + entry->index = index; + insert_hash_table(cache, entry); + } + pthread_mutex_unlock(&cache->mutex); + + return entry; + +failed: + pthread_mutex_unlock(&cache->mutex); + BAD_ERROR("Out of memory in cache_get\n"); +} + + +void cache_rehash(struct file_buffer *entry, long long index) +{ + struct cache *cache = entry->cache; + + pthread_mutex_lock(&cache->mutex); + if(entry->keep) + remove_hash_table(cache, entry); + entry->keep = TRUE; + entry->index = index; + insert_hash_table(cache, entry); + pthread_mutex_unlock(&cache->mutex); +} + + +void cache_block_put(struct file_buffer *entry) +{ + struct cache *cache; + + /* finished with this cache entry, once the usage count reaches zero it + * can be reused and if a keep block put onto the free list. As keep + * blocks remain accessible via the hash table they can be found + * getting a new lease of life before they are reused. */ + + if(entry == NULL) + return; + + cache = entry->cache; + + pthread_mutex_lock(&cache->mutex); + + entry->used --; + if(entry->used == 0) { + if(entry->keep) + insert_free_list(&cache->free_list, entry); + else { + free(entry); + cache->count --; + } + + /* One or more threads may be waiting on this block */ + pthread_cond_signal(&cache->wait_for_free); + } + + pthread_mutex_unlock(&cache->mutex); +} + + +#define MKINODE(A) ((squashfs_inode)(((squashfs_inode) inode_bytes << 16) \ + + (((char *)A) - data_cache))) + + +inline void inc_progress_bar() +{ + cur_uncompressed ++; +} + + +inline void update_progress_bar() +{ + pthread_mutex_lock(&progress_mutex); + pthread_cond_signal(&progress_wait); + pthread_mutex_unlock(&progress_mutex); +} + + +inline void waitforthread(int i) +{ + TRACE("Waiting for thread %d\n", i); + while(thread[i] != 0) + sched_yield(); +} + + +void restorefs() +{ + int i; + + if(thread == NULL || thread[0] == 0) + return; + + if(restoring++) + /* + * Recursive failure when trying to restore filesystem! + * Nothing to do except to exit, otherwise we'll just appear + * to hang. The user should be able to restore from the + * recovery file (which is why it was added, in case of + * catastrophic failure in Mksquashfs) + */ + exit(1); + + ERROR("Exiting - restoring original filesystem!\n\n"); + + for(i = 0; i < 2 + processors * 2; i++) + if(thread[i]) + pthread_kill(thread[i], SIGUSR1); + for(i = 0; i < 2 + processors * 2; i++) + waitforthread(i); + TRACE("All threads in signal handler\n"); + bytes = sbytes; + memcpy(data_cache, sdata_cache, cache_bytes = scache_bytes); + memcpy(directory_data_cache, sdirectory_data_cache, + sdirectory_cache_bytes); + directory_cache_bytes = sdirectory_cache_bytes; + inode_bytes = sinode_bytes; + directory_bytes = sdirectory_bytes; + memcpy(directory_table + directory_bytes, sdirectory_compressed, + sdirectory_compressed_bytes); + directory_bytes += sdirectory_compressed_bytes; + total_bytes = stotal_bytes; + total_inode_bytes = stotal_inode_bytes; + total_directory_bytes = stotal_directory_bytes; + inode_count = sinode_count; + file_count = sfile_count; + sym_count = ssym_count; + dev_count = sdev_count; + dir_count = sdir_count; + fifo_count = sfifo_count; + sock_count = ssock_count; + dup_files = sdup_files; + fragments = sfragments; + fragment_size = 0; + id_count = sid_count; + restore_xattrs(); + longjmp(env, 1); +} + + +void sighandler() +{ + if(++interrupted > 2) + return; + if(interrupted == 2) + restorefs(); + else { + ERROR("Interrupting will restore original filesystem!\n"); + ERROR("Interrupt again to quit\n"); + } +} + + +void sighandler2() +{ + EXIT_MKSQUASHFS(); +} + + +void sigusr1_handler() +{ + int i; + sigset_t sigmask; + pthread_t thread_id = pthread_self(); + + for(i = 0; i < (2 + processors * 2) && thread[i] != thread_id; i++); + thread[i] = (pthread_t) 0; + + TRACE("Thread %d(%p) in sigusr1_handler\n", i, &thread_id); + + sigemptyset(&sigmask); + sigaddset(&sigmask, SIGINT); + sigaddset(&sigmask, SIGQUIT); + sigaddset(&sigmask, SIGUSR1); + while(1) { + sigsuspend(&sigmask); + TRACE("After wait in sigusr1_handler :(\n"); + } +} + + +void sigwinch_handler() +{ + struct winsize winsize; + + if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { + if(isatty(STDOUT_FILENO)) + printf("TIOCGWINSZ ioctl failed, defaulting to 80 " + "columns\n"); + columns = 80; + } else + columns = winsize.ws_col; +} + + +void sigalrm_handler() +{ + rotate = (rotate + 1) % 4; +} + + +int mangle2(void *strm, char *d, char *s, int size, + int block_size, int uncompressed, int data_block) +{ + int error, c_byte = 0; + + if(!uncompressed) { + c_byte = compressor_compress(comp, strm, d, s, size, block_size, + &error); + if(c_byte == -1) + BAD_ERROR("mangle2:: %s compress failed with error " + "code %d\n", comp->name, error); + } + + if(c_byte == 0 || c_byte >= size) { + memcpy(d, s, size); + return size | (data_block ? SQUASHFS_COMPRESSED_BIT_BLOCK : + SQUASHFS_COMPRESSED_BIT); + } + + return c_byte; +} + + +int mangle(char *d, char *s, int size, int block_size, + int uncompressed, int data_block) +{ + return mangle2(stream, d, s, size, block_size, uncompressed, + data_block); +} + + +void *get_inode(int req_size) +{ + int data_space; + unsigned short c_byte; + + while(cache_bytes >= SQUASHFS_METADATA_SIZE) { + if((inode_size - inode_bytes) < + ((SQUASHFS_METADATA_SIZE << 1)) + 2) { + void *it = realloc(inode_table, inode_size + + (SQUASHFS_METADATA_SIZE << 1) + 2); + if(it == NULL) { + goto failed; + } + inode_table = it; + inode_size += (SQUASHFS_METADATA_SIZE << 1) + 2; + } + + c_byte = mangle(inode_table + inode_bytes + BLOCK_OFFSET, + data_cache, SQUASHFS_METADATA_SIZE, + SQUASHFS_METADATA_SIZE, noI, 0); + TRACE("Inode block @ 0x%x, size %d\n", inode_bytes, c_byte); + SQUASHFS_SWAP_SHORTS(&c_byte, inode_table + inode_bytes, 1); + inode_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; + total_inode_bytes += SQUASHFS_METADATA_SIZE + BLOCK_OFFSET; + memmove(data_cache, data_cache + SQUASHFS_METADATA_SIZE, + cache_bytes - SQUASHFS_METADATA_SIZE); + cache_bytes -= SQUASHFS_METADATA_SIZE; + } + + data_space = (cache_size - cache_bytes); + if(data_space < req_size) { + int realloc_size = cache_size == 0 ? + ((req_size + SQUASHFS_METADATA_SIZE) & + ~(SQUASHFS_METADATA_SIZE - 1)) : req_size - + data_space; + + void *dc = realloc(data_cache, cache_size + + realloc_size); + if(dc == NULL) { + goto failed; + } + cache_size += realloc_size; + data_cache = dc; + } + + cache_bytes += req_size; + + return data_cache + cache_bytes - req_size; + +failed: + BAD_ERROR("Out of memory in inode table reallocation!\n"); +} + + +int read_bytes(int fd, void *buff, int bytes) +{ + int res, count; + + for(count = 0; count < bytes; count += res) { + res = read(fd, buff + count, bytes - count); + if(res < 1) { + if(res == 0) + goto bytes_read; + else if(errno != EINTR) { + ERROR("Read failed because %s\n", + strerror(errno)); + return -1; + } else + res = 0; + } + } + +bytes_read: + return count; +} + + +int read_fs_bytes(int fd, long long byte, int bytes, void *buff) +{ + off_t off = byte; + + TRACE("read_fs_bytes: reading from position 0x%llx, bytes %d\n", + byte, bytes); + + pthread_mutex_lock(&pos_mutex); + if(lseek(fd, off, SEEK_SET) == -1) { + ERROR("Lseek on destination failed because %s\n", + strerror(errno)); + goto failed; + } + + if(read_bytes(fd, buff, bytes) < bytes) { + ERROR("Read on destination failed\n"); + goto failed; + } + + pthread_mutex_unlock(&pos_mutex); + return 1; + +failed: + pthread_mutex_unlock(&pos_mutex); + return 0; +} + + +int write_bytes(int fd, void *buff, int bytes) +{ + int res, count; + + for(count = 0; count < bytes; count += res) { + res = write(fd, buff + count, bytes - count); + if(res == -1) { + if(errno != EINTR) { + ERROR("Write failed because %s\n", + strerror(errno)); + return -1; + } + res = 0; + } + } + + return 0; +} + + +void write_destination(int fd, long long byte, int bytes, void *buff) +{ + off_t off = byte; + + if(!restoring) + pthread_mutex_lock(&pos_mutex); + + if(lseek(fd, off, SEEK_SET) == -1) + BAD_ERROR("Lseek on destination failed because %s\n", + strerror(errno)); + + if(write_bytes(fd, buff, bytes) == -1) + BAD_ERROR("Write on destination failed\n"); + + if(!restoring) + pthread_mutex_unlock(&pos_mutex); +} + + +long long write_inodes() +{ + unsigned short c_byte; + int avail_bytes; + char *datap = data_cache; + long long start_bytes = bytes; + + while(cache_bytes) { + if(inode_size - inode_bytes < + ((SQUASHFS_METADATA_SIZE << 1) + 2)) { + void *it = realloc(inode_table, inode_size + + ((SQUASHFS_METADATA_SIZE << 1) + 2)); + if(it == NULL) { + BAD_ERROR("Out of memory in inode table " + "reallocation!\n"); + } + inode_size += (SQUASHFS_METADATA_SIZE << 1) + 2; + inode_table = it; + } + avail_bytes = cache_bytes > SQUASHFS_METADATA_SIZE ? + SQUASHFS_METADATA_SIZE : cache_bytes; + c_byte = mangle(inode_table + inode_bytes + BLOCK_OFFSET, datap, + avail_bytes, SQUASHFS_METADATA_SIZE, noI, 0); + TRACE("Inode block @ 0x%x, size %d\n", inode_bytes, c_byte); + SQUASHFS_SWAP_SHORTS(&c_byte, inode_table + inode_bytes, 1); + inode_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; + total_inode_bytes += avail_bytes + BLOCK_OFFSET; + datap += avail_bytes; + cache_bytes -= avail_bytes; + } + + write_destination(fd, bytes, inode_bytes, inode_table); + bytes += inode_bytes; + + return start_bytes; +} + + +long long write_directories() +{ + unsigned short c_byte; + int avail_bytes; + char *directoryp = directory_data_cache; + long long start_bytes = bytes; + + while(directory_cache_bytes) { + if(directory_size - directory_bytes < + ((SQUASHFS_METADATA_SIZE << 1) + 2)) { + void *dt = realloc(directory_table, + directory_size + ((SQUASHFS_METADATA_SIZE << 1) + + 2)); + if(dt == NULL) { + BAD_ERROR("Out of memory in directory table " + "reallocation!\n"); + } + directory_size += (SQUASHFS_METADATA_SIZE << 1) + 2; + directory_table = dt; + } + avail_bytes = directory_cache_bytes > SQUASHFS_METADATA_SIZE ? + SQUASHFS_METADATA_SIZE : directory_cache_bytes; + c_byte = mangle(directory_table + directory_bytes + + BLOCK_OFFSET, directoryp, avail_bytes, + SQUASHFS_METADATA_SIZE, noI, 0); + TRACE("Directory block @ 0x%x, size %d\n", directory_bytes, + c_byte); + SQUASHFS_SWAP_SHORTS(&c_byte, + directory_table + directory_bytes, 1); + directory_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + + BLOCK_OFFSET; + total_directory_bytes += avail_bytes + BLOCK_OFFSET; + directoryp += avail_bytes; + directory_cache_bytes -= avail_bytes; + } + write_destination(fd, bytes, directory_bytes, directory_table); + bytes += directory_bytes; + + return start_bytes; +} + + +long long write_id_table() +{ + unsigned int id_bytes = SQUASHFS_ID_BYTES(id_count); + unsigned int p[id_count]; + int i; + + TRACE("write_id_table: ids %d, id_bytes %d\n", id_count, id_bytes); + for(i = 0; i < id_count; i++) { + TRACE("write_id_table: id index %d, id %d", i, id_table[i]->id); + SQUASHFS_SWAP_INTS(&id_table[i]->id, p + i, 1); + } + + return generic_write_table(id_bytes, p, 0, NULL, noI); +} + + +struct id *get_id(unsigned int id) +{ + int hash = ID_HASH(id); + struct id *entry = id_hash_table[hash]; + + for(; entry; entry = entry->next) + if(entry->id == id) + break; + + return entry; +} + + +struct id *create_id(unsigned int id) +{ + int hash = ID_HASH(id); + struct id *entry = malloc(sizeof(struct id)); + if(entry == NULL) + BAD_ERROR("Out of memory in create_id\n"); + entry->id = id; + entry->index = id_count ++; + entry->flags = 0; + entry->next = id_hash_table[hash]; + id_hash_table[hash] = entry; + id_table[entry->index] = entry; + return entry; +} + + +unsigned int get_uid(unsigned int uid) +{ + struct id *entry = get_id(uid); + + if(entry == NULL) { + if(id_count == SQUASHFS_IDS) + BAD_ERROR("Out of uids!\n"); + entry = create_id(uid); + } + + if((entry->flags & ISA_UID) == 0) { + entry->flags |= ISA_UID; + uid_count ++; + } + + return entry->index; +} + + +unsigned int get_guid(unsigned int guid) +{ + struct id *entry = get_id(guid); + + if(entry == NULL) { + if(id_count == SQUASHFS_IDS) + BAD_ERROR("Out of gids!\n"); + entry = create_id(guid); + } + + if((entry->flags & ISA_GID) == 0) { + entry->flags |= ISA_GID; + guid_count ++; + } + + return entry->index; +} + + +int create_inode(squashfs_inode *i_no, struct dir_info *dir_info, + struct dir_ent *dir_ent, int type, long long byte_size, + long long start_block, unsigned int offset, unsigned int *block_list, + struct fragment *fragment, struct directory *dir_in, long long sparse) +{ + struct stat *buf = &dir_ent->inode->buf; + union squashfs_inode_header inode_header; + struct squashfs_base_inode_header *base = &inode_header.base; + void *inode; + char *filename = dir_ent->pathname; + int nlink = dir_ent->inode->nlink; + int inode_number = type == SQUASHFS_DIR_TYPE ? + dir_ent->inode->inode_number : + dir_ent->inode->inode_number + dir_inode_no; + int xattr = read_xattrs(dir_ent); + + switch(type) { + case SQUASHFS_FILE_TYPE: + if(dir_ent->inode->nlink > 1 || + byte_size >= (1LL << 32) || + start_block >= (1LL << 32) || + sparse || IS_XATTR(xattr)) + type = SQUASHFS_LREG_TYPE; + break; + case SQUASHFS_DIR_TYPE: + if(dir_info->dir_is_ldir || IS_XATTR(xattr)) + type = SQUASHFS_LDIR_TYPE; + break; + case SQUASHFS_SYMLINK_TYPE: + if(IS_XATTR(xattr)) + type = SQUASHFS_LSYMLINK_TYPE; + break; + case SQUASHFS_BLKDEV_TYPE: + if(IS_XATTR(xattr)) + type = SQUASHFS_LBLKDEV_TYPE; + break; + case SQUASHFS_CHRDEV_TYPE: + if(IS_XATTR(xattr)) + type = SQUASHFS_LCHRDEV_TYPE; + break; + case SQUASHFS_FIFO_TYPE: + if(IS_XATTR(xattr)) + type = SQUASHFS_LFIFO_TYPE; + break; + case SQUASHFS_SOCKET_TYPE: + if(IS_XATTR(xattr)) + type = SQUASHFS_LSOCKET_TYPE; + break; + } + + base->mode = SQUASHFS_MODE(buf->st_mode); + base->uid = get_uid((unsigned int) global_uid == -1 ? + buf->st_uid : global_uid); + base->inode_type = type; + base->guid = get_guid((unsigned int) global_gid == -1 ? + buf->st_gid : global_gid); + base->mtime = buf->st_mtime; + base->inode_number = inode_number; + + if(type == SQUASHFS_FILE_TYPE) { + int i; + struct squashfs_reg_inode_header *reg = &inode_header.reg; + size_t off = offsetof(struct squashfs_reg_inode_header, block_list); + + inode = get_inode(sizeof(*reg) + offset * sizeof(unsigned int)); + reg->file_size = byte_size; + reg->start_block = start_block; + reg->fragment = fragment->index; + reg->offset = fragment->offset; + SQUASHFS_SWAP_REG_INODE_HEADER(reg, inode); + SQUASHFS_SWAP_INTS(block_list, inode + off, offset); + TRACE("File inode, file_size %lld, start_block 0x%llx, blocks " + "%d, fragment %d, offset %d, size %d\n", byte_size, + start_block, offset, fragment->index, fragment->offset, + fragment->size); + for(i = 0; i < offset; i++) + TRACE("Block %d, size %d\n", i, block_list[i]); + } + else if(type == SQUASHFS_LREG_TYPE) { + int i; + struct squashfs_lreg_inode_header *reg = &inode_header.lreg; + size_t off = offsetof(struct squashfs_lreg_inode_header, block_list); + + inode = get_inode(sizeof(*reg) + offset * sizeof(unsigned int)); + reg->nlink = nlink; + reg->file_size = byte_size; + reg->start_block = start_block; + reg->fragment = fragment->index; + reg->offset = fragment->offset; + if(sparse && sparse >= byte_size) + sparse = byte_size - 1; + reg->sparse = sparse; + reg->xattr = xattr; + SQUASHFS_SWAP_LREG_INODE_HEADER(reg, inode); + SQUASHFS_SWAP_INTS(block_list, inode + off, offset); + TRACE("Long file inode, file_size %lld, start_block 0x%llx, " + "blocks %d, fragment %d, offset %d, size %d, nlink %d" + "\n", byte_size, start_block, offset, fragment->index, + fragment->offset, fragment->size, nlink); + for(i = 0; i < offset; i++) + TRACE("Block %d, size %d\n", i, block_list[i]); + } + else if(type == SQUASHFS_LDIR_TYPE) { + int i; + unsigned char *p; + struct squashfs_ldir_inode_header *dir = &inode_header.ldir; + struct cached_dir_index *index = dir_in->index; + unsigned int i_count = dir_in->i_count; + unsigned int i_size = dir_in->i_size; + + if(byte_size >= 1 << 27) + BAD_ERROR("directory greater than 2^27-1 bytes!\n"); + + inode = get_inode(sizeof(*dir) + i_size); + dir->inode_type = SQUASHFS_LDIR_TYPE; + dir->nlink = dir_ent->dir->directory_count + 2; + dir->file_size = byte_size; + dir->offset = offset; + dir->start_block = start_block; + dir->i_count = i_count; + dir->parent_inode = dir_ent->our_dir ? + dir_ent->our_dir->dir_ent->inode->inode_number : + dir_inode_no + inode_no; + dir->xattr = xattr; + + SQUASHFS_SWAP_LDIR_INODE_HEADER(dir, inode); + p = inode + offsetof(struct squashfs_ldir_inode_header, index); + for(i = 0; i < i_count; i++) { + SQUASHFS_SWAP_DIR_INDEX(&index[i].index, p); + p += offsetof(struct squashfs_dir_index, name); + memcpy(p, index[i].name, index[i].index.size + 1); + p += index[i].index.size + 1; + } + TRACE("Long directory inode, file_size %lld, start_block " + "0x%llx, offset 0x%x, nlink %d\n", byte_size, + start_block, offset, dir_ent->dir->directory_count + 2); + } + else if(type == SQUASHFS_DIR_TYPE) { + struct squashfs_dir_inode_header *dir = &inode_header.dir; + + inode = get_inode(sizeof(*dir)); + dir->nlink = dir_ent->dir->directory_count + 2; + dir->file_size = byte_size; + dir->offset = offset; + dir->start_block = start_block; + dir->parent_inode = dir_ent->our_dir ? + dir_ent->our_dir->dir_ent->inode->inode_number : + dir_inode_no + inode_no; + SQUASHFS_SWAP_DIR_INODE_HEADER(dir, inode); + TRACE("Directory inode, file_size %lld, start_block 0x%llx, " + "offset 0x%x, nlink %d\n", byte_size, start_block, + offset, dir_ent->dir->directory_count + 2); + } + else if(type == SQUASHFS_CHRDEV_TYPE || type == SQUASHFS_BLKDEV_TYPE) { + struct squashfs_dev_inode_header *dev = &inode_header.dev; + unsigned int major = major(buf->st_rdev); + unsigned int minor = minor(buf->st_rdev); + + if(major > 0xfff) { + ERROR("Major %d out of range in device node %s, " + "truncating to %d\n", major, filename, + major & 0xfff); + major &= 0xfff; + } + if(minor > 0xfffff) { + ERROR("Minor %d out of range in device node %s, " + "truncating to %d\n", minor, filename, + minor & 0xfffff); + minor &= 0xfffff; + } + inode = get_inode(sizeof(*dev)); + dev->nlink = nlink; + dev->rdev = (major << 8) | (minor & 0xff) | + ((minor & ~0xff) << 12); + SQUASHFS_SWAP_DEV_INODE_HEADER(dev, inode); + TRACE("Device inode, rdev 0x%x, nlink %d\n", dev->rdev, nlink); + } + else if(type == SQUASHFS_LCHRDEV_TYPE || type == SQUASHFS_LBLKDEV_TYPE) { + struct squashfs_ldev_inode_header *dev = &inode_header.ldev; + unsigned int major = major(buf->st_rdev); + unsigned int minor = minor(buf->st_rdev); + + if(major > 0xfff) { + ERROR("Major %d out of range in device node %s, " + "truncating to %d\n", major, filename, + major & 0xfff); + major &= 0xfff; + } + if(minor > 0xfffff) { + ERROR("Minor %d out of range in device node %s, " + "truncating to %d\n", minor, filename, + minor & 0xfffff); + minor &= 0xfffff; + } + inode = get_inode(sizeof(*dev)); + dev->nlink = nlink; + dev->rdev = (major << 8) | (minor & 0xff) | + ((minor & ~0xff) << 12); + dev->xattr = xattr; + SQUASHFS_SWAP_LDEV_INODE_HEADER(dev, inode); + TRACE("Device inode, rdev 0x%x, nlink %d\n", dev->rdev, nlink); + } + else if(type == SQUASHFS_SYMLINK_TYPE) { + struct squashfs_symlink_inode_header *symlink = &inode_header.symlink; + int byte; + char buff[65536]; + size_t off = offsetof(struct squashfs_symlink_inode_header, symlink); + + byte = readlink(filename, buff, 65536); + if(byte == -1) { + ERROR("Failed to read symlink %s, creating empty " + "symlink\n", filename); + byte = 0; + } + + if(byte == 65536) { + ERROR("Symlink %s is greater than 65536 bytes! " + "Creating empty symlink\n", filename); + byte = 0; + } + + inode = get_inode(sizeof(*symlink) + byte); + symlink->nlink = nlink; + symlink->symlink_size = byte; + SQUASHFS_SWAP_SYMLINK_INODE_HEADER(symlink, inode); + strncpy(inode + off, buff, byte); + TRACE("Symbolic link inode, symlink_size %d, nlink %d\n", byte, + nlink); + } + else if(type == SQUASHFS_LSYMLINK_TYPE) { + struct squashfs_symlink_inode_header *symlink = &inode_header.symlink; + int byte; + char buff[65536]; + size_t off = offsetof(struct squashfs_symlink_inode_header, symlink); + + byte = readlink(filename, buff, 65536); + if(byte == -1) { + ERROR("Failed to read symlink %s, creating empty " + "symlink\n", filename); + byte = 0; + } + + if(byte == 65536) { + ERROR("Symlink %s is greater than 65536 bytes! " + "Creating empty symlink\n", filename); + byte = 0; + } + + inode = get_inode(sizeof(*symlink) + byte + + sizeof(unsigned int)); + symlink->nlink = nlink; + symlink->symlink_size = byte; + SQUASHFS_SWAP_SYMLINK_INODE_HEADER(symlink, inode); + strncpy(inode + off, buff, byte); + SQUASHFS_SWAP_INTS(&xattr, inode + off + byte, 1); + TRACE("Symbolic link inode, symlink_size %d, nlink %d\n", byte, + nlink); + } + else if(type == SQUASHFS_FIFO_TYPE || type == SQUASHFS_SOCKET_TYPE) { + struct squashfs_ipc_inode_header *ipc = &inode_header.ipc; + + inode = get_inode(sizeof(*ipc)); + ipc->nlink = nlink; + SQUASHFS_SWAP_IPC_INODE_HEADER(ipc, inode); + TRACE("ipc inode, type %s, nlink %d\n", type == + SQUASHFS_FIFO_TYPE ? "fifo" : "socket", nlink); + } + else if(type == SQUASHFS_LFIFO_TYPE || type == SQUASHFS_LSOCKET_TYPE) { + struct squashfs_lipc_inode_header *ipc = &inode_header.lipc; + + inode = get_inode(sizeof(*ipc)); + ipc->nlink = nlink; + ipc->xattr = xattr; + SQUASHFS_SWAP_LIPC_INODE_HEADER(ipc, inode); + TRACE("ipc inode, type %s, nlink %d\n", type == + SQUASHFS_FIFO_TYPE ? "fifo" : "socket", nlink); + } else + BAD_ERROR("Unrecognised inode %d in create_inode\n", type); + + *i_no = MKINODE(inode); + inode_count ++; + + TRACE("Created inode 0x%llx, type %d, uid %d, guid %d\n", *i_no, type, + base->uid, base->guid); + + return TRUE; +} + + +void scan3_init_dir(struct directory *dir) +{ + dir->buff = malloc(SQUASHFS_METADATA_SIZE); + if(dir->buff == NULL) { + BAD_ERROR("Out of memory allocating directory buffer\n"); + } + + dir->size = SQUASHFS_METADATA_SIZE; + dir->p = dir->index_count_p = dir->buff; + dir->entry_count = 256; + dir->entry_count_p = NULL; + dir->index = NULL; + dir->i_count = dir->i_size = 0; +} + + +void add_dir(squashfs_inode inode, unsigned int inode_number, char *name, + int type, struct directory *dir) +{ + unsigned char *buff; + struct squashfs_dir_entry idir; + unsigned int start_block = inode >> 16; + unsigned int offset = inode & 0xffff; + unsigned int size = strlen(name); + size_t name_off = offsetof(struct squashfs_dir_entry, name); + + if(size > SQUASHFS_NAME_LEN) { + size = SQUASHFS_NAME_LEN; + ERROR("Filename is greater than %d characters, truncating! ..." + "\n", SQUASHFS_NAME_LEN); + } + + if(dir->p + sizeof(struct squashfs_dir_entry) + size + + sizeof(struct squashfs_dir_header) + >= dir->buff + dir->size) { + buff = realloc(dir->buff, dir->size += SQUASHFS_METADATA_SIZE); + if(buff == NULL) { + BAD_ERROR("Out of memory reallocating directory buffer" + "\n"); + } + + dir->p = (dir->p - dir->buff) + buff; + if(dir->entry_count_p) + dir->entry_count_p = (dir->entry_count_p - dir->buff + + buff); + dir->index_count_p = dir->index_count_p - dir->buff + buff; + dir->buff = buff; + } + + if(dir->entry_count == 256 || start_block != dir->start_block || + ((dir->entry_count_p != NULL) && + ((dir->p + sizeof(struct squashfs_dir_entry) + size - + dir->index_count_p) > SQUASHFS_METADATA_SIZE)) || + ((long long) inode_number - dir->inode_number) > 32767 + || ((long long) inode_number - dir->inode_number) + < -32768) { + if(dir->entry_count_p) { + struct squashfs_dir_header dir_header; + + if((dir->p + sizeof(struct squashfs_dir_entry) + size - + dir->index_count_p) > + SQUASHFS_METADATA_SIZE) { + if(dir->i_count % I_COUNT_SIZE == 0) { + dir->index = realloc(dir->index, + (dir->i_count + I_COUNT_SIZE) * + sizeof(struct cached_dir_index)); + if(dir->index == NULL) + BAD_ERROR("Out of memory in " + "directory index table " + "reallocation!\n"); + } + dir->index[dir->i_count].index.index = + dir->p - dir->buff; + dir->index[dir->i_count].index.size = size - 1; + dir->index[dir->i_count++].name = name; + dir->i_size += sizeof(struct squashfs_dir_index) + + size; + dir->index_count_p = dir->p; + } + + dir_header.count = dir->entry_count - 1; + dir_header.start_block = dir->start_block; + dir_header.inode_number = dir->inode_number; + SQUASHFS_SWAP_DIR_HEADER(&dir_header, + dir->entry_count_p); + + } + + + dir->entry_count_p = dir->p; + dir->start_block = start_block; + dir->entry_count = 0; + dir->inode_number = inode_number; + dir->p += sizeof(struct squashfs_dir_header); + } + + idir.offset = offset; + idir.type = type; + idir.size = size - 1; + idir.inode_number = ((long long) inode_number - dir->inode_number); + SQUASHFS_SWAP_DIR_ENTRY(&idir, dir->p); + strncpy((char *) dir->p + name_off, name, size); + dir->p += sizeof(struct squashfs_dir_entry) + size; + dir->entry_count ++; +} + + +void write_dir(squashfs_inode *inode, struct dir_info *dir_info, + struct directory *dir) +{ + unsigned int dir_size = dir->p - dir->buff; + int data_space = directory_cache_size - directory_cache_bytes; + unsigned int directory_block, directory_offset, i_count, index; + unsigned short c_byte; + + if(data_space < dir_size) { + int realloc_size = directory_cache_size == 0 ? + ((dir_size + SQUASHFS_METADATA_SIZE) & + ~(SQUASHFS_METADATA_SIZE - 1)) : dir_size - data_space; + + void *dc = realloc(directory_data_cache, + directory_cache_size + realloc_size); + if(dc == NULL) { + goto failed; + } + directory_cache_size += realloc_size; + directory_data_cache = dc; + } + + if(dir_size) { + struct squashfs_dir_header dir_header; + + dir_header.count = dir->entry_count - 1; + dir_header.start_block = dir->start_block; + dir_header.inode_number = dir->inode_number; + SQUASHFS_SWAP_DIR_HEADER(&dir_header, dir->entry_count_p); + memcpy(directory_data_cache + directory_cache_bytes, dir->buff, + dir_size); + } + directory_offset = directory_cache_bytes; + directory_block = directory_bytes; + directory_cache_bytes += dir_size; + i_count = 0; + index = SQUASHFS_METADATA_SIZE - directory_offset; + + while(1) { + while(i_count < dir->i_count && + dir->index[i_count].index.index < index) + dir->index[i_count++].index.start_block = + directory_bytes; + index += SQUASHFS_METADATA_SIZE; + + if(directory_cache_bytes < SQUASHFS_METADATA_SIZE) + break; + + if((directory_size - directory_bytes) < + ((SQUASHFS_METADATA_SIZE << 1) + 2)) { + void *dt = realloc(directory_table, + directory_size + (SQUASHFS_METADATA_SIZE << 1) + + 2); + if(dt == NULL) { + goto failed; + } + directory_size += SQUASHFS_METADATA_SIZE << 1; + directory_table = dt; + } + + c_byte = mangle(directory_table + directory_bytes + + BLOCK_OFFSET, directory_data_cache, + SQUASHFS_METADATA_SIZE, SQUASHFS_METADATA_SIZE, + noI, 0); + TRACE("Directory block @ 0x%x, size %d\n", directory_bytes, + c_byte); + SQUASHFS_SWAP_SHORTS(&c_byte, + directory_table + directory_bytes, 1); + directory_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + + BLOCK_OFFSET; + total_directory_bytes += SQUASHFS_METADATA_SIZE + BLOCK_OFFSET; + memmove(directory_data_cache, directory_data_cache + + SQUASHFS_METADATA_SIZE, directory_cache_bytes - + SQUASHFS_METADATA_SIZE); + directory_cache_bytes -= SQUASHFS_METADATA_SIZE; + } + + create_inode(inode, dir_info, dir_info->dir_ent, SQUASHFS_DIR_TYPE, + dir_size + 3, directory_block, directory_offset, NULL, NULL, + dir, 0); + +#ifdef SQUASHFS_TRACE + { + unsigned char *dirp; + int count; + + TRACE("Directory contents of inode 0x%llx\n", *inode); + dirp = dir->buff; + while(dirp < dir->p) { + char buffer[SQUASHFS_NAME_LEN + 1]; + struct squashfs_dir_entry idir, *idirp; + struct squashfs_dir_header dirh; + SQUASHFS_SWAP_DIR_HEADER((struct squashfs_dir_header *) dirp, + &dirh); + count = dirh.count + 1; + dirp += sizeof(struct squashfs_dir_header); + + TRACE("\tStart block 0x%x, count %d\n", + dirh.start_block, count); + + while(count--) { + idirp = (struct squashfs_dir_entry *) dirp; + SQUASHFS_SWAP_DIR_ENTRY(idirp, &idir); + strncpy(buffer, idirp->name, idir.size + 1); + buffer[idir.size + 1] = '\0'; + TRACE("\t\tname %s, inode offset 0x%x, type " + "%d\n", buffer, idir.offset, idir.type); + dirp += sizeof(struct squashfs_dir_entry) + idir.size + + 1; + } + } + } +#endif + dir_count ++; + + return; + +failed: + BAD_ERROR("Out of memory in directory table reallocation!\n"); +} + + +struct file_buffer *get_fragment(struct fragment *fragment) +{ + struct squashfs_fragment_entry *disk_fragment; + int res, size; + long long start_block; + struct file_buffer *buffer, *compressed_buffer; + + if(fragment->index == SQUASHFS_INVALID_FRAG) + return NULL; + + buffer = cache_lookup(fragment_buffer, fragment->index); + if(buffer) + return buffer; + + compressed_buffer = cache_lookup(writer_buffer, fragment->index + + FRAG_INDEX); + + buffer = cache_get(fragment_buffer, fragment->index, 1); + + pthread_mutex_lock(&fragment_mutex); + disk_fragment = &fragment_table[fragment->index]; + size = SQUASHFS_COMPRESSED_SIZE_BLOCK(disk_fragment->size); + start_block = disk_fragment->start_block; + pthread_mutex_unlock(&fragment_mutex); + + if(SQUASHFS_COMPRESSED_BLOCK(disk_fragment->size)) { + int error; + char *data; + + if(compressed_buffer) + data = compressed_buffer->data; + else + data = read_from_disk(start_block, size); + + res = compressor_uncompress(comp, buffer->data, data, size, + block_size, &error); + if(res == -1) + BAD_ERROR("%s uncompress failed with error code %d\n", + comp->name, error); + } else if(compressed_buffer) + memcpy(buffer->data, compressed_buffer->data, size); + else { + res = read_fs_bytes(fd, start_block, size, buffer->data); + if(res == 0) + EXIT_MKSQUASHFS(); + } + + cache_block_put(compressed_buffer); + + return buffer; +} + + +struct frag_locked { + struct file_buffer *buffer; + int c_byte; + int fragment; + struct frag_locked *fragment_prev; + struct frag_locked *fragment_next; +}; + +int fragments_locked = FALSE; +struct frag_locked *frag_locked_list = NULL; + +INSERT_LIST(fragment, struct frag_locked) +REMOVE_LIST(fragment, struct frag_locked) + +int lock_fragments() +{ + int count; + pthread_mutex_lock(&fragment_mutex); + fragments_locked = TRUE; + count = fragments_outstanding; + pthread_mutex_unlock(&fragment_mutex); + return count; +} + + +void unlock_fragments() +{ + struct frag_locked *entry; + int compressed_size; + + pthread_mutex_lock(&fragment_mutex); + while(frag_locked_list) { + entry = frag_locked_list; + remove_fragment_list(&frag_locked_list, entry); + compressed_size = SQUASHFS_COMPRESSED_SIZE_BLOCK(entry->c_byte); + fragment_table[entry->fragment].size = entry->c_byte; + fragment_table[entry->fragment].start_block = bytes; + entry->buffer->block = bytes; + bytes += compressed_size; + fragments_outstanding --; + queue_put(to_writer, entry->buffer); + TRACE("fragment_locked writing fragment %d, compressed size %d" + "\n", entry->fragment, compressed_size); + free(entry); + } + fragments_locked = FALSE; + pthread_mutex_unlock(&fragment_mutex); +} + + +void add_pending_fragment(struct file_buffer *write_buffer, int c_byte, + int fragment) +{ + struct frag_locked *entry = malloc(sizeof(struct frag_locked)); + if(entry == NULL) + BAD_ERROR("Out of memory in add_pending fragment\n"); + entry->buffer = write_buffer; + entry->c_byte = c_byte; + entry->fragment = fragment; + entry->fragment_prev = entry->fragment_next = NULL; + pthread_mutex_lock(&fragment_mutex); + insert_fragment_list(&frag_locked_list, entry); + pthread_mutex_unlock(&fragment_mutex); +} + + +void write_fragment() +{ + if(fragment_size == 0) + return; + + pthread_mutex_lock(&fragment_mutex); + if(fragments % FRAG_SIZE == 0) { + void *ft = realloc(fragment_table, (fragments + + FRAG_SIZE) * sizeof(struct squashfs_fragment_entry)); + if(ft == NULL) { + pthread_mutex_unlock(&fragment_mutex); + BAD_ERROR("Out of memory in fragment table\n"); + } + fragment_table = ft; + } + fragment_data->size = fragment_size; + fragment_data->block = fragments; + fragment_table[fragments].unused = 0; + fragments_outstanding ++; + queue_put(to_frag, fragment_data); + fragments ++; + fragment_size = 0; + pthread_mutex_unlock(&fragment_mutex); +} + + +static struct fragment empty_fragment = {SQUASHFS_INVALID_FRAG, 0, 0}; +struct fragment *get_and_fill_fragment(struct file_buffer *file_buffer) +{ + struct fragment *ffrg; + + + if(file_buffer == NULL || file_buffer->size == 0) + return &empty_fragment; + + if(fragment_size + file_buffer->size > block_size) + write_fragment(); + + ffrg = malloc(sizeof(struct fragment)); + if(ffrg == NULL) + BAD_ERROR("Out of memory in fragment block allocation!\n"); + + if(fragment_size == 0) + fragment_data = cache_get(fragment_buffer, fragments, 1); + + ffrg->index = fragments; + ffrg->offset = fragment_size; + ffrg->size = file_buffer->size; + memcpy(fragment_data->data + fragment_size, file_buffer->data, + file_buffer->size); + fragment_size += file_buffer->size; + + return ffrg; +} + + +long long generic_write_table(int length, void *buffer, int length2, + void *buffer2, int uncompressed) +{ + int meta_blocks = (length + SQUASHFS_METADATA_SIZE - 1) / + SQUASHFS_METADATA_SIZE; + long long list[meta_blocks], start_bytes; + int compressed_size, i; + unsigned short c_byte; + char cbuffer[(SQUASHFS_METADATA_SIZE << 2) + 2]; + +#ifdef SQUASHFS_TRACE + long long obytes = bytes; + int olength = length; +#endif + + for(i = 0; i < meta_blocks; i++) { + int avail_bytes = length > SQUASHFS_METADATA_SIZE ? + SQUASHFS_METADATA_SIZE : length; + c_byte = mangle(cbuffer + BLOCK_OFFSET, buffer + i * + SQUASHFS_METADATA_SIZE , avail_bytes, + SQUASHFS_METADATA_SIZE, uncompressed, 0); + SQUASHFS_SWAP_SHORTS(&c_byte, cbuffer, 1); + list[i] = bytes; + compressed_size = SQUASHFS_COMPRESSED_SIZE(c_byte) + + BLOCK_OFFSET; + TRACE("block %d @ 0x%llx, compressed size %d\n", i, bytes, + compressed_size); + write_destination(fd, bytes, compressed_size, cbuffer); + bytes += compressed_size; + total_bytes += avail_bytes; + length -= avail_bytes; + } + + start_bytes = bytes; + if(length2) { + write_destination(fd, bytes, length2, buffer2); + bytes += length2; + total_bytes += length2; + } + + SQUASHFS_INSWAP_LONG_LONGS(list, meta_blocks); + write_destination(fd, bytes, sizeof(list), list); + bytes += sizeof(list); + total_bytes += sizeof(list); + + TRACE("generic_write_table: total uncompressed %d compressed %lld\n", + olength, bytes - obytes); + + return start_bytes; +} + + +long long write_fragment_table() +{ + unsigned int frag_bytes = SQUASHFS_FRAGMENT_BYTES(fragments); + struct squashfs_fragment_entry p[fragments]; + int i; + + TRACE("write_fragment_table: fragments %d, frag_bytes %d\n", fragments, + frag_bytes); + for(i = 0; i < fragments; i++) { + TRACE("write_fragment_table: fragment %d, start_block 0x%llx, " + "size %d\n", i, fragment_table[i].start_block, + fragment_table[i].size); + SQUASHFS_SWAP_FRAGMENT_ENTRY(&fragment_table[i], p + i); + } + + return generic_write_table(frag_bytes, p, 0, NULL, noF); +} + + +char read_from_file_buffer[SQUASHFS_FILE_MAX_SIZE]; +char *read_from_disk(long long start, unsigned int avail_bytes) +{ + int res; + + res = read_fs_bytes(fd, start, avail_bytes, read_from_file_buffer); + if(res == 0) + EXIT_MKSQUASHFS(); + + return read_from_file_buffer; +} + + +char read_from_file_buffer2[SQUASHFS_FILE_MAX_SIZE]; +char *read_from_disk2(long long start, unsigned int avail_bytes) +{ + int res; + + res = read_fs_bytes(fd, start, avail_bytes, read_from_file_buffer2); + if(res == 0) + EXIT_MKSQUASHFS(); + + return read_from_file_buffer2; +} + + +/* + * Compute 16 bit BSD checksum over the data + */ +unsigned short get_checksum(char *buff, int bytes, unsigned short chksum) +{ + unsigned char *b = (unsigned char *) buff; + + while(bytes --) { + chksum = (chksum & 1) ? (chksum >> 1) | 0x8000 : chksum >> 1; + chksum += *b++; + } + + return chksum; +} + + +unsigned short get_checksum_disk(long long start, long long l, + unsigned int *blocks) +{ + unsigned short chksum = 0; + unsigned int bytes; + struct file_buffer *write_buffer; + int i; + + for(i = 0; l; i++) { + bytes = SQUASHFS_COMPRESSED_SIZE_BLOCK(blocks[i]); + if(bytes == 0) /* sparse block */ + continue; + write_buffer = cache_lookup(writer_buffer, start); + if(write_buffer) { + chksum = get_checksum(write_buffer->data, bytes, + chksum); + cache_block_put(write_buffer); + } else + chksum = get_checksum(read_from_disk(start, bytes), + bytes, chksum); + l -= bytes; + start += bytes; + } + + return chksum; +} + + +unsigned short get_checksum_mem(char *buff, int bytes) +{ + return get_checksum(buff, bytes, 0); +} + + +unsigned short get_checksum_mem_buffer(struct file_buffer *file_buffer) +{ + if(file_buffer == NULL) + return 0; + else + return get_checksum(file_buffer->data, file_buffer->size, 0); +} + + +#define DUP_HASH(a) (a & 0xffff) +void add_file(long long start, long long file_size, long long file_bytes, + unsigned int *block_listp, int blocks, unsigned int fragment, + int offset, int bytes) +{ + struct fragment *frg; + unsigned int *block_list = block_listp; + struct file_info *dupl_ptr = dupl[DUP_HASH(file_size)]; + + if(!duplicate_checking || file_size == 0) + return; + + for(; dupl_ptr; dupl_ptr = dupl_ptr->next) { + if(file_size != dupl_ptr->file_size) + continue; + if(blocks != 0 && start != dupl_ptr->start) + continue; + if(fragment != dupl_ptr->fragment->index) + continue; + if(fragment != SQUASHFS_INVALID_FRAG && (offset != + dupl_ptr->fragment->offset || bytes != + dupl_ptr->fragment->size)) + continue; + return; + } + + frg = malloc(sizeof(struct fragment)); + if(frg == NULL) + BAD_ERROR("Out of memory in fragment block allocation!\n"); + + frg->index = fragment; + frg->offset = offset; + frg->size = bytes; + + add_non_dup(file_size, file_bytes, block_list, start, frg, 0, 0, FALSE); +} + + +int pre_duplicate(long long file_size) +{ + struct file_info *dupl_ptr = dupl[DUP_HASH(file_size)]; + + for(; dupl_ptr; dupl_ptr = dupl_ptr->next) + if(dupl_ptr->file_size == file_size) + return TRUE; + + return FALSE; +} + + +int pre_duplicate_frag(long long file_size, unsigned short checksum) +{ + struct file_info *dupl_ptr = dupl[DUP_HASH(file_size)]; + + for(; dupl_ptr; dupl_ptr = dupl_ptr->next) + if(file_size == dupl_ptr->file_size && file_size == + dupl_ptr->fragment->size) { + if(dupl_ptr->checksum_flag == FALSE) { + struct file_buffer *frag_buffer = + get_fragment(dupl_ptr->fragment); + dupl_ptr->checksum = + get_checksum_disk(dupl_ptr->start, + dupl_ptr->bytes, dupl_ptr->block_list); + dupl_ptr->fragment_checksum = + get_checksum_mem(frag_buffer->data + + dupl_ptr->fragment->offset, file_size); + cache_block_put(frag_buffer); + dupl_ptr->checksum_flag = TRUE; + } + if(dupl_ptr->fragment_checksum == checksum) + return TRUE; + } + + return FALSE; +} + + +struct file_info *add_non_dup(long long file_size, long long bytes, + unsigned int *block_list, long long start, struct fragment *fragment, + unsigned short checksum, unsigned short fragment_checksum, + int checksum_flag) +{ + struct file_info *dupl_ptr = malloc(sizeof(struct file_info)); + + if(dupl_ptr == NULL) { + BAD_ERROR("Out of memory in dup_files allocation!\n"); + } + + dupl_ptr->file_size = file_size; + dupl_ptr->bytes = bytes; + dupl_ptr->block_list = block_list; + dupl_ptr->start = start; + dupl_ptr->fragment = fragment; + dupl_ptr->checksum = checksum; + dupl_ptr->fragment_checksum = fragment_checksum; + dupl_ptr->checksum_flag = checksum_flag; + dupl_ptr->next = dupl[DUP_HASH(file_size)]; + dupl[DUP_HASH(file_size)] = dupl_ptr; + dup_files ++; + + return dupl_ptr; +} + + +struct file_info *duplicate(long long file_size, long long bytes, + unsigned int **block_list, long long *start, struct fragment **fragment, + struct file_buffer *file_buffer, int blocks, unsigned short checksum, + unsigned short fragment_checksum, int checksum_flag) +{ + struct file_info *dupl_ptr = dupl[DUP_HASH(file_size)]; + int frag_bytes = file_buffer ? file_buffer->size : 0; + + for(; dupl_ptr; dupl_ptr = dupl_ptr->next) + if(file_size == dupl_ptr->file_size && bytes == dupl_ptr->bytes + && frag_bytes == dupl_ptr->fragment->size) { + long long target_start, dup_start = dupl_ptr->start; + int block; + + if(memcmp(*block_list, dupl_ptr->block_list, blocks * + sizeof(unsigned int)) != 0) + continue; + + if(checksum_flag == FALSE) { + checksum = get_checksum_disk(*start, bytes, + *block_list); + fragment_checksum = + get_checksum_mem_buffer(file_buffer); + checksum_flag = TRUE; + } + + if(dupl_ptr->checksum_flag == FALSE) { + struct file_buffer *frag_buffer = + get_fragment(dupl_ptr->fragment); + dupl_ptr->checksum = + get_checksum_disk(dupl_ptr->start, + dupl_ptr->bytes, dupl_ptr->block_list); + dupl_ptr->fragment_checksum = + get_checksum_mem(frag_buffer->data + + dupl_ptr->fragment->offset, frag_bytes); + cache_block_put(frag_buffer); + dupl_ptr->checksum_flag = TRUE; + } + + if(checksum != dupl_ptr->checksum || + fragment_checksum != + dupl_ptr->fragment_checksum) + continue; + + target_start = *start; + for(block = 0; block < blocks; block ++) { + int size = SQUASHFS_COMPRESSED_SIZE_BLOCK + ((*block_list)[block]); + struct file_buffer *target_buffer = NULL; + struct file_buffer *dup_buffer = NULL; + char *target_data, *dup_data; + int res; + + if(size == 0) + continue; + target_buffer = cache_lookup(writer_buffer, + target_start); + if(target_buffer) + target_data = target_buffer->data; + else + target_data = + read_from_disk(target_start, + size); + + dup_buffer = cache_lookup(writer_buffer, + dup_start); + if(dup_buffer) + dup_data = dup_buffer->data; + else + dup_data = read_from_disk2(dup_start, + size); + + res = memcmp(target_data, dup_data, size); + cache_block_put(target_buffer); + cache_block_put(dup_buffer); + if(res != 0) + break; + target_start += size; + dup_start += size; + } + if(block == blocks) { + struct file_buffer *frag_buffer = + get_fragment(dupl_ptr->fragment); + + if(frag_bytes == 0 || + memcmp(file_buffer->data, + frag_buffer->data + + dupl_ptr->fragment->offset, + frag_bytes) == 0) { + TRACE("Found duplicate file, start " + "0x%llx, size %lld, checksum " + "0x%x, fragment %d, size %d, " + "offset %d, checksum 0x%x\n", + dupl_ptr->start, + dupl_ptr->bytes, + dupl_ptr->checksum, + dupl_ptr->fragment->index, + frag_bytes, + dupl_ptr->fragment->offset, + fragment_checksum); + *block_list = dupl_ptr->block_list; + *start = dupl_ptr->start; + *fragment = dupl_ptr->fragment; + cache_block_put(frag_buffer); + return 0; + } + cache_block_put(frag_buffer); + } + } + + + return add_non_dup(file_size, bytes, *block_list, *start, *fragment, + checksum, fragment_checksum, checksum_flag); +} + + +static int seq = 0; +void reader_read_process(struct dir_ent *dir_ent) +{ + struct file_buffer *prev_buffer = NULL, *file_buffer; + int status, res, byte, count = 0; + int file = get_pseudo_file(dir_ent->inode->pseudo_id)->fd; + int child = get_pseudo_file(dir_ent->inode->pseudo_id)->child; + long long bytes = 0; + + while(1) { + file_buffer = cache_get(reader_buffer, 0, 0); + file_buffer->sequence = seq ++; + + byte = read_bytes(file, file_buffer->data, block_size); + if(byte == -1) + goto read_err; + + file_buffer->size = byte; + file_buffer->file_size = -1; + file_buffer->block = count ++; + file_buffer->error = FALSE; + file_buffer->fragment = FALSE; + bytes += byte; + + if(byte == 0) + break; + + /* + * Update estimated_uncompressed block count. This is done + * on every block rather than waiting for all blocks to be + * read incase write_file_process() is running in parallel + * with this. Otherwise cur uncompressed block count may + * get ahead of the total uncompressed block count. + */ + estimated_uncompressed ++; + + if(prev_buffer) + queue_put(from_reader, prev_buffer); + prev_buffer = file_buffer; + } + + /* + * Update inode file size now that the size of the dynamic pseudo file + * is known. This is needed for the -info option. + */ + dir_ent->inode->buf.st_size = bytes; + + res = waitpid(child, &status, 0); + if(res == -1 || !WIFEXITED(status) || WEXITSTATUS(status) != 0) + goto read_err; + + if(prev_buffer == NULL) + prev_buffer = file_buffer; + else { + cache_block_put(file_buffer); + seq --; + } + prev_buffer->file_size = bytes; + prev_buffer->fragment = !no_fragments && + (count == 2 || always_use_fragments) && (byte < block_size); + queue_put(from_reader, prev_buffer); + + return; + +read_err: + if(prev_buffer) { + cache_block_put(file_buffer); + seq --; + file_buffer = prev_buffer; + } + file_buffer->error = TRUE; + queue_put(from_deflate, file_buffer); +} + + +void reader_read_file(struct dir_ent *dir_ent) +{ + struct stat *buf = &dir_ent->inode->buf, buf2; + struct file_buffer *file_buffer; + int blocks, byte, count, expected, file, frag_block; + long long bytes, read_size; + + if(dir_ent->inode->read) + return; + + dir_ent->inode->read = TRUE; +again: + bytes = 0; + count = 0; + file_buffer = NULL; + read_size = buf->st_size; + blocks = (read_size + block_size - 1) >> block_log; + frag_block = !no_fragments && (always_use_fragments || + (read_size < block_size)) ? read_size >> block_log : -1; + + file = open(dir_ent->pathname, O_RDONLY); + if(file == -1) { + file_buffer = cache_get(reader_buffer, 0, 0); + file_buffer->sequence = seq ++; + goto read_err; + } + + do { + expected = read_size - ((long long) count * block_size) > + block_size ? block_size : + read_size - ((long long) count * block_size); + + if(file_buffer) + queue_put(from_reader, file_buffer); + file_buffer = cache_get(reader_buffer, 0, 0); + file_buffer->sequence = seq ++; + + /* + * Always try to read block_size bytes from the file rather + * than expected bytes (which will be less than the block_size + * at the file tail) to check that the file hasn't grown + * since being stated. If it is longer (or shorter) than + * expected, then restat, and try again. Note the special + * case where the file is an exact multiple of the block_size + * is dealt with later. + */ + byte = file_buffer->size = read_bytes(file, file_buffer->data, + block_size); + + file_buffer->file_size = read_size; + + if(byte == -1) + goto read_err; + + if(byte != expected) + goto restat; + + file_buffer->block = count; + file_buffer->error = FALSE; + file_buffer->fragment = (file_buffer->block == frag_block); + + bytes += byte; + count ++; + } while(count < blocks); + + if(read_size != bytes) + goto restat; + + if(expected == block_size) { + /* + * Special case where we've not tried to read past the end of + * the file. We expect to get EOF, i.e. the file isn't larger + * than we expect. + */ + char buffer; + int res; + + res = read_bytes(file, &buffer, 1); + if(res == -1) + goto read_err; + + if(res != 0) + goto restat; + } + + queue_put(from_reader, file_buffer); + + close(file); + + return; + +restat: + fstat(file, &buf2); + close(file); + if(read_size != buf2.st_size) { + memcpy(buf, &buf2, sizeof(struct stat)); + file_buffer->error = 2; + queue_put(from_deflate, file_buffer); + goto again; + } +read_err: + file_buffer->error = TRUE; + queue_put(from_deflate, file_buffer); +} + + +void reader_scan(struct dir_info *dir) { + int i; + + for(i = 0; i < dir->count; i++) { + struct dir_ent *dir_ent = dir->list[i]; + struct stat *buf = &dir_ent->inode->buf; + if(dir_ent->inode->root_entry) + continue; + + if(IS_PSEUDO_PROCESS(dir_ent->inode)) { + reader_read_process(dir_ent); + continue; + } + + switch(buf->st_mode & S_IFMT) { + case S_IFREG: + reader_read_file(dir_ent); + break; + case S_IFDIR: + reader_scan(dir_ent->dir); + break; + } + } +} + + +void *reader(void *arg) +{ + int oldstate; + + pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &oldstate); + pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, &oldstate); + + if(!sorted) + reader_scan(queue_get(to_reader)); + else { + int i; + struct priority_entry *entry; + + queue_get(to_reader); + for(i = 65535; i >= 0; i--) + for(entry = priority_list[i]; entry; + entry = entry->next) + reader_read_file(entry->dir); + } + + thread[0] = 0; + + pthread_exit(NULL); +} + + +void *writer(void *arg) +{ + int write_error = FALSE; + int oldstate; + + pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &oldstate); + pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, &oldstate); + + while(1) { + struct file_buffer *file_buffer = queue_get(to_writer); + off_t off; + + if(file_buffer == NULL) { + queue_put(from_writer, + write_error ? &write_error : NULL); + continue; + } + + off = file_buffer->block; + + pthread_mutex_lock(&pos_mutex); + + if(!write_error && lseek(fd, off, SEEK_SET) == -1) { + ERROR("Lseek on destination failed because %s\n", + strerror(errno)); + write_error = TRUE; + } + + if(!write_error && write_bytes(fd, file_buffer->data, + file_buffer->size) == -1) { + ERROR("Write on destination failed because %s\n", + strerror(errno)); + write_error = TRUE; + } + pthread_mutex_unlock(&pos_mutex); + + cache_block_put(file_buffer); + } +} + + +int all_zero(struct file_buffer *file_buffer) +{ + int i; + long entries = file_buffer->size / sizeof(long); + long *p = (long *) file_buffer->data; + + for(i = 0; i < entries && p[i] == 0; i++); + + if(i == entries) { + for(i = file_buffer->size & ~(sizeof(long) - 1); + i < file_buffer->size && file_buffer->data[i] == 0; + i++); + + return i == file_buffer->size; + } + + return 0; +} + + +void *deflator(void *arg) +{ + void *stream = NULL; + int res, oldstate; + + pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &oldstate); + pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, &oldstate); + + res = compressor_init(comp, &stream, block_size, 1); + if(res) + BAD_ERROR("deflator:: compressor_init failed\n"); + + while(1) { + struct file_buffer *file_buffer = queue_get(from_reader); + struct file_buffer *write_buffer; + + if(sparse_files && all_zero(file_buffer)) { + file_buffer->c_byte = 0; + queue_put(from_deflate, file_buffer); + } else if(file_buffer->fragment) { + file_buffer->c_byte = file_buffer->size; + queue_put(from_deflate, file_buffer); + } else { + write_buffer = cache_get(writer_buffer, 0, 0); + write_buffer->c_byte = mangle2(stream, + write_buffer->data, file_buffer->data, + file_buffer->size, block_size, noD, 1); + write_buffer->sequence = file_buffer->sequence; + write_buffer->file_size = file_buffer->file_size; + write_buffer->block = file_buffer->block; + write_buffer->size = SQUASHFS_COMPRESSED_SIZE_BLOCK + (write_buffer->c_byte); + write_buffer->fragment = FALSE; + write_buffer->error = FALSE; + cache_block_put(file_buffer); + queue_put(from_deflate, write_buffer); + } + } +} + + +void *frag_deflator(void *arg) +{ + void *stream = NULL; + int res, oldstate; + + pthread_setcancelstate(PTHREAD_CANCEL_ENABLE, &oldstate); + pthread_setcanceltype(PTHREAD_CANCEL_ASYNCHRONOUS, &oldstate); + + res = compressor_init(comp, &stream, block_size, 1); + if(res) + BAD_ERROR("frag_deflator:: compressor_init failed\n"); + + while(1) { + int c_byte, compressed_size; + struct file_buffer *file_buffer = queue_get(to_frag); + struct file_buffer *write_buffer = + cache_get(writer_buffer, file_buffer->block + + FRAG_INDEX, 1); + + c_byte = mangle2(stream, write_buffer->data, file_buffer->data, + file_buffer->size, block_size, noF, 1); + compressed_size = SQUASHFS_COMPRESSED_SIZE_BLOCK(c_byte); + write_buffer->size = compressed_size; + pthread_mutex_lock(&fragment_mutex); + if(fragments_locked == FALSE) { + fragment_table[file_buffer->block].size = c_byte; + fragment_table[file_buffer->block].start_block = bytes; + write_buffer->block = bytes; + bytes += compressed_size; + fragments_outstanding --; + queue_put(to_writer, write_buffer); + pthread_mutex_unlock(&fragment_mutex); + TRACE("Writing fragment %lld, uncompressed size %d, " + "compressed size %d\n", file_buffer->block, + file_buffer->size, compressed_size); + } else { + pthread_mutex_unlock(&fragment_mutex); + add_pending_fragment(write_buffer, c_byte, + file_buffer->block); + } + cache_block_put(file_buffer); + } +} + + +#define HASH_ENTRIES 256 +#define BLOCK_HASH(a) (a % HASH_ENTRIES) +struct file_buffer *block_hash[HASH_ENTRIES]; + +void push_buffer(struct file_buffer *file_buffer) +{ + int hash = BLOCK_HASH(file_buffer->sequence); + + file_buffer->next = block_hash[hash]; + block_hash[hash] = file_buffer; +} + + +struct file_buffer *get_file_buffer(struct queue *queue) +{ + static unsigned int sequence = 0; + int hash = BLOCK_HASH(sequence); + struct file_buffer *file_buffer = block_hash[hash], *prev = NULL; + + for(;file_buffer; prev = file_buffer, file_buffer = file_buffer->next) + if(file_buffer->sequence == sequence) + break; + + if(file_buffer) { + if(prev) + prev->next = file_buffer->next; + else + block_hash[hash] = file_buffer->next; + } else { + while(1) { + file_buffer = queue_get(queue); + if(file_buffer->sequence == sequence) + break; + push_buffer(file_buffer); + } + } + + sequence ++; + + return file_buffer; +} + + +void *progress_thrd(void *arg) +{ + struct timeval timeval; + struct timespec timespec; + struct itimerval itimerval; + struct winsize winsize; + + if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { + if(isatty(STDOUT_FILENO)) + printf("TIOCGWINSZ ioctl failed, defaulting to 80 " + "columns\n"); + columns = 80; + } else + columns = winsize.ws_col; + signal(SIGWINCH, sigwinch_handler); + signal(SIGALRM, sigalrm_handler); + + itimerval.it_value.tv_sec = 0; + itimerval.it_value.tv_usec = 250000; + itimerval.it_interval.tv_sec = 0; + itimerval.it_interval.tv_usec = 250000; + setitimer(ITIMER_REAL, &itimerval, NULL); + + pthread_cond_init(&progress_wait, NULL); + + pthread_mutex_lock(&progress_mutex); + + while(1) { + gettimeofday(&timeval, NULL); + timespec.tv_sec = timeval.tv_sec; + if(timeval.tv_usec + 250000 > 999999) + timespec.tv_sec++; + timespec.tv_nsec = ((timeval.tv_usec + 250000) % 1000000) * + 1000; + pthread_cond_timedwait(&progress_wait, &progress_mutex, + ×pec); + if(progress_enabled && estimated_uncompressed) + progress_bar(cur_uncompressed, estimated_uncompressed, + columns); + } +} + + +void enable_progress_bar() +{ + pthread_mutex_lock(&progress_mutex); + progress_enabled = TRUE; + pthread_mutex_unlock(&progress_mutex); +} + + +void disable_progress_bar() +{ + pthread_mutex_lock(&progress_mutex); + progress_enabled = FALSE; + pthread_mutex_unlock(&progress_mutex); +} + + +void progress_bar(long long current, long long max, int columns) +{ + char rotate_list[] = { '|', '/', '-', '\\' }; + int max_digits, used, hashes, spaces; + static int tty = -1; + + if(max == 0) + return; + + max_digits = floor(log10(max)) + 1; + used = max_digits * 2 + 11; + hashes = (current * (columns - used)) / max; + spaces = columns - used - hashes; + + if((current > max) || (columns - used < 0)) + return; + + if(tty == -1) + tty = isatty(STDOUT_FILENO); + if(!tty) { + static long long previous = -1; + + /* Updating much more frequently than this results in huge + * log files. */ + if((current % 100) != 0 && current != max) + return; + /* Don't update just to rotate the spinner. */ + if(current == previous) + return; + previous = current; + } + + printf("\r["); + + while (hashes --) + putchar('='); + + putchar(rotate_list[rotate]); + + while(spaces --) + putchar(' '); + + printf("] %*lld/%*lld", max_digits, current, max_digits, max); + printf(" %3lld%%", current * 100 / max); + fflush(stdout); +} + + +void write_file_empty(squashfs_inode *inode, struct dir_ent *dir_ent, + int *duplicate_file) +{ + file_count ++; + *duplicate_file = FALSE; + create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, 0, 0, 0, + NULL, &empty_fragment, NULL, 0); +} + + +void write_file_frag_dup(squashfs_inode *inode, struct dir_ent *dir_ent, + int size, int *duplicate_file, struct file_buffer *file_buffer, + unsigned short checksum) +{ + struct file_info *dupl_ptr; + struct fragment *fragment; + unsigned int *block_listp = NULL; + long long start = 0; + + dupl_ptr = duplicate(size, 0, &block_listp, &start, &fragment, + file_buffer, 0, 0, checksum, TRUE); + + if(dupl_ptr) { + *duplicate_file = FALSE; + fragment = get_and_fill_fragment(file_buffer); + dupl_ptr->fragment = fragment; + } else + *duplicate_file = TRUE; + + cache_block_put(file_buffer); + + total_bytes += size; + file_count ++; + + inc_progress_bar(); + + create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, size, 0, + 0, NULL, fragment, NULL, 0); +} + + +void write_file_frag(squashfs_inode *inode, struct dir_ent *dir_ent, int size, + struct file_buffer *file_buffer, int *duplicate_file) +{ + struct fragment *fragment; + unsigned short checksum; + + checksum = get_checksum_mem_buffer(file_buffer); + + if(pre_duplicate_frag(size, checksum)) { + write_file_frag_dup(inode, dir_ent, size, duplicate_file, + file_buffer, checksum); + return; + } + + fragment = get_and_fill_fragment(file_buffer); + + cache_block_put(file_buffer); + + if(duplicate_checking) + add_non_dup(size, 0, NULL, 0, fragment, 0, checksum, TRUE); + + total_bytes += size; + file_count ++; + + *duplicate_file = FALSE; + + inc_progress_bar(); + + create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, size, 0, + 0, NULL, fragment, NULL, 0); + + return; +} + + +int write_file_process(squashfs_inode *inode, struct dir_ent *dir_ent, + struct file_buffer *read_buffer, int *duplicate_file) +{ + long long read_size, file_bytes, start; + struct fragment *fragment; + unsigned int *block_list = NULL; + int block = 0, status; + long long sparse = 0; + struct file_buffer *fragment_buffer = NULL; + + *duplicate_file = FALSE; + + lock_fragments(); + + file_bytes = 0; + start = bytes; + while (1) { + read_size = read_buffer->file_size; + if(read_buffer->fragment && read_buffer->c_byte) + fragment_buffer = read_buffer; + else { + block_list = realloc(block_list, (block + 1) * + sizeof(unsigned int)); + if(block_list == NULL) + BAD_ERROR("Out of memory allocating block_list" + "\n"); + block_list[block ++] = read_buffer->c_byte; + if(read_buffer->c_byte) { + read_buffer->block = bytes; + bytes += read_buffer->size; + cache_rehash(read_buffer, read_buffer->block); + file_bytes += read_buffer->size; + queue_put(to_writer, read_buffer); + } else { + sparse += read_buffer->size; + cache_block_put(read_buffer); + } + } + inc_progress_bar(); + + if(read_size != -1) + break; + + read_buffer = get_file_buffer(from_deflate); + if(read_buffer->error) + goto read_err; + } + + unlock_fragments(); + fragment = get_and_fill_fragment(fragment_buffer); + cache_block_put(fragment_buffer); + + if(duplicate_checking) + add_non_dup(read_size, file_bytes, block_list, start, fragment, + 0, 0, FALSE); + file_count ++; + total_bytes += read_size; + + create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, read_size, start, + block, block_list, fragment, NULL, sparse); + + if(duplicate_checking == FALSE) + free(block_list); + + return 0; + +read_err: + cur_uncompressed -= block; + status = read_buffer->error; + bytes = start; + if(!block_device) { + int res; + + queue_put(to_writer, NULL); + if(queue_get(from_writer) != 0) + EXIT_MKSQUASHFS(); + res = ftruncate(fd, bytes); + if(res != 0) + BAD_ERROR("Failed to truncate dest file because %s\n", + strerror(errno)); + } + unlock_fragments(); + free(block_list); + cache_block_put(read_buffer); + return status; +} + + +int write_file_blocks(squashfs_inode *inode, struct dir_ent *dir_ent, + long long read_size, struct file_buffer *read_buffer, + int *duplicate_file) +{ + long long file_bytes, start; + struct fragment *fragment; + unsigned int *block_list; + int block, status; + int blocks = (read_size + block_size - 1) >> block_log; + long long sparse = 0; + struct file_buffer *fragment_buffer = NULL; + + *duplicate_file = FALSE; + + block_list = malloc(blocks * sizeof(unsigned int)); + if(block_list == NULL) + BAD_ERROR("Out of memory allocating block_list\n"); + + lock_fragments(); + + file_bytes = 0; + start = bytes; + for(block = 0; block < blocks;) { + if(read_buffer->fragment && read_buffer->c_byte) { + fragment_buffer = read_buffer; + blocks = read_size >> block_log; + } else { + block_list[block] = read_buffer->c_byte; + if(read_buffer->c_byte) { + read_buffer->block = bytes; + bytes += read_buffer->size; + cache_rehash(read_buffer, read_buffer->block); + file_bytes += read_buffer->size; + queue_put(to_writer, read_buffer); + } else { + sparse += read_buffer->size; + cache_block_put(read_buffer); + } + } + inc_progress_bar(); + + if(++block < blocks) { + read_buffer = get_file_buffer(from_deflate); + if(read_buffer->error) + goto read_err; + } + } + + unlock_fragments(); + fragment = get_and_fill_fragment(fragment_buffer); + cache_block_put(fragment_buffer); + + if(duplicate_checking) + add_non_dup(read_size, file_bytes, block_list, start, fragment, + 0, 0, FALSE); + file_count ++; + total_bytes += read_size; + + /* + * sparse count is needed to ensure squashfs correctly reports a + * a smaller block count on stat calls to sparse files. This is + * to ensure intelligent applications like cp correctly handle the + * file as a sparse file. If the file in the original filesystem isn't + * stored as a sparse file then still store it sparsely in squashfs, but + * report it as non-sparse on stat calls to preserve semantics + */ + if(sparse && (dir_ent->inode->buf.st_blocks << 9) >= read_size) + sparse = 0; + + create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, read_size, start, + blocks, block_list, fragment, NULL, sparse); + + if(duplicate_checking == FALSE) + free(block_list); + + return 0; + +read_err: + cur_uncompressed -= block; + status = read_buffer->error; + bytes = start; + if(!block_device) { + int res; + + queue_put(to_writer, NULL); + if(queue_get(from_writer) != 0) + EXIT_MKSQUASHFS(); + res = ftruncate(fd, bytes); + if(res != 0) + BAD_ERROR("Failed to truncate dest file because %s\n", + strerror(errno)); + } + unlock_fragments(); + free(block_list); + cache_block_put(read_buffer); + return status; +} + + +int write_file_blocks_dup(squashfs_inode *inode, struct dir_ent *dir_ent, + long long read_size, struct file_buffer *read_buffer, + int *duplicate_file) +{ + int block, thresh; + long long file_bytes, dup_start, start; + struct fragment *fragment; + struct file_info *dupl_ptr; + int blocks = (read_size + block_size - 1) >> block_log; + unsigned int *block_list, *block_listp; + struct file_buffer **buffer_list; + int status, num_locked_fragments; + long long sparse = 0; + struct file_buffer *fragment_buffer = NULL; + + block_list = malloc(blocks * sizeof(unsigned int)); + if(block_list == NULL) + BAD_ERROR("Out of memory allocating block_list\n"); + block_listp = block_list; + + buffer_list = malloc(blocks * sizeof(struct file_buffer *)); + if(buffer_list == NULL) + BAD_ERROR("Out of memory allocating file block list\n"); + + num_locked_fragments = lock_fragments(); + + file_bytes = 0; + start = dup_start = bytes; + thresh = blocks > (writer_buffer_size - num_locked_fragments) ? + blocks - (writer_buffer_size - num_locked_fragments): 0; + + for(block = 0; block < blocks;) { + if(read_buffer->fragment && read_buffer->c_byte) { + fragment_buffer = read_buffer; + blocks = read_size >> block_log; + } else { + block_list[block] = read_buffer->c_byte; + + if(read_buffer->c_byte) { + read_buffer->block = bytes; + bytes += read_buffer->size; + file_bytes += read_buffer->size; + cache_rehash(read_buffer, read_buffer->block); + if(block < thresh) { + buffer_list[block] = NULL; + queue_put(to_writer, read_buffer); + } else + buffer_list[block] = read_buffer; + } else { + buffer_list[block] = NULL; + sparse += read_buffer->size; + cache_block_put(read_buffer); + } + } + inc_progress_bar(); + + if(++block < blocks) { + read_buffer = get_file_buffer(from_deflate); + if(read_buffer->error) + goto read_err; + } + } + + dupl_ptr = duplicate(read_size, file_bytes, &block_listp, &dup_start, + &fragment, fragment_buffer, blocks, 0, 0, FALSE); + + if(dupl_ptr) { + *duplicate_file = FALSE; + for(block = thresh; block < blocks; block ++) + if(buffer_list[block]) + queue_put(to_writer, buffer_list[block]); + fragment = get_and_fill_fragment(fragment_buffer); + dupl_ptr->fragment = fragment; + } else { + *duplicate_file = TRUE; + for(block = thresh; block < blocks; block ++) + cache_block_put(buffer_list[block]); + bytes = start; + if(thresh && !block_device) { + int res; + + queue_put(to_writer, NULL); + if(queue_get(from_writer) != 0) + EXIT_MKSQUASHFS(); + res = ftruncate(fd, bytes); + if(res != 0) + BAD_ERROR("Failed to truncate dest file because" + " %s\n", strerror(errno)); + } + } + + unlock_fragments(); + cache_block_put(fragment_buffer); + free(buffer_list); + file_count ++; + total_bytes += read_size; + + /* + * sparse count is needed to ensure squashfs correctly reports a + * a smaller block count on stat calls to sparse files. This is + * to ensure intelligent applications like cp correctly handle the + * file as a sparse file. If the file in the original filesystem isn't + * stored as a sparse file then still store it sparsely in squashfs, but + * report it as non-sparse on stat calls to preserve semantics + */ + if(sparse && (dir_ent->inode->buf.st_blocks << 9) >= read_size) + sparse = 0; + + create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, read_size, + dup_start, blocks, block_listp, fragment, NULL, sparse); + + if(*duplicate_file == TRUE) + free(block_list); + + return 0; + +read_err: + cur_uncompressed -= block; + status = read_buffer->error; + bytes = start; + if(thresh && !block_device) { + int res; + + queue_put(to_writer, NULL); + if(queue_get(from_writer) != 0) + EXIT_MKSQUASHFS(); + res = ftruncate(fd, bytes); + if(res != 0) + BAD_ERROR("Failed to truncate dest file because %s\n", + strerror(errno)); + } + unlock_fragments(); + for(blocks = thresh; blocks < block; blocks ++) + cache_block_put(buffer_list[blocks]); + free(buffer_list); + free(block_list); + cache_block_put(read_buffer); + return status; +} + + +void write_file(squashfs_inode *inode, struct dir_ent *dir_ent, + int *duplicate_file) +{ + int status; + struct file_buffer *read_buffer; + long long read_size; + +again: + read_buffer = get_file_buffer(from_deflate); + + status = read_buffer->error; + if(status) { + cache_block_put(read_buffer); + goto file_err; + } + + read_size = read_buffer->file_size; + + if(read_size == -1) + status = write_file_process(inode, dir_ent, read_buffer, + duplicate_file); + else if(read_size == 0) { + write_file_empty(inode, dir_ent, duplicate_file); + cache_block_put(read_buffer); + } else if(read_buffer->fragment && read_buffer->c_byte) + write_file_frag(inode, dir_ent, read_size, read_buffer, + duplicate_file); + else if(pre_duplicate(read_size)) + status = write_file_blocks_dup(inode, dir_ent, read_size, + read_buffer, duplicate_file); + else + status = write_file_blocks(inode, dir_ent, read_size, + read_buffer, duplicate_file); + +file_err: + if(status == 2) { + ERROR("File %s changed size while reading filesystem, " + "attempting to re-read\n", dir_ent->pathname); + goto again; + } else if(status == 1) { + ERROR("Failed to read file %s, creating empty file\n", + dir_ent->pathname); + write_file_empty(inode, dir_ent, duplicate_file); + } +} + + +#define BUFF_SIZE 8192 +char b_buffer[BUFF_SIZE]; +char *name; +char *basename_r(); + +char *getbase(char *pathname) +{ + char *result; + + if(*pathname != '/') { + result = getcwd(b_buffer, BUFF_SIZE); + if(result == NULL) + return NULL; + strcat(strcat(b_buffer, "/"), pathname); + } else + strcpy(b_buffer, pathname); + name = b_buffer; + if(((result = basename_r()) == NULL) || (strcmp(result, "..") == 0)) + return NULL; + else + return result; +} + + +char *basename_r() +{ + char *s; + char *p; + int n = 1; + + for(;;) { + s = name; + if(*name == '\0') + return NULL; + if(*name != '/') { + while(*name != '\0' && *name != '/') name++; + n = name - s; + } + while(*name == '/') name++; + if(strncmp(s, ".", n) == 0) + continue; + if((*name == '\0') || (strncmp(s, "..", n) == 0) || + ((p = basename_r()) == NULL)) { + s[n] = '\0'; + return s; + } + if(strcmp(p, "..") == 0) + continue; + return p; + } +} + + +struct inode_info *lookup_inode(struct stat *buf) +{ + int inode_hash = INODE_HASH(buf->st_dev, buf->st_ino); + struct inode_info *inode = inode_info[inode_hash]; + + while(inode != NULL) { + if(memcmp(buf, &inode->buf, sizeof(struct stat)) == 0) { + inode->nlink ++; + return inode; + } + inode = inode->next; + } + + inode = malloc(sizeof(struct inode_info)); + if(inode == NULL) + BAD_ERROR("Out of memory in inode hash table entry allocation" + "\n"); + + memcpy(&inode->buf, buf, sizeof(struct stat)); + inode->read = FALSE; + inode->root_entry = FALSE; + inode->pseudo_file = FALSE; + inode->inode = SQUASHFS_INVALID_BLK; + inode->nlink = 1; + + if((buf->st_mode & S_IFMT) == S_IFREG) + estimated_uncompressed += (buf->st_size + block_size - 1) >> + block_log; + + if((buf->st_mode & S_IFMT) == S_IFDIR) + inode->inode_number = dir_inode_no ++; + else + inode->inode_number = inode_no ++; + + inode->next = inode_info[inode_hash]; + inode_info[inode_hash] = inode; + + return inode; +} + + +inline void add_dir_entry(char *name, char *pathname, struct dir_info *sub_dir, + struct inode_info *inode_info, struct dir_info *dir) +{ + if((dir->count % DIR_ENTRIES) == 0) { + dir->list = realloc(dir->list, (dir->count + DIR_ENTRIES) * + sizeof(struct dir_ent *)); + if(dir->list == NULL) + BAD_ERROR("Out of memory in add_dir_entry\n"); + } + + dir->list[dir->count] = malloc(sizeof(struct dir_ent)); + if(dir->list[dir->count] == NULL) + BAD_ERROR("Out of memory in linux_opendir\n"); + + if(sub_dir) + sub_dir->dir_ent = dir->list[dir->count]; + dir->list[dir->count]->name = strdup(name); + dir->list[dir->count]->pathname = pathname != NULL ? strdup(pathname) : + NULL; + dir->list[dir->count]->inode = inode_info; + dir->list[dir->count]->dir = sub_dir; + dir->list[dir->count++]->our_dir = dir; + dir->byte_count += strlen(name) + sizeof(struct squashfs_dir_entry); +} + + +int compare_name(const void *ent1_ptr, const void *ent2_ptr) +{ + struct dir_ent *ent1 = *((struct dir_ent **) ent1_ptr); + struct dir_ent *ent2 = *((struct dir_ent **) ent2_ptr); + + return strcmp(ent1->name, ent2->name); +} + + +void sort_directory(struct dir_info *dir) +{ + qsort(dir->list, dir->count, sizeof(struct dir_ent *), compare_name); + + if((dir->count < 257 && dir->byte_count < SQUASHFS_METADATA_SIZE)) + dir->dir_is_ldir = FALSE; +} + + +struct dir_info *scan1_opendir(char *pathname) +{ + struct dir_info *dir; + + dir = malloc(sizeof(struct dir_info)); + if(dir == NULL) + BAD_ERROR("Out of memory in scan1_opendir\n"); + + if(pathname[0] != '\0' && (dir->linuxdir = opendir(pathname)) == NULL) { + free(dir); + return NULL; + } + dir->pathname = strdup(pathname); + dir->count = dir->directory_count = dir->current_count = dir->byte_count + = 0; + dir->dir_is_ldir = TRUE; + dir->list = NULL; + + return dir; +} + + +int scan1_encomp_readdir(char *pathname, char *dir_name, struct dir_info *dir) +{ + static int index = 0; + + if(dir->count < old_root_entries) { + int i; + + for(i = 0; i < old_root_entries; i++) { + if(old_root_entry[i].inode.type == SQUASHFS_DIR_TYPE) + dir->directory_count ++; + add_dir_entry(old_root_entry[i].name, "", NULL, + &old_root_entry[i].inode, dir); + } + } + + while(index < source) { + char *basename = getbase(source_path[index]); + int n, pass = 1; + + if(basename == NULL) { + ERROR("Bad source directory %s - skipping ...\n", + source_path[index]); + index ++; + continue; + } + strcpy(dir_name, basename); + for(;;) { + for(n = 0; n < dir->count && + strcmp(dir->list[n]->name, dir_name) != 0; n++); + if(n == dir->count) + break; + ERROR("Source directory entry %s already used! - trying" + " ", dir_name); + sprintf(dir_name, "%s_%d", basename, pass++); + ERROR("%s\n", dir_name); + } + strcpy(pathname, source_path[index ++]); + return 1; + } + return 0; +} + + +int scan1_single_readdir(char *pathname, char *dir_name, struct dir_info *dir) +{ + struct dirent *d_name; + int i; + + if(dir->count < old_root_entries) { + for(i = 0; i < old_root_entries; i++) { + if(old_root_entry[i].inode.type == SQUASHFS_DIR_TYPE) + dir->directory_count ++; + add_dir_entry(old_root_entry[i].name, "", NULL, + &old_root_entry[i].inode, dir); + } + } + + if((d_name = readdir(dir->linuxdir)) != NULL) { + int pass = 1; + + strcpy(dir_name, d_name->d_name); + for(;;) { + for(i = 0; i < dir->count && + strcmp(dir->list[i]->name, dir_name) != 0; i++); + if(i == dir->count) + break; + ERROR("Source directory entry %s already used! - trying" + " ", dir_name); + sprintf(dir_name, "%s_%d", d_name->d_name, pass++); + ERROR("%s\n", dir_name); + } + strcat(strcat(strcpy(pathname, dir->pathname), "/"), + d_name->d_name); + return 1; + } + + return 0; +} + + +int scan1_readdir(char *pathname, char *dir_name, struct dir_info *dir) +{ + struct dirent *d_name = readdir(dir->linuxdir); + + if(d_name != NULL) { + strcpy(dir_name, d_name->d_name); + strcat(strcat(strcpy(pathname, dir->pathname), "/"), + d_name->d_name); + return 1; + } + + return 0; +} + + +struct dir_ent *scan2_readdir(struct dir_info *dir_info) +{ + int current_count; + + while((current_count = dir_info->current_count++) < dir_info->count) + if(dir_info->list[current_count]->inode->root_entry) + continue; + else + return dir_info->list[current_count]; + return NULL; +} + + +struct dir_ent *scan2_lookup(struct dir_info *dir, char *name) +{ + int i; + + for(i = 0; i < dir->count; i++) + if(strcmp(dir->list[i]->name, name) == 0) + return dir->list[i]; + + return NULL; +} + + +struct dir_ent *scan3_readdir(struct directory *dir, struct dir_info *dir_info) +{ + int current_count; + + while((current_count = dir_info->current_count++) < dir_info->count) + if(dir_info->list[current_count]->inode->root_entry) + add_dir(dir_info->list[current_count]->inode->inode, + dir_info->list[current_count]->inode->inode_number, + dir_info->list[current_count]->name, + dir_info->list[current_count]->inode->type, dir); + else + return dir_info->list[current_count]; + return NULL; +} + + +void scan1_freedir(struct dir_info *dir) +{ + if(dir->pathname[0] != '\0') + closedir(dir->linuxdir); + free(dir->pathname); + dir->pathname = NULL; +} + + +void scan2_freedir(struct dir_info *dir) +{ + dir->current_count = 0; + if(dir->pathname) { + free(dir->pathname); + dir->pathname = NULL; + } +} + + +void scan3_freedir(struct directory *dir) +{ + if(dir->index) + free(dir->index); + free(dir->buff); +} + + +void dir_scan(squashfs_inode *inode, char *pathname, + int (_readdir)(char *, char *, struct dir_info *)) +{ + struct stat buf; + struct dir_info *dir_info = dir_scan1(pathname, paths, _readdir); + struct dir_ent *dir_ent; + + if(dir_info == NULL) + return; + + dir_scan2(dir_info, pseudo); + + dir_ent = malloc(sizeof(struct dir_ent)); + if(dir_ent == NULL) + BAD_ERROR("Out of memory in dir_scan\n"); + + if(pathname[0] == '\0') { + /* + * dummy top level directory, if multiple sources specified on + * command line + */ + memset(&buf, 0, sizeof(buf)); + buf.st_mode = S_IRWXU | S_IRWXG | S_IRWXO | S_IFDIR; + buf.st_uid = getuid(); + buf.st_gid = getgid(); + buf.st_mtime = time(NULL); + buf.st_dev = 0; + buf.st_ino = 0; + dir_ent->inode = lookup_inode(&buf); + dir_ent->inode->pseudo_file = PSEUDO_FILE_OTHER; + } else { + if(lstat(pathname, &buf) == -1) { + ERROR("Cannot stat dir/file %s because %s, ignoring", + pathname, strerror(errno)); + return; + } + dir_ent->inode = lookup_inode(&buf); + } + + if(root_inode_number) { + dir_ent->inode->inode_number = root_inode_number; + dir_inode_no --; + } + dir_ent->name = dir_ent->pathname = strdup(pathname); + dir_ent->dir = dir_info; + dir_ent->our_dir = NULL; + dir_info->dir_ent = dir_ent; + + if(sorted) { + int res = generate_file_priorities(dir_info, 0, + &dir_info->dir_ent->inode->buf); + + if(res == FALSE) + BAD_ERROR("generate_file_priorities failed\n"); + } + queue_put(to_reader, dir_info); + if(sorted) + sort_files_and_write(dir_info); + if(progress) + enable_progress_bar(); + dir_scan3(inode, dir_info); + dir_ent->inode->inode = *inode; + dir_ent->inode->type = SQUASHFS_DIR_TYPE; +} + + +struct dir_info *dir_scan1(char *pathname, struct pathnames *paths, + int (_readdir)(char *, char *, struct dir_info *)) +{ + char filename[8192], dir_name[8192]; + struct dir_info *dir = scan1_opendir(pathname); + + if(dir == NULL) { + ERROR("Could not open %s, skipping...\n", pathname); + goto error; + } + + while(_readdir(filename, dir_name, dir) != FALSE) { + struct dir_info *sub_dir; + struct stat buf; + struct pathnames *new; + + if(strcmp(dir_name, ".") == 0 || strcmp(dir_name, "..") == 0) + continue; + + if(lstat(filename, &buf) == -1) { + ERROR("Cannot stat dir/file %s because %s, ignoring", + filename, strerror(errno)); + continue; + } + + if((buf.st_mode & S_IFMT) != S_IFREG && + (buf.st_mode & S_IFMT) != S_IFDIR && + (buf.st_mode & S_IFMT) != S_IFLNK && + (buf.st_mode & S_IFMT) != S_IFCHR && + (buf.st_mode & S_IFMT) != S_IFBLK && + (buf.st_mode & S_IFMT) != S_IFIFO && + (buf.st_mode & S_IFMT) != S_IFSOCK) { + ERROR("File %s has unrecognised filetype %d, ignoring" + "\n", filename, buf.st_mode & S_IFMT); + continue; + } + + if(old_exclude) { + if(old_excluded(filename, &buf)) + continue; + } else { + if(excluded(paths, dir_name, &new)) + continue; + } + + if((buf.st_mode & S_IFMT) == S_IFDIR) { + sub_dir = dir_scan1(filename, new, scan1_readdir); + if(sub_dir == NULL) + continue; + dir->directory_count ++; + } else + sub_dir = NULL; + + add_dir_entry(dir_name, filename, sub_dir, lookup_inode(&buf), + dir); + } + + scan1_freedir(dir); + +error: + return dir; +} + + +struct dir_info *dir_scan2(struct dir_info *dir, struct pseudo *pseudo) +{ + struct dir_info *sub_dir; + struct dir_ent *dir_ent; + struct pseudo_entry *pseudo_ent; + struct stat buf; + static int pseudo_ino = 1; + + if(dir == NULL && (dir = scan1_opendir("")) == NULL) + return NULL; + + while((dir_ent = scan2_readdir(dir)) != NULL) { + struct inode_info *inode_info = dir_ent->inode; + struct stat *buf = &inode_info->buf; + char *name = dir_ent->name; + + if((buf->st_mode & S_IFMT) == S_IFDIR) + dir_scan2(dir_ent->dir, pseudo_subdir(name, pseudo)); + } + + while((pseudo_ent = pseudo_readdir(pseudo)) != NULL) { + dir_ent = scan2_lookup(dir, pseudo_ent->name); + if(pseudo_ent->dev->type == 'm') { + struct stat *buf; + if(dir_ent == NULL) { + ERROR("Pseudo modify file \"%s\" does not exist " + "in source filesystem. Ignoring.\n", + pseudo_ent->pathname); + continue; + } + if(dir_ent->inode->root_entry) { + ERROR("Pseudo modify file \"%s\" is a pre-existing" + " file in the filesystem being appended" + " to. It cannot be modified. " + "Ignoring.\n", pseudo_ent->pathname); + continue; + } + buf = &dir_ent->inode->buf; + buf->st_mode = (buf->st_mode & S_IFMT) | + pseudo_ent->dev->mode; + buf->st_uid = pseudo_ent->dev->uid; + buf->st_gid = pseudo_ent->dev->gid; + continue; + } + + if(dir_ent) { + if(dir_ent->inode->root_entry) + ERROR("Pseudo file \"%s\" is a pre-existing" + " file in the filesystem being appended" + " to. Ignoring.\n", + pseudo_ent->pathname); + else + ERROR("Pseudo file \"%s\" exists in source " + "filesystem \"%s\".\nIgnoring, " + "exclude it (-e/-ef) to override.\n", + pseudo_ent->pathname, + dir_ent->pathname); + continue; + } + + if(pseudo_ent->dev->type == 'd') { + sub_dir = dir_scan2(NULL, pseudo_ent->pseudo); + if(sub_dir == NULL) { + ERROR("Could not create pseudo directory \"%s\"" + ", skipping...\n", + pseudo_ent->pathname); + continue; + } + dir->directory_count ++; + } else + sub_dir = NULL; + + memset(&buf, 0, sizeof(buf)); + buf.st_mode = pseudo_ent->dev->mode; + buf.st_uid = pseudo_ent->dev->uid; + buf.st_gid = pseudo_ent->dev->gid; + buf.st_rdev = makedev(pseudo_ent->dev->major, + pseudo_ent->dev->minor); + buf.st_mtime = time(NULL); + buf.st_ino = pseudo_ino ++; + + if(pseudo_ent->dev->type == 'f') { +#ifdef USE_TMP_FILE + struct stat buf2; + int res = stat(pseudo_ent->dev->filename, &buf2); + struct inode_info *inode; + if(res == -1) { + ERROR("Stat on pseudo file \"%s\" failed, " + "skipping...", pseudo_ent->pathname); + continue; + } + buf.st_size = buf2.st_size; + inode = lookup_inode(&buf); + inode->pseudo_file = PSEUDO_FILE_OTHER; + add_dir_entry(pseudo_ent->name, + pseudo_ent->dev->filename, sub_dir, inode, + dir); +#else + struct inode_info *inode = lookup_inode(&buf); + inode->pseudo_id = pseudo_ent->dev->pseudo_id; + inode->pseudo_file = PSEUDO_FILE_PROCESS; + add_dir_entry(pseudo_ent->name, pseudo_ent->pathname, + sub_dir, inode, dir); +#endif + } else { + struct inode_info *inode = lookup_inode(&buf); + inode->pseudo_file = PSEUDO_FILE_OTHER; + add_dir_entry(pseudo_ent->name, pseudo_ent->pathname, + sub_dir, inode, dir); + } + } + + scan2_freedir(dir); + sort_directory(dir); + + return dir; +} + + +void dir_scan3(squashfs_inode *inode, struct dir_info *dir_info) +{ + int squashfs_type; + int duplicate_file; + char *pathname = dir_info->dir_ent->pathname; + struct directory dir; + struct dir_ent *dir_ent; + + scan3_init_dir(&dir); + + while((dir_ent = scan3_readdir(&dir, dir_info)) != NULL) { + struct inode_info *inode_info = dir_ent->inode; + struct stat *buf = &inode_info->buf; + char *filename = dir_ent->pathname; + char *dir_name = dir_ent->name; + unsigned int inode_number = ((buf->st_mode & S_IFMT) == S_IFDIR) + ? dir_ent->inode->inode_number : + dir_ent->inode->inode_number + dir_inode_no; + + if(dir_ent->inode->inode == SQUASHFS_INVALID_BLK) { + switch(buf->st_mode & S_IFMT) { + case S_IFREG: + squashfs_type = SQUASHFS_FILE_TYPE; + write_file(inode, dir_ent, + &duplicate_file); + INFO("file %s, uncompressed size %lld " + "bytes %s\n", filename, + (long long) buf->st_size, + duplicate_file ? "DUPLICATE" : + ""); + break; + + case S_IFDIR: + squashfs_type = SQUASHFS_DIR_TYPE; + dir_scan3(inode, dir_ent->dir); + break; + + case S_IFLNK: + squashfs_type = SQUASHFS_SYMLINK_TYPE; + create_inode(inode, NULL, dir_ent, + squashfs_type, 0, 0, 0, NULL, + NULL, NULL, 0); + INFO("symbolic link %s inode 0x%llx\n", + dir_name, *inode); + sym_count ++; + break; + + case S_IFCHR: + squashfs_type = SQUASHFS_CHRDEV_TYPE; + create_inode(inode, NULL, dir_ent, + squashfs_type, 0, 0, 0, NULL, + NULL, NULL, 0); + INFO("character device %s inode 0x%llx" + "\n", dir_name, *inode); + dev_count ++; + break; + + case S_IFBLK: + squashfs_type = SQUASHFS_BLKDEV_TYPE; + create_inode(inode, NULL, dir_ent, + squashfs_type, 0, 0, 0, NULL, + NULL, NULL, 0); + INFO("block device %s inode 0x%llx\n", + dir_name, *inode); + dev_count ++; + break; + + case S_IFIFO: + squashfs_type = SQUASHFS_FIFO_TYPE; + create_inode(inode, NULL, dir_ent, + squashfs_type, 0, 0, 0, NULL, + NULL, NULL, 0); + INFO("fifo %s inode 0x%llx\n",dir_name, + *inode); + fifo_count ++; + break; + + case S_IFSOCK: + squashfs_type = SQUASHFS_SOCKET_TYPE; + create_inode(inode, NULL, dir_ent, + squashfs_type, 0, 0, 0, NULL, + NULL, NULL, 0); + INFO("unix domain socket %s inode " + "0x%llx\n", dir_name, *inode); + sock_count ++; + break; + + default: + BAD_ERROR("%s unrecognised file type, " + "mode is %x\n", filename, + buf->st_mode); + } + dir_ent->inode->inode = *inode; + dir_ent->inode->type = squashfs_type; + } else { + *inode = dir_ent->inode->inode; + squashfs_type = dir_ent->inode->type; + switch(squashfs_type) { + case SQUASHFS_FILE_TYPE: + if(!sorted) + INFO("file %s, uncompressed " + "size %lld bytes LINK" + "\n", filename, + (long long) + buf->st_size); + break; + case SQUASHFS_SYMLINK_TYPE: + INFO("symbolic link %s inode 0x%llx " + "LINK\n", dir_name, *inode); + break; + case SQUASHFS_CHRDEV_TYPE: + INFO("character device %s inode 0x%llx " + "LINK\n", dir_name, *inode); + break; + case SQUASHFS_BLKDEV_TYPE: + INFO("block device %s inode 0x%llx " + "LINK\n", dir_name, *inode); + break; + case SQUASHFS_FIFO_TYPE: + INFO("fifo %s inode 0x%llx LINK\n", + dir_name, *inode); + break; + case SQUASHFS_SOCKET_TYPE: + INFO("unix domain socket %s inode " + "0x%llx LINK\n", dir_name, + *inode); + break; + } + } + + add_dir(*inode, inode_number, dir_name, squashfs_type, &dir); + update_progress_bar(); + } + + write_dir(inode, dir_info, &dir); + INFO("directory %s inode 0x%llx\n", pathname, *inode); + + scan3_freedir(&dir); +} + + +unsigned int slog(unsigned int block) +{ + int i; + + for(i = 12; i <= 20; i++) + if(block == (1 << i)) + return i; + return 0; +} + + +int old_excluded(char *filename, struct stat *buf) +{ + int i; + + for(i = 0; i < exclude; i++) + if((exclude_paths[i].st_dev == buf->st_dev) && + (exclude_paths[i].st_ino == buf->st_ino)) + return TRUE; + return FALSE; +} + + +#define ADD_ENTRY(buf) \ + if(exclude % EXCLUDE_SIZE == 0) { \ + exclude_paths = realloc(exclude_paths, (exclude + EXCLUDE_SIZE) \ + * sizeof(struct exclude_info)); \ + if(exclude_paths == NULL) \ + BAD_ERROR("Out of memory in exclude dir/file table\n"); \ + } \ + exclude_paths[exclude].st_dev = buf.st_dev; \ + exclude_paths[exclude++].st_ino = buf.st_ino; +int old_add_exclude(char *path) +{ + int i; + char filename[4096]; + struct stat buf; + + if(path[0] == '/' || strncmp(path, "./", 2) == 0 || + strncmp(path, "../", 3) == 0) { + if(lstat(path, &buf) == -1) { + ERROR("Cannot stat exclude dir/file %s because %s, " + "ignoring", path, strerror(errno)); + return TRUE; + } + ADD_ENTRY(buf); + return TRUE; + } + + for(i = 0; i < source; i++) { + strcat(strcat(strcpy(filename, source_path[i]), "/"), path); + if(lstat(filename, &buf) == -1) { + if(!(errno == ENOENT || errno == ENOTDIR)) + ERROR("Cannot stat exclude dir/file %s because " + "%s, ignoring", filename, + strerror(errno)); + continue; + } + ADD_ENTRY(buf); + } + return TRUE; +} + + +void add_old_root_entry(char *name, squashfs_inode inode, int inode_number, + int type) +{ + old_root_entry = realloc(old_root_entry, + sizeof(struct old_root_entry_info) * (old_root_entries + 1)); + if(old_root_entry == NULL) + BAD_ERROR("Out of memory in old root directory entries " + "reallocation\n"); + + old_root_entry[old_root_entries].name = strdup(name); + old_root_entry[old_root_entries].inode.inode = inode; + old_root_entry[old_root_entries].inode.inode_number = inode_number; + old_root_entry[old_root_entries].inode.type = type; + old_root_entry[old_root_entries++].inode.root_entry = TRUE; +} + + +void initialise_threads(int readb_mbytes, int writeb_mbytes, + int fragmentb_mbytes) +{ + int i; + sigset_t sigmask, old_mask; + int reader_buffer_size = readb_mbytes << (20 - block_log); + int fragment_buffer_size = fragmentb_mbytes << (20 - block_log); + + /* + * writer_buffer_size is global because it is needed in + * write_file_blocks_dup() + */ + writer_buffer_size = writeb_mbytes << (20 - block_log); + + sigemptyset(&sigmask); + sigaddset(&sigmask, SIGINT); + sigaddset(&sigmask, SIGQUIT); + if(sigprocmask(SIG_BLOCK, &sigmask, &old_mask) == -1) + BAD_ERROR("Failed to set signal mask in intialise_threads\n"); + + signal(SIGUSR1, sigusr1_handler); + + if(processors == -1) { +#ifndef linux + int mib[2]; + size_t len = sizeof(processors); + + mib[0] = CTL_HW; +#ifdef HW_AVAILCPU + mib[1] = HW_AVAILCPU; +#else + mib[1] = HW_NCPU; +#endif + + if(sysctl(mib, 2, &processors, &len, NULL, 0) == -1) { + ERROR("Failed to get number of available processors. " + "Defaulting to 1\n"); + processors = 1; + } +#else + processors = sysconf(_SC_NPROCESSORS_ONLN); +#endif + } + + thread = malloc((2 + processors * 2) * sizeof(pthread_t)); + if(thread == NULL) + BAD_ERROR("Out of memory allocating thread descriptors\n"); + deflator_thread = &thread[2]; + frag_deflator_thread = &deflator_thread[processors]; + + to_reader = queue_init(1); + from_reader = queue_init(reader_buffer_size); + to_writer = queue_init(writer_buffer_size); + from_writer = queue_init(1); + from_deflate = queue_init(reader_buffer_size); + to_frag = queue_init(fragment_buffer_size); + reader_buffer = cache_init(block_size, reader_buffer_size); + writer_buffer = cache_init(block_size, writer_buffer_size); + fragment_buffer = cache_init(block_size, fragment_buffer_size); + pthread_create(&thread[0], NULL, reader, NULL); + pthread_create(&thread[1], NULL, writer, NULL); + pthread_create(&progress_thread, NULL, progress_thrd, NULL); + pthread_mutex_init(&fragment_mutex, NULL); + pthread_cond_init(&fragment_waiting, NULL); + + for(i = 0; i < processors; i++) { + if(pthread_create(&deflator_thread[i], NULL, deflator, NULL) != + 0) + BAD_ERROR("Failed to create thread\n"); + if(pthread_create(&frag_deflator_thread[i], NULL, frag_deflator, + NULL) != 0) + BAD_ERROR("Failed to create thread\n"); + } + + printf("Parallel mksquashfs: Using %d processor%s\n", processors, + processors == 1 ? "" : "s"); + + if(sigprocmask(SIG_SETMASK, &old_mask, NULL) == -1) + BAD_ERROR("Failed to set signal mask in intialise_threads\n"); +} + + +long long write_inode_lookup_table() +{ + int i, inode_number, lookup_bytes = SQUASHFS_LOOKUP_BYTES(inode_count); + void *it; + + if(inode_count == sinode_count) + goto skip_inode_hash_table; + + it = realloc(inode_lookup_table, lookup_bytes); + if(it == NULL) + BAD_ERROR("Out of memory in write_inode_table\n"); + inode_lookup_table = it; + + for(i = 0; i < INODE_HASH_SIZE; i ++) { + struct inode_info *inode = inode_info[i]; + + for(inode = inode_info[i]; inode; inode = inode->next) { + + inode_number = inode->type == SQUASHFS_DIR_TYPE ? + inode->inode_number : inode->inode_number + + dir_inode_no; + + SQUASHFS_SWAP_LONG_LONGS(&inode->inode, + &inode_lookup_table[inode_number - 1], 1); + + } + } + +skip_inode_hash_table: + return generic_write_table(lookup_bytes, inode_lookup_table, 0, NULL, + noI); +} + + +char *get_component(char *target, char *targname) +{ + while(*target == '/') + target ++; + + while(*target != '/' && *target!= '\0') + *targname ++ = *target ++; + + *targname = '\0'; + + return target; +} + + +void free_path(struct pathname *paths) +{ + int i; + + for(i = 0; i < paths->names; i++) { + if(paths->name[i].paths) + free_path(paths->name[i].paths); + free(paths->name[i].name); + if(paths->name[i].preg) { + regfree(paths->name[i].preg); + free(paths->name[i].preg); + } + } + + free(paths); +} + + +struct pathname *add_path(struct pathname *paths, char *target, char *alltarget) +{ + char targname[1024]; + int i, error; + + target = get_component(target, targname); + + if(paths == NULL) { + paths = malloc(sizeof(struct pathname)); + if(paths == NULL) + BAD_ERROR("failed to allocate paths\n"); + + paths->names = 0; + paths->name = NULL; + } + + for(i = 0; i < paths->names; i++) + if(strcmp(paths->name[i].name, targname) == 0) + break; + + if(i == paths->names) { + /* allocate new name entry */ + paths->names ++; + paths->name = realloc(paths->name, (i + 1) * + sizeof(struct path_entry)); + if(paths->name == NULL) + BAD_ERROR("Out of memory in add path\n"); + paths->name[i].name = strdup(targname); + paths->name[i].paths = NULL; + if(use_regex) { + paths->name[i].preg = malloc(sizeof(regex_t)); + if(paths->name[i].preg == NULL) + BAD_ERROR("Out of memory in add_path\n"); + error = regcomp(paths->name[i].preg, targname, + REG_EXTENDED|REG_NOSUB); + if(error) { + char str[1024]; + + regerror(error, paths->name[i].preg, str, 1024); + BAD_ERROR("invalid regex %s in export %s, " + "because %s\n", targname, alltarget, + str); + } + } else + paths->name[i].preg = NULL; + + if(target[0] == '\0') + /* at leaf pathname component */ + paths->name[i].paths = NULL; + else + /* recurse adding child components */ + paths->name[i].paths = add_path(NULL, target, + alltarget); + } else { + /* existing matching entry */ + if(paths->name[i].paths == NULL) { + /* No sub-directory which means this is the leaf + * component of a pre-existing exclude which subsumes + * the exclude currently being added, in which case stop + * adding components */ + } else if(target[0] == '\0') { + /* at leaf pathname component and child components exist + * from more specific excludes, delete as they're + * subsumed by this exclude */ + free_path(paths->name[i].paths); + paths->name[i].paths = NULL; + } else + /* recurse adding child components */ + add_path(paths->name[i].paths, target, alltarget); + } + + return paths; +} + + +void add_exclude(char *target) +{ + + if(target[0] == '/' || strncmp(target, "./", 2) == 0 || + strncmp(target, "../", 3) == 0) + BAD_ERROR("/, ./ and ../ prefixed excludes not supported with " + "-wildcards or -regex options\n"); + else if(strncmp(target, "... ", 4) == 0) + stickypath = add_path(stickypath, target + 4, target + 4); + else + path = add_path(path, target, target); +} + + +void display_path(int depth, struct pathname *paths) +{ + int i, n; + + if(paths == NULL) + return; + + for(i = 0; i < paths->names; i++) { + for(n = 0; n < depth; n++) + printf("\t"); + printf("%d: %s\n", depth, paths->name[i].name); + display_path(depth + 1, paths->name[i].paths); + } +} + + +void display_path2(struct pathname *paths, char *string) +{ + int i; + char path[1024]; + + if(paths == NULL) { + printf("%s\n", string); + return; + } + + for(i = 0; i < paths->names; i++) { + strcat(strcat(strcpy(path, string), "/"), paths->name[i].name); + display_path2(paths->name[i].paths, path); + } +} + + +struct pathnames *init_subdir() +{ + struct pathnames *new = malloc(sizeof(struct pathnames)); + if(new == NULL) + BAD_ERROR("Out of memory in init_subdir\n"); + new->count = 0; + return new; +} + + +struct pathnames *add_subdir(struct pathnames *paths, struct pathname *path) +{ + if(paths->count % PATHS_ALLOC_SIZE == 0) { + paths = realloc(paths, sizeof(struct pathnames *) + + (paths->count + PATHS_ALLOC_SIZE) * + sizeof(struct pathname *)); + if(paths == NULL) + BAD_ERROR("Out of memory in add_subdir\n"); + } + + paths->path[paths->count++] = path; + return paths; +} + + +void free_subdir(struct pathnames *paths) +{ + free(paths); +} + + +int excluded(struct pathnames *paths, char *name, struct pathnames **new) +{ + int i, n, res; + + if(paths == NULL) { + *new = NULL; + return FALSE; + } + + + *new = init_subdir(); + if(stickypath) + *new = add_subdir(*new, stickypath); + + for(n = 0; n < paths->count; n++) { + struct pathname *path = paths->path[n]; + + for(i = 0; i < path->names; i++) { + int match = use_regex ? + regexec(path->name[i].preg, name, (size_t) 0, + NULL, 0) == 0 : + fnmatch(path->name[i].name, name, + FNM_PATHNAME|FNM_PERIOD|FNM_EXTMATCH) == + 0; + + if(match && path->name[i].paths == NULL) { + /* match on a leaf component, any subdirectories + * in the filesystem should be excluded */ + res = TRUE; + goto empty_set; + } + + if(match) + /* match on a non-leaf component, add any + * subdirectories to the new set of + * subdirectories to scan for this name */ + *new = add_subdir(*new, path->name[i].paths); + } + } + + if((*new)->count == 0) { + /* no matching names found, return empty new search set + */ + res = FALSE; + goto empty_set; + } + + /* one or more matches with sub-directories found (no leaf matches). + * Return new set */ + return FALSE; + +empty_set: + free_subdir(*new); + *new = NULL; + return res; +} + + +#define RECOVER_ID "Squashfs recovery file v1.0\n" +#define RECOVER_ID_SIZE 28 + +void write_recovery_data(struct squashfs_super_block *sBlk) +{ + int res, recoverfd, bytes = sBlk->bytes_used - sBlk->inode_table_start; + pid_t pid = getpid(); + char *metadata; + char header[] = RECOVER_ID; + + if(recover == FALSE) { + printf("No recovery data option specified.\n"); + printf("Skipping saving recovery file.\n\n"); + return; + } + + metadata = malloc(bytes); + if(metadata == NULL) + BAD_ERROR("Failed to alloc metadata buffer in " + "write_recovery_data\n"); + + res = read_fs_bytes(fd, sBlk->inode_table_start, bytes, metadata); + if(res == 0) + EXIT_MKSQUASHFS(); + + sprintf(recovery_file, "squashfs_recovery_%s_%d", + getbase(destination_file), pid); + recoverfd = open(recovery_file, O_CREAT | O_TRUNC | O_RDWR, S_IRWXU); + if(recoverfd == -1) + BAD_ERROR("Failed to create recovery file, because %s. " + "Aborting\n", strerror(errno)); + + if(write_bytes(recoverfd, header, RECOVER_ID_SIZE) == -1) + BAD_ERROR("Failed to write recovery file, because %s\n", + strerror(errno)); + + if(write_bytes(recoverfd, sBlk, sizeof(struct squashfs_super_block)) == -1) + BAD_ERROR("Failed to write recovery file, because %s\n", + strerror(errno)); + + if(write_bytes(recoverfd, metadata, bytes) == -1) + BAD_ERROR("Failed to write recovery file, because %s\n", + strerror(errno)); + + close(recoverfd); + free(metadata); + + printf("Recovery file \"%s\" written\n", recovery_file); + printf("If Mksquashfs aborts abnormally (i.e. power failure), run\n"); + printf("mksquashfs dummy %s -recover %s\n", destination_file, + recovery_file); + printf("to restore filesystem\n\n"); +} + + +void read_recovery_data(char *recovery_file, char *destination_file) +{ + int fd, recoverfd, bytes; + struct squashfs_super_block orig_sBlk, sBlk; + char *metadata; + int res; + struct stat buf; + char header[] = RECOVER_ID; + char header2[RECOVER_ID_SIZE]; + + recoverfd = open(recovery_file, O_RDONLY); + if(recoverfd == -1) + BAD_ERROR("Failed to open recovery file because %s\n", + strerror(errno)); + + if(stat(destination_file, &buf) == -1) + BAD_ERROR("Failed to stat destination file, because %s\n", + strerror(errno)); + + fd = open(destination_file, O_RDWR); + if(fd == -1) + BAD_ERROR("Failed to open destination file because %s\n", + strerror(errno)); + + res = read_bytes(recoverfd, header2, RECOVER_ID_SIZE); + if(res == -1) + BAD_ERROR("Failed to read recovery file, because %s\n", + strerror(errno)); + if(res < RECOVER_ID_SIZE) + BAD_ERROR("Recovery file appears to be truncated\n"); + if(strncmp(header, header2, RECOVER_ID_SIZE) !=0 ) + BAD_ERROR("Not a recovery file\n"); + + res = read_bytes(recoverfd, &sBlk, sizeof(struct squashfs_super_block)); + if(res == -1) + BAD_ERROR("Failed to read recovery file, because %s\n", + strerror(errno)); + if(res < sizeof(struct squashfs_super_block)) + BAD_ERROR("Recovery file appears to be truncated\n"); + + res = read_fs_bytes(fd, 0, sizeof(struct squashfs_super_block), &orig_sBlk); + if(res == 0) + EXIT_MKSQUASHFS(); + + if(memcmp(((char *) &sBlk) + 4, ((char *) &orig_sBlk) + 4, + sizeof(struct squashfs_super_block) - 4) != 0) + BAD_ERROR("Recovery file and destination file do not seem to " + "match\n"); + + bytes = sBlk.bytes_used - sBlk.inode_table_start; + + metadata = malloc(bytes); + if(metadata == NULL) + BAD_ERROR("Failed to alloc metadata buffer in " + "read_recovery_data\n"); + + res = read_bytes(recoverfd, metadata, bytes); + if(res == -1) + BAD_ERROR("Failed to read recovery file, because %s\n", + strerror(errno)); + if(res < bytes) + BAD_ERROR("Recovery file appears to be truncated\n"); + + write_destination(fd, 0, sizeof(struct squashfs_super_block), &sBlk); + + write_destination(fd, sBlk.inode_table_start, bytes, metadata); + + close(recoverfd); + close(fd); + + printf("Successfully wrote recovery file \"%s\". Exiting\n", + recovery_file); + + exit(0); +} + + +#define VERSION() \ + printf("mksquashfs version 4.2 (2011/02/28)\n");\ + printf("copyright (C) 2011 Phillip Lougher "\ + "\n\n"); \ + printf("This program is free software; you can redistribute it and/or"\ + "\n");\ + printf("modify it under the terms of the GNU General Public License"\ + "\n");\ + printf("as published by the Free Software Foundation; either version "\ + "2,\n");\ + printf("or (at your option) any later version.\n\n");\ + printf("This program is distributed in the hope that it will be "\ + "useful,\n");\ + printf("but WITHOUT ANY WARRANTY; without even the implied warranty "\ + "of\n");\ + printf("MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the"\ + "\n");\ + printf("GNU General Public License for more details.\n"); +int main(int argc, char *argv[]) +{ + struct stat buf, source_buf; + int res, i; + struct squashfs_super_block sBlk; + char *b, *root_name = NULL; + int nopad = FALSE, keep_as_directory = FALSE; + squashfs_inode inode; + int readb_mbytes = READER_BUFFER_DEFAULT, + writeb_mbytes = WRITER_BUFFER_DEFAULT, + fragmentb_mbytes = FRAGMENT_BUFFER_DEFAULT; + + pthread_mutex_init(&progress_mutex, NULL); + block_log = slog(block_size); + if(argc > 1 && strcmp(argv[1], "-version") == 0) { + VERSION(); + exit(0); + } + for(i = 1; i < argc && argv[i][0] != '-'; i++); + if(i < 3) + goto printOptions; + source_path = argv + 1; + source = i - 2; + /* + * lookup default compressor. Note the Makefile ensures the default + * compressor has been built, and so we don't need to to check + * for failure here + */ + comp = lookup_compressor(COMP_DEFAULT); + for(; i < argc; i++) { + if(strcmp(argv[i], "-comp") == 0) { + if(compressor_opts_parsed) { + ERROR("%s: -comp must appear before -X options" + "\n", argv[0]); + exit(1); + } + if(++i == argc) { + ERROR("%s: -comp missing compression type\n", + argv[0]); + exit(1); + } + comp = lookup_compressor(argv[i]); + if(!comp->supported) { + ERROR("%s: Compressor \"%s\" is not supported!" + "\n", argv[0], argv[i]); + ERROR("%s: Compressors available:\n", argv[0]); + display_compressors("", COMP_DEFAULT); + exit(1); + } + + } else if(strncmp(argv[i], "-X", 2) == 0) { + int args = compressor_options(comp, argv + i, argc - i); + if(args < 0) { + if(args == -1) { + ERROR("%s: Unrecognised compressor" + " option %s\n", argv[0], + argv[i]); + ERROR("%s: Did you forget to specify" + " -comp, or specify it after" + " the compressor specific" + " option?\n", argv[0]); + } + exit(1); + } + i += args; + compressor_opts_parsed = 1; + + } else if(strcmp(argv[i], "-pf") == 0) { + if(++i == argc) { + ERROR("%s: -pf missing filename\n", argv[0]); + exit(1); + } + if(read_pseudo_file(&pseudo, argv[i]) == FALSE) + exit(1); + } else if(strcmp(argv[i], "-p") == 0) { + if(++i == argc) { + ERROR("%s: -p missing pseudo file definition\n", + argv[0]); + exit(1); + } + if(read_pseudo_def(&pseudo, argv[i]) == FALSE) + exit(1); + } else if(strcmp(argv[i], "-recover") == 0) { + if(++i == argc) { + ERROR("%s: -recover missing recovery file\n", + argv[0]); + exit(1); + } + read_recovery_data(argv[i], argv[source + 1]); + } else if(strcmp(argv[i], "-no-recovery") == 0) + recover = FALSE; + else if(strcmp(argv[i], "-wildcards") == 0) { + old_exclude = FALSE; + use_regex = FALSE; + } else if(strcmp(argv[i], "-regex") == 0) { + old_exclude = FALSE; + use_regex = TRUE; + } else if(strcmp(argv[i], "-no-sparse") == 0) + sparse_files = FALSE; + else if(strcmp(argv[i], "-no-progress") == 0) + progress = FALSE; + else if(strcmp(argv[i], "-no-exports") == 0) + exportable = FALSE; + else if(strcmp(argv[i], "-processors") == 0) { + if((++i == argc) || (processors = + strtol(argv[i], &b, 10), *b != '\0')) { + ERROR("%s: -processors missing or invalid " + "processor number\n", argv[0]); + exit(1); + } + if(processors < 1) { + ERROR("%s: -processors should be 1 or larger\n", + argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-read-queue") == 0) { + if((++i == argc) || (readb_mbytes = + strtol(argv[i], &b, 10), *b != '\0')) { + ERROR("%s: -read-queue missing or invalid " + "queue size\n", argv[0]); + exit(1); + } + if(readb_mbytes < 1) { + ERROR("%s: -read-queue should be 1 megabyte or " + "larger\n", argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-write-queue") == 0) { + if((++i == argc) || (writeb_mbytes = + strtol(argv[i], &b, 10), *b != '\0')) { + ERROR("%s: -write-queue missing or invalid " + "queue size\n", argv[0]); + exit(1); + } + if(writeb_mbytes < 1) { + ERROR("%s: -write-queue should be 1 megabyte " + "or larger\n", argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-fragment-queue") == 0) { + if((++i == argc) || + (fragmentb_mbytes = + strtol(argv[i], &b, 10), *b != '\0')) { + ERROR("%s: -fragment-queue missing or invalid " + "queue size\n", argv[0]); + exit(1); + } + if(fragmentb_mbytes < 1) { + ERROR("%s: -fragment-queue should be 1 " + "megabyte or larger\n", argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-b") == 0) { + if(++i == argc) { + ERROR("%s: -b missing block size\n", argv[0]); + exit(1); + } + block_size = strtol(argv[i], &b, 10); + if(*b == 'm' || *b == 'M') + block_size *= 1048576; + else if(*b == 'k' || *b == 'K') + block_size *= 1024; + else if(*b != '\0') { + ERROR("%s: -b invalid block size\n", argv[0]); + exit(1); + } + if((block_log = slog(block_size)) == 0) { + ERROR("%s: -b block size not power of two or " + "not between 4096 and 1Mbyte\n", + argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-ef") == 0) { + if(++i == argc) { + ERROR("%s: -ef missing filename\n", argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-no-duplicates") == 0) + duplicate_checking = FALSE; + + else if(strcmp(argv[i], "-no-fragments") == 0) + no_fragments = TRUE; + + else if(strcmp(argv[i], "-always-use-fragments") == 0) + always_use_fragments = TRUE; + + else if(strcmp(argv[i], "-sort") == 0) { + if(++i == argc) { + ERROR("%s: -sort missing filename\n", argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-all-root") == 0 || + strcmp(argv[i], "-root-owned") == 0) + global_uid = global_gid = 0; + + else if(strcmp(argv[i], "-force-uid") == 0) { + if(++i == argc) { + ERROR("%s: -force-uid missing uid or user\n", + argv[0]); + exit(1); + } + if((global_uid = strtoll(argv[i], &b, 10)), *b =='\0') { + if(global_uid < 0 || global_uid > + (((long long) 1 << 32) - 1)) { + ERROR("%s: -force-uid uid out of range" + "\n", argv[0]); + exit(1); + } + } else { + struct passwd *uid = getpwnam(argv[i]); + if(uid) + global_uid = uid->pw_uid; + else { + ERROR("%s: -force-uid invalid uid or " + "unknown user\n", argv[0]); + exit(1); + } + } + } else if(strcmp(argv[i], "-force-gid") == 0) { + if(++i == argc) { + ERROR("%s: -force-gid missing gid or group\n", + argv[0]); + exit(1); + } + if((global_gid = strtoll(argv[i], &b, 10)), *b =='\0') { + if(global_gid < 0 || global_gid > + (((long long) 1 << 32) - 1)) { + ERROR("%s: -force-gid gid out of range" + "\n", argv[0]); + exit(1); + } + } else { + struct group *gid = getgrnam(argv[i]); + if(gid) + global_gid = gid->gr_gid; + else { + ERROR("%s: -force-gid invalid gid or " + "unknown group\n", argv[0]); + exit(1); + } + } + } else if(strcmp(argv[i], "-noI") == 0 || + strcmp(argv[i], "-noInodeCompression") == 0) + noI = TRUE; + + else if(strcmp(argv[i], "-noD") == 0 || + strcmp(argv[i], "-noDataCompression") == 0) + noD = TRUE; + + else if(strcmp(argv[i], "-noF") == 0 || + strcmp(argv[i], "-noFragmentCompression") == 0) + noF = TRUE; + + else if(strcmp(argv[i], "-noX") == 0 || + strcmp(argv[i], "-noXattrCompression") == 0) + noX = TRUE; + + else if(strcmp(argv[i], "-no-xattrs") == 0) + no_xattrs = TRUE; + + else if(strcmp(argv[i], "-xattrs") == 0) + no_xattrs = FALSE; + + else if(strcmp(argv[i], "-nopad") == 0) + nopad = TRUE; + + else if(strcmp(argv[i], "-info") == 0) { + silent = FALSE; + progress = FALSE; + } + + else if(strcmp(argv[i], "-e") == 0) + break; + + else if(strcmp(argv[i], "-noappend") == 0) + delete = TRUE; + + else if(strcmp(argv[i], "-keep-as-directory") == 0) + keep_as_directory = TRUE; + + else if(strcmp(argv[i], "-root-becomes") == 0) { + if(++i == argc) { + ERROR("%s: -root-becomes: missing name\n", + argv[0]); + exit(1); + } + root_name = argv[i]; + } else if(strcmp(argv[i], "-version") == 0) { + VERSION(); + } else { + ERROR("%s: invalid option\n\n", argv[0]); +printOptions: + ERROR("SYNTAX:%s source1 source2 ... dest [options] " + "[-e list of exclude\ndirs/files]\n", argv[0]); + ERROR("\nFilesystem build options:\n"); + ERROR("-comp \t\tselect compression\n"); + ERROR("\t\t\tCompressors available:\n"); + display_compressors("\t\t\t", COMP_DEFAULT); + ERROR("-b \t\tset data block to " + ". Default %d bytes\n", + SQUASHFS_FILE_SIZE); + ERROR("-no-exports\t\tdon't make the filesystem " + "exportable via NFS\n"); + ERROR("-no-sparse\t\tdon't detect sparse files\n"); + ERROR("-no-xattrs\t\tdon't store extended attributes" + NOXOPT_STR "\n"); + ERROR("-xattrs\t\t\tstore extended attributes" XOPT_STR + "\n"); + ERROR("-noI\t\t\tdo not compress inode table\n"); + ERROR("-noD\t\t\tdo not compress data blocks\n"); + ERROR("-noF\t\t\tdo not compress fragment blocks\n"); + ERROR("-noX\t\t\tdo not compress extended " + "attributes\n"); + ERROR("-no-fragments\t\tdo not use fragments\n"); + ERROR("-always-use-fragments\tuse fragment blocks for " + "files larger than block size\n"); + ERROR("-no-duplicates\t\tdo not perform duplicate " + "checking\n"); + ERROR("-all-root\t\tmake all files owned by root\n"); + ERROR("-force-uid uid\t\tset all file uids to uid\n"); + ERROR("-force-gid gid\t\tset all file gids to gid\n"); + ERROR("-nopad\t\t\tdo not pad filesystem to a multiple " + "of 4K\n"); + ERROR("-keep-as-directory\tif one source directory is " + "specified, create a root\n"); + ERROR("\t\t\tdirectory containing that directory, " + "rather than the\n"); + ERROR("\t\t\tcontents of the directory\n"); + ERROR("\nFilesystem filter options:\n"); + ERROR("-p \tAdd pseudo file " + "definition\n"); + ERROR("-pf \tAdd list of pseudo file " + "definitions\n"); + ERROR("-sort \tsort files according to " + "priorities in . One\n"); + ERROR("\t\t\tfile or dir with priority per line. " + "Priority -32768 to\n"); + ERROR("\t\t\t32767, default priority 0\n"); + ERROR("-ef \tlist of exclude dirs/files." + " One per line\n"); + ERROR("-wildcards\t\tAllow extended shell wildcards " + "(globbing) to be used in\n\t\t\texclude " + "dirs/files\n"); + ERROR("-regex\t\t\tAllow POSIX regular expressions to " + "be used in exclude\n\t\t\tdirs/files\n"); + ERROR("\nFilesystem append options:\n"); + ERROR("-noappend\t\tdo not append to existing " + "filesystem\n"); + ERROR("-root-becomes \twhen appending source " + "files/directories, make the\n"); + ERROR("\t\t\toriginal root become a subdirectory in " + "the new root\n"); + ERROR("\t\t\tcalled , rather than adding the new " + "source items\n"); + ERROR("\t\t\tto the original root\n"); + ERROR("\nMksquashfs runtime options:\n"); + ERROR("-version\t\tprint version, licence and " + "copyright message\n"); + ERROR("-recover \t\trecover filesystem data " + "using recovery file \n"); + ERROR("-no-recovery\t\tdon't generate a recovery " + "file\n"); + ERROR("-info\t\t\tprint files written to filesystem\n"); + ERROR("-no-progress\t\tdon't display the progress " + "bar\n"); + ERROR("-processors \tUse processors." + " By default will use number of\n"); + ERROR("\t\t\tprocessors available\n"); + ERROR("-read-queue \tSet input queue to " + "Mbytes. Default %d Mbytes\n", + READER_BUFFER_DEFAULT); + ERROR("-write-queue \tSet output queue to " + "Mbytes. Default %d Mbytes\n", + WRITER_BUFFER_DEFAULT); + ERROR("-fragment-queue \tSet fragment queue to " + " Mbytes. Default %d Mbytes\n", + FRAGMENT_BUFFER_DEFAULT); + ERROR("\nMiscellaneous options:\n"); + ERROR("-root-owned\t\talternative name for -all-root" + "\n"); + ERROR("-noInodeCompression\talternative name for -noI" + "\n"); + ERROR("-noDataCompression\talternative name for -noD" + "\n"); + ERROR("-noFragmentCompression\talternative name for " + "-noF\n"); + ERROR("-noXattrCompression\talternative name for " + "-noX\n"); + ERROR("\nCompressors available and compressor specific " + "options:\n"); + display_compressor_usage(COMP_DEFAULT); + exit(1); + } + } + + /* + * Some compressors may need the options to be checked for validity + * once all the options have been processed + */ + res = compressor_options_post(comp, block_size); + if(res) + EXIT_MKSQUASHFS(); + + for(i = 0; i < source; i++) + if(lstat(source_path[i], &source_buf) == -1) { + fprintf(stderr, "Cannot stat source directory \"%s\" " + "because %s\n", source_path[i], + strerror(errno)); + EXIT_MKSQUASHFS(); + } + + destination_file = argv[source + 1]; + if(stat(argv[source + 1], &buf) == -1) { + if(errno == ENOENT) { /* Does not exist */ + fd = open(argv[source + 1], O_CREAT | O_TRUNC | O_RDWR, + S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); + if(fd == -1) { + perror("Could not create destination file"); + exit(1); + } + delete = TRUE; + } else { + perror("Could not stat destination file"); + exit(1); + } + + } else { + if(S_ISBLK(buf.st_mode)) { + if((fd = open(argv[source + 1], O_RDWR)) == -1) { + perror("Could not open block device as " + "destination"); + exit(1); + } + block_device = 1; + + } else if(S_ISREG(buf.st_mode)) { + fd = open(argv[source + 1], (delete ? O_TRUNC : 0) | + O_RDWR); + if(fd == -1) { + perror("Could not open regular file for " + "writing as destination"); + exit(1); + } + } + else { + ERROR("Destination not block device or regular file\n"); + exit(1); + } + + } + + signal(SIGTERM, sighandler2); + signal(SIGINT, sighandler2); + + /* + * process the exclude files - must be done afer destination file has + * been possibly created + */ + for(i = source + 2; i < argc; i++) + if(strcmp(argv[i], "-ef") == 0) { + FILE *fd; + char filename[16385]; + if((fd = fopen(argv[++i], "r")) == NULL) { + perror("Could not open exclude file..."); + EXIT_MKSQUASHFS(); + } + while(fscanf(fd, "%16384[^\n]\n", filename) != EOF) + if(old_exclude) + old_add_exclude(filename); + else + add_exclude(filename); + fclose(fd); + } else if(strcmp(argv[i], "-e") == 0) + break; + else if(strcmp(argv[i], "-root-becomes") == 0 || + strcmp(argv[i], "-sort") == 0 || + strcmp(argv[i], "-pf") == 0 || + strcmp(argv[i], "-comp") == 0) + i++; + + if(i != argc) { + if(++i == argc) { + ERROR("%s: -e missing arguments\n", argv[0]); + EXIT_MKSQUASHFS(); + } + while(i < argc) + if(old_exclude) + old_add_exclude(argv[i++]); + else + add_exclude(argv[i++]); + } + + /* process the sort files - must be done afer the exclude files */ + for(i = source + 2; i < argc; i++) + if(strcmp(argv[i], "-sort") == 0) { + int res = read_sort_file(argv[++i], source, + source_path); + if(res == FALSE) + BAD_ERROR("Failed to read sort file\n"); + sorted ++; + } else if(strcmp(argv[i], "-e") == 0) + break; + else if(strcmp(argv[i], "-root-becomes") == 0 || + strcmp(argv[i], "-ef") == 0 || + strcmp(argv[i], "-pf") == 0 || + strcmp(argv[i], "-comp") == 0) + i++; + +#ifdef SQUASHFS_TRACE + progress = FALSE; +#endif + + if(!delete) { + comp = read_super(fd, &sBlk, argv[source + 1]); + if(comp == NULL) { + ERROR("Failed to read existing filesystem - will not " + "overwrite - ABORTING!\n"); + ERROR("To force Mksquashfs to write to this block " + "device or file use -noappend\n"); + EXIT_MKSQUASHFS(); + } + + block_log = slog(block_size = sBlk.block_size); + noI = SQUASHFS_UNCOMPRESSED_INODES(sBlk.flags); + noD = SQUASHFS_UNCOMPRESSED_DATA(sBlk.flags); + noF = SQUASHFS_UNCOMPRESSED_FRAGMENTS(sBlk.flags); + noX = SQUASHFS_UNCOMPRESSED_XATTRS(sBlk.flags); + no_fragments = SQUASHFS_NO_FRAGMENTS(sBlk.flags); + always_use_fragments = SQUASHFS_ALWAYS_FRAGMENTS(sBlk.flags); + duplicate_checking = SQUASHFS_DUPLICATES(sBlk.flags); + exportable = SQUASHFS_EXPORTABLE(sBlk.flags); + no_xattrs = SQUASHFS_NO_XATTRS(sBlk.flags); + comp_opts = SQUASHFS_COMP_OPTS(sBlk.flags); + } + + initialise_threads(readb_mbytes, writeb_mbytes, fragmentb_mbytes); + + res = compressor_init(comp, &stream, SQUASHFS_METADATA_SIZE, 0); + if(res) + BAD_ERROR("compressor_init failed\n"); + + if(delete) { + int size; + void *comp_data = compressor_dump_options(comp, block_size, + &size); + + printf("Creating %d.%d filesystem on %s, block size %d.\n", + SQUASHFS_MAJOR, SQUASHFS_MINOR, argv[source + 1], block_size); + + /* + * store any compressor specific options after the superblock, + * and set the COMP_OPT flag to show that the filesystem has + * compressor specfic options + */ + if(comp_data) { + unsigned short c_byte = size | SQUASHFS_COMPRESSED_BIT; + + SQUASHFS_INSWAP_SHORTS(&c_byte, 1); + write_destination(fd, sizeof(struct squashfs_super_block), + sizeof(c_byte), &c_byte); + write_destination(fd, sizeof(struct squashfs_super_block) + + sizeof(c_byte), size, comp_data); + bytes = sizeof(struct squashfs_super_block) + sizeof(c_byte) + + size; + comp_opts = TRUE; + } else + bytes = sizeof(struct squashfs_super_block); + } else { + unsigned int last_directory_block, inode_dir_offset, + inode_dir_file_size, root_inode_size, + inode_dir_start_block, uncompressed_data, + compressed_data, inode_dir_inode_number, + inode_dir_parent_inode; + unsigned int root_inode_start = + SQUASHFS_INODE_BLK(sBlk.root_inode), + root_inode_offset = + SQUASHFS_INODE_OFFSET(sBlk.root_inode); + + if((bytes = read_filesystem(root_name, fd, &sBlk, &inode_table, + &data_cache, &directory_table, + &directory_data_cache, &last_directory_block, + &inode_dir_offset, &inode_dir_file_size, + &root_inode_size, &inode_dir_start_block, + &file_count, &sym_count, &dev_count, &dir_count, + &fifo_count, &sock_count, &total_bytes, + &total_inode_bytes, &total_directory_bytes, + &inode_dir_inode_number, + &inode_dir_parent_inode, add_old_root_entry, + &fragment_table, &inode_lookup_table)) == 0) { + ERROR("Failed to read existing filesystem - will not " + "overwrite - ABORTING!\n"); + ERROR("To force Mksquashfs to write to this block " + "device or file use -noappend\n"); + EXIT_MKSQUASHFS(); + } + if((fragments = sBlk.fragments)) { + fragment_table = realloc((char *) fragment_table, + ((fragments + FRAG_SIZE - 1) & ~(FRAG_SIZE - 1)) + * sizeof(struct squashfs_fragment_entry)); + if(fragment_table == NULL) + BAD_ERROR("Out of memory in save filesystem state\n"); + } + + printf("Appending to existing %d.%d filesystem on %s, block " + "size %d\n", SQUASHFS_MAJOR, SQUASHFS_MINOR, argv[source + 1], + block_size); + printf("All -b, -noI, -noD, -noF, -noX, no-duplicates, no-fragments, " + "-always-use-fragments,\n-exportable and -comp options " + "ignored\n"); + printf("\nIf appending is not wanted, please re-run with " + "-noappend specified!\n\n"); + + compressed_data = (inode_dir_offset + inode_dir_file_size) & + ~(SQUASHFS_METADATA_SIZE - 1); + uncompressed_data = (inode_dir_offset + inode_dir_file_size) & + (SQUASHFS_METADATA_SIZE - 1); + + /* save original filesystem state for restoring ... */ + sfragments = fragments; + sbytes = bytes; + sinode_count = sBlk.inodes; + scache_bytes = root_inode_offset + root_inode_size; + sdirectory_cache_bytes = uncompressed_data; + sdata_cache = malloc(scache_bytes); + if(sdata_cache == NULL) + BAD_ERROR("Out of memory in save filesystem state\n"); + sdirectory_data_cache = malloc(sdirectory_cache_bytes); + if(sdirectory_data_cache == NULL) + BAD_ERROR("Out of memory in save filesystem state\n"); + memcpy(sdata_cache, data_cache, scache_bytes); + memcpy(sdirectory_data_cache, directory_data_cache + + compressed_data, sdirectory_cache_bytes); + sinode_bytes = root_inode_start; + stotal_bytes = total_bytes; + stotal_inode_bytes = total_inode_bytes; + stotal_directory_bytes = total_directory_bytes + + compressed_data; + sfile_count = file_count; + ssym_count = sym_count; + sdev_count = dev_count; + sdir_count = dir_count + 1; + sfifo_count = fifo_count; + ssock_count = sock_count; + sdup_files = dup_files; + sid_count = id_count; + write_recovery_data(&sBlk); + if(save_xattrs() == FALSE) + BAD_ERROR("Failed to save xattrs from existing " + "filesystem\n"); + restore = TRUE; + if(setjmp(env)) + goto restore_filesystem; + signal(SIGTERM, sighandler); + signal(SIGINT, sighandler); + write_destination(fd, SQUASHFS_START, 4, "\0\0\0\0"); + + /* + * set the filesystem state up to be able to append to the + * original filesystem. The filesystem state differs depending + * on whether we're appending to the original root directory, or + * if the original root directory becomes a sub-directory + * (root-becomes specified on command line, here root_name != + * NULL) + */ + inode_bytes = inode_size = root_inode_start; + directory_size = last_directory_block; + cache_size = root_inode_offset + root_inode_size; + directory_cache_size = inode_dir_offset + inode_dir_file_size; + if(root_name) { + sdirectory_bytes = last_directory_block; + sdirectory_compressed_bytes = 0; + root_inode_number = inode_dir_parent_inode; + dir_inode_no = sBlk.inodes + 2; + directory_bytes = last_directory_block; + directory_cache_bytes = uncompressed_data; + memmove(directory_data_cache, directory_data_cache + + compressed_data, uncompressed_data); + cache_bytes = root_inode_offset + root_inode_size; + add_old_root_entry(root_name, sBlk.root_inode, + inode_dir_inode_number, SQUASHFS_DIR_TYPE); + total_directory_bytes += compressed_data; + dir_count ++; + } else { + sdirectory_compressed_bytes = last_directory_block - + inode_dir_start_block; + sdirectory_compressed = + malloc(sdirectory_compressed_bytes); + if(sdirectory_compressed == NULL) + BAD_ERROR("Out of memory in save filesystem " + "state\n"); + memcpy(sdirectory_compressed, directory_table + + inode_dir_start_block, + sdirectory_compressed_bytes); + sdirectory_bytes = inode_dir_start_block; + root_inode_number = inode_dir_inode_number; + dir_inode_no = sBlk.inodes + 1; + directory_bytes = inode_dir_start_block; + directory_cache_bytes = inode_dir_offset; + cache_bytes = root_inode_offset; + } + + inode_count = file_count + dir_count + sym_count + dev_count + + fifo_count + sock_count; + + /* + * The default use freelist before growing cache policy behaves + * poorly with appending - with many deplicates the caches + * do not grow due to the fact that large queues of outstanding + * fragments/writer blocks do not occur, leading to small caches + * and un-uncessary performance loss to frequent cache + * replacement in the small caches. Therefore with appending + * change the policy to grow the caches before reusing blocks + * from the freelist + */ + first_freelist = FALSE; + } + + if(path || stickypath) { + paths = init_subdir(); + if(path) + paths = add_subdir(paths, path); + if(stickypath) + paths = add_subdir(paths, stickypath); + } + + if(delete && !keep_as_directory && source == 1 && + S_ISDIR(source_buf.st_mode)) + dir_scan(&inode, source_path[0], scan1_readdir); + else if(!keep_as_directory && source == 1 && + S_ISDIR(source_buf.st_mode)) + dir_scan(&inode, source_path[0], scan1_single_readdir); + else + dir_scan(&inode, "", scan1_encomp_readdir); + sBlk.root_inode = inode; + sBlk.inodes = inode_count; + sBlk.s_magic = SQUASHFS_MAGIC; + sBlk.s_major = SQUASHFS_MAJOR; + sBlk.s_minor = SQUASHFS_MINOR; + sBlk.block_size = block_size; + sBlk.block_log = block_log; + sBlk.flags = SQUASHFS_MKFLAGS(noI, noD, noF, noX, no_fragments, + always_use_fragments, duplicate_checking, exportable, + no_xattrs, comp_opts); + sBlk.mkfs_time = time(NULL); + +restore_filesystem: + if(progress && estimated_uncompressed) { + disable_progress_bar(); + progress_bar(cur_uncompressed, estimated_uncompressed, columns); + } + + write_fragment(); + sBlk.fragments = fragments; + if(!restoring) { + unlock_fragments(); + pthread_mutex_lock(&fragment_mutex); + while(fragments_outstanding) { + pthread_mutex_unlock(&fragment_mutex); + sched_yield(); + pthread_mutex_lock(&fragment_mutex); + } + queue_put(to_writer, NULL); + if(queue_get(from_writer) != 0) + EXIT_MKSQUASHFS(); + } + + sBlk.no_ids = id_count; + sBlk.inode_table_start = write_inodes(); + sBlk.directory_table_start = write_directories(); + sBlk.fragment_table_start = write_fragment_table(); + sBlk.lookup_table_start = exportable ? write_inode_lookup_table() : + SQUASHFS_INVALID_BLK; + sBlk.id_table_start = write_id_table(); + sBlk.xattr_id_table_start = write_xattrs(); + + TRACE("sBlk->inode_table_start 0x%llx\n", sBlk.inode_table_start); + TRACE("sBlk->directory_table_start 0x%llx\n", + sBlk.directory_table_start); + TRACE("sBlk->fragment_table_start 0x%llx\n", sBlk.fragment_table_start); + if(exportable) + TRACE("sBlk->lookup_table_start 0x%llx\n", + sBlk.lookup_table_start); + + sBlk.bytes_used = bytes; + + sBlk.compression = comp->id; + + SQUASHFS_INSWAP_SUPER_BLOCK(&sBlk); + write_destination(fd, SQUASHFS_START, sizeof(sBlk), &sBlk); + + if(!nopad && (i = bytes & (4096 - 1))) { + char temp[4096] = {0}; + write_destination(fd, bytes, 4096 - i, temp); + } + + close(fd); + + delete_pseudo_files(); + + if(recovery_file[0] != '\0') + unlink(recovery_file); + + total_bytes += total_inode_bytes + total_directory_bytes + + sizeof(struct squashfs_super_block) + total_xattr_bytes; + + printf("\n%sSquashfs %d.%d filesystem, %s compressed, data block size" + " %d\n", exportable ? "Exportable " : "", SQUASHFS_MAJOR, + SQUASHFS_MINOR, comp->name, block_size); + printf("\t%s data, %s metadata, %s fragments, %s xattrs\n", + noD ? "uncompressed" : "compressed", noI ? "uncompressed" : + "compressed", no_fragments ? "no" : noF ? "uncompressed" : + "compressed", no_xattrs ? "no" : noX ? "uncompressed" : + "compressed"); + printf("\tduplicates are %sremoved\n", duplicate_checking ? "" : + "not "); + printf("Filesystem size %.2f Kbytes (%.2f Mbytes)\n", bytes / 1024.0, + bytes / (1024.0 * 1024.0)); + printf("\t%.2f%% of uncompressed filesystem size (%.2f Kbytes)\n", + ((float) bytes / total_bytes) * 100.0, total_bytes / 1024.0); + printf("Inode table size %d bytes (%.2f Kbytes)\n", + inode_bytes, inode_bytes / 1024.0); + printf("\t%.2f%% of uncompressed inode table size (%d bytes)\n", + ((float) inode_bytes / total_inode_bytes) * 100.0, + total_inode_bytes); + printf("Directory table size %d bytes (%.2f Kbytes)\n", + directory_bytes, directory_bytes / 1024.0); + printf("\t%.2f%% of uncompressed directory table size (%d bytes)\n", + ((float) directory_bytes / total_directory_bytes) * 100.0, + total_directory_bytes); + if(total_xattr_bytes) { + printf("Xattr table size %d bytes (%.2f Kbytes)\n", + xattr_bytes, xattr_bytes / 1024.0); + printf("\t%.2f%% of uncompressed xattr table size (%d bytes)\n", + ((float) xattr_bytes / total_xattr_bytes) * 100.0, + total_xattr_bytes); + } + if(duplicate_checking) + printf("Number of duplicate files found %d\n", file_count - + dup_files); + else + printf("No duplicate files removed\n"); + printf("Number of inodes %d\n", inode_count); + printf("Number of files %d\n", file_count); + if(!no_fragments) + printf("Number of fragments %d\n", fragments); + printf("Number of symbolic links %d\n", sym_count); + printf("Number of device nodes %d\n", dev_count); + printf("Number of fifo nodes %d\n", fifo_count); + printf("Number of socket nodes %d\n", sock_count); + printf("Number of directories %d\n", dir_count); + printf("Number of ids (unique uids + gids) %d\n", id_count); + printf("Number of uids %d\n", uid_count); + + for(i = 0; i < id_count; i++) { + if(id_table[i]->flags & ISA_UID) { + struct passwd *user = getpwuid(id_table[i]->id); + printf("\t%s (%d)\n", user == NULL ? "unknown" : + user->pw_name, id_table[i]->id); + } + } + + printf("Number of gids %d\n", guid_count); + + for(i = 0; i < id_count; i++) { + if(id_table[i]->flags & ISA_GID) { + struct group *group = getgrgid(id_table[i]->id); + printf("\t%s (%d)\n", group == NULL ? "unknown" : + group->gr_name, id_table[i]->id); + } + } + + return 0; +} diff --git a/squashfs-tools/mksquashfs.h b/squashfs-tools/mksquashfs.h new file mode 100644 index 0000000..08d060f --- /dev/null +++ b/squashfs-tools/mksquashfs.h @@ -0,0 +1,86 @@ +#ifndef MKSQUASHFS_H +#define MKSQUASHFS_H +/* + * Squashfs + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * mksquashfs.h + * + */ + +#if __BYTE_ORDER == __BIG_ENDIAN +#define SQUASHFS_SWAP_SHORTS(s, d, n) swap_le16_num(s, d, n) +#define SQUASHFS_SWAP_INTS(s, d, n) swap_le32_num(s, d, n) +#define SQUASHFS_SWAP_LONG_LONGS(s, d, n) swap_le64_num(s, d, n) + +#define SWAP_LE16(s, d) swap_le16(s, d) +#define SWAP_LE32(s, d) swap_le32(s, d) +#define SWAP_LE64(s, d) swap_le64(s, d) +#else +#define SQUASHFS_MEMCPY(s, d, n) memcpy(d, s, n) +#define SQUASHFS_SWAP_SHORTS(s, d, n) memcpy(d, s, n * sizeof(short)) +#define SQUASHFS_SWAP_INTS(s, d, n) memcpy(d, s, n * sizeof(int)) +#define SQUASHFS_SWAP_LONG_LONGS(s, d, n) \ + memcpy(d, s, n * sizeof(long long)) +#endif + +struct dir_info { + char *pathname; + unsigned int count; + unsigned int directory_count; + unsigned int current_count; + unsigned int byte_count; + char dir_is_ldir; + struct dir_ent *dir_ent; + struct dir_ent **list; + DIR *linuxdir; +}; + +struct dir_ent { + char *name; + char *pathname; + struct inode_info *inode; + struct dir_info *dir; + struct dir_info *our_dir; +}; + +struct inode_info { + struct stat buf; + struct inode_info *next; + squashfs_inode inode; + unsigned int inode_number; + unsigned int nlink; + int pseudo_id; + char type; + char read; + char root_entry; + char pseudo_file; +}; +#endif + +#define PSEUDO_FILE_OTHER 1 +#define PSEUDO_FILE_PROCESS 2 + +#define IS_PSEUDO(a) ((a)->pseudo_file) +#define IS_PSEUDO_PROCESS(a) ((a)->pseudo_file & PSEUDO_FILE_PROCESS) +#define IS_PSEUDO_OTHER(a) ((a)->pseudo_file & PSEUDO_FILE_OTHER) + +/* offset of data in compressed metadata blocks (allowing room for + * compressed size */ +#define BLOCK_OFFSET 2 diff --git a/squashfs-tools/pseudo.c b/squashfs-tools/pseudo.c new file mode 100644 index 0000000..6ddc839 --- /dev/null +++ b/squashfs-tools/pseudo.c @@ -0,0 +1,524 @@ +/* + * Create a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * pseudo.c + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "pseudo.h" + +#ifdef SQUASHFS_TRACE +#define TRACE(s, args...) \ + do { \ + printf("mksquashfs: "s, ## args); \ + } while(0) +#else +#define TRACE(s, args...) +#endif + +#define ERROR(s, args...) \ + do { \ + fprintf(stderr, s, ## args); \ + } while(0) + +#define EXIT_MKSQUASHFS() \ + do { \ + exit(1); \ + } while(0) + +#define BAD_ERROR(s, args...) \ + do {\ + fprintf(stderr, "FATAL ERROR:" s, ##args);\ + EXIT_MKSQUASHFS();\ + } while(0); + +#define TRUE 1 +#define FALSE 0 + +struct pseudo_dev **pseudo_file = NULL; +int pseudo_count = 0; + +static void dump_pseudo(struct pseudo *pseudo, char *string) +{ + int i; + char path[1024]; + + for(i = 0; i < pseudo->names; i++) { + struct pseudo_entry *entry = &pseudo->name[i]; + if(string) + strcat(strcat(strcpy(path, string), "/"), entry->name); + else + strcpy(path, entry->name); + if(entry->pseudo == NULL) + ERROR("%s %c %o %d %d %d %d\n", path, entry->dev->type, + entry->dev->mode, entry->dev->uid, + entry->dev->gid, entry->dev->major, + entry->dev->minor); + else + dump_pseudo(entry->pseudo, path); + } +} + + +static char *get_component(char *target, char *targname) +{ + while(*target == '/') + target ++; + + while(*target != '/' && *target!= '\0') + *targname ++ = *target ++; + + *targname = '\0'; + + return target; +} + + +/* + * Add pseudo device target to the set of pseudo devices. Pseudo_dev + * describes the pseudo device attributes. + */ +struct pseudo *add_pseudo(struct pseudo *pseudo, struct pseudo_dev *pseudo_dev, + char *target, char *alltarget) +{ + char targname[1024]; + int i; + + target = get_component(target, targname); + + if(pseudo == NULL) { + if((pseudo = malloc(sizeof(struct pseudo))) == NULL) + BAD_ERROR("failed to allocate pseudo file\n"); + + pseudo->names = 0; + pseudo->count = 0; + pseudo->name = NULL; + } + + for(i = 0; i < pseudo->names; i++) + if(strcmp(pseudo->name[i].name, targname) == 0) + break; + + if(i == pseudo->names) { + /* allocate new name entry */ + pseudo->names ++; + pseudo->name = realloc(pseudo->name, (i + 1) * + sizeof(struct pseudo_entry)); + if(pseudo->name == NULL) + BAD_ERROR("failed to allocate pseudo file\n"); + pseudo->name[i].name = strdup(targname); + + if(target[0] == '\0') { + /* at leaf pathname component */ + pseudo->name[i].pseudo = NULL; + pseudo->name[i].pathname = strdup(alltarget); + pseudo->name[i].dev = pseudo_dev; + } else { + /* recurse adding child components */ + pseudo->name[i].dev = NULL; + pseudo->name[i].pseudo = add_pseudo(NULL, pseudo_dev, + target, alltarget); + } + } else { + /* existing matching entry */ + if(pseudo->name[i].pseudo == NULL) { + /* No sub-directory which means this is the leaf + * component of a pre-existing pseudo file. + */ + if(target[0] != '\0') { + /* entry must exist as a 'd' type pseudo file */ + if(pseudo->name[i].dev->type == 'd') + /* recurse adding child components */ + pseudo->name[i].pseudo = + add_pseudo(NULL, pseudo_dev, + target, alltarget); + else + ERROR("%s already exists as a non " + "directory. Ignoring %s!\n", + targname, alltarget); + } else if(memcmp(pseudo_dev, pseudo->name[i].dev, + sizeof(struct pseudo_dev)) != 0) + ERROR("%s already exists as a different pseudo " + "definition. Ignoring!\n", alltarget); + else ERROR("%s already exists as an identical " + "pseudo definition!\n", alltarget); + } else { + /* sub-directory exists which means this can only be a + * 'd' type pseudo file */ + if(target[0] == '\0') { + if(pseudo->name[i].dev == NULL && + pseudo_dev->type == 'd') { + pseudo->name[i].pathname = + strdup(alltarget); + pseudo->name[i].dev = pseudo_dev; + } else + ERROR("%s already exists as a " + "directory. Ignoring %s!\n", + targname, alltarget); + } else + /* recurse adding child components */ + add_pseudo(pseudo->name[i].pseudo, pseudo_dev, + target, alltarget); + } + } + + return pseudo; +} + + +/* + * Find subdirectory in pseudo directory referenced by pseudo, matching + * filename. If filename doesn't exist or if filename is a leaf file + * return NULL + */ +struct pseudo *pseudo_subdir(char *filename, struct pseudo *pseudo) +{ + int i; + + if(pseudo == NULL) + return NULL; + + for(i = 0; i < pseudo->names; i++) + if(strcmp(filename, pseudo->name[i].name) == 0) + return pseudo->name[i].pseudo; + + return NULL; +} + + +struct pseudo_entry *pseudo_readdir(struct pseudo *pseudo) +{ + if(pseudo == NULL) + return NULL; + + while(pseudo->count < pseudo->names) { + if(pseudo->name[pseudo->count].dev != NULL) + return &pseudo->name[pseudo->count++]; + else + pseudo->count++; + } + + return NULL; +} + + +int exec_file(char *command, struct pseudo_dev *dev) +{ + int child, res; + static pid_t pid = -1; + int pipefd[2]; +#ifdef USE_TMP_FILE + char filename[1024]; + int status; + static int number = 0; +#endif + + if(pid == -1) + pid = getpid(); + +#ifdef USE_TMP_FILE + sprintf(filename, "/tmp/squashfs_pseudo_%d_%d", pid, number ++); + pipefd[1] = open(filename, O_CREAT | O_TRUNC | O_RDWR, S_IRWXU); + if(pipefd[1] == -1) { + printf("open failed\n"); + return -1; + } +#else + res = pipe(pipefd); + if(res == -1) { + printf("pipe failed\n"); + return -1; + } +#endif + + child = fork(); + if(child == -1) { + printf("fork failed\n"); + goto failed; + } + + if(child == 0) { + close(STDOUT_FILENO); + res = dup(pipefd[1]); + if(res == -1) { + printf("dup failed\n"); + exit(EXIT_FAILURE); + } + execl("/bin/sh", "sh", "-c", command, (char *) NULL); + printf("execl failed\n"); + exit(EXIT_FAILURE); + } + +#ifdef USE_TMP_FILE + res = waitpid(child, &status, 0); + close(pipefd[1]); + if(res != -1 && WIFEXITED(status) && WEXITSTATUS(status) == 0) { + dev->filename = strdup(filename); + return 0; + } +failed: + unlink(filename); + return -1; +#else + close(pipefd[1]); + dev->fd = pipefd[0]; + dev->child = child; + return 0; +failed: + return -1; +#endif +} + + +void add_pseudo_file(struct pseudo_dev *dev) +{ + pseudo_file = realloc(pseudo_file, (pseudo_count + 1) * + sizeof(struct pseudo_dev *)); + if(pseudo_file == NULL) + BAD_ERROR("Failed to realloc pseudo_file\n"); + + dev->pseudo_id = pseudo_count; + pseudo_file[pseudo_count ++] = dev; +} + + +void delete_pseudo_files() +{ +#ifdef USE_TMP_FILE + int i; + + for(i = 0; i < pseudo_count; i++) + unlink(pseudo_file[i]->filename); +#endif +} + + +struct pseudo_dev *get_pseudo_file(int pseudo_id) +{ + return pseudo_file[pseudo_id]; +} + + +int read_pseudo_def(struct pseudo **pseudo, char *def) +{ + int n, bytes; + unsigned int major = 0, minor = 0, mode; + char filename[2048], type, suid[100], sgid[100], *ptr; + long long uid, gid; + struct pseudo_dev *dev; + + n = sscanf(def, "%s %c %o %s %s %n", filename, &type, &mode, suid, + sgid, &bytes); + + if(n < 5) { + ERROR("Not enough or invalid arguments in pseudo file " + "definition\n"); + goto error; + } + + switch(type) { + case 'b': + case 'c': + n = sscanf(def + bytes, "%u %u", &major, &minor); + + if(n < 2) { + ERROR("Not enough or invalid arguments in pseudo file " + "definition\n"); + goto error; + } + + if(major > 0xfff) { + ERROR("Major %d out of range\n", major); + goto error; + } + + if(minor > 0xfffff) { + ERROR("Minor %d out of range\n", minor); + goto error; + } + + case 'f': + if(def[bytes] == '\0') { + ERROR("Not enough arguments in pseudo file " + "definition\n"); + goto error; + } + break; + case 'd': + case 'm': + break; + default: + ERROR("Unsupported type %c\n", type); + goto error; + } + + + if(mode > 07777) { + ERROR("Mode %o out of range\n", mode); + goto error; + } + + uid = strtoll(suid, &ptr, 10); + if(*ptr == '\0') { + if(uid < 0 || uid > ((1LL << 32) - 1)) { + ERROR("Uid %s out of range\n", suid); + goto error; + } + } else { + struct passwd *pwuid = getpwnam(suid); + if(pwuid) + uid = pwuid->pw_uid; + else { + ERROR("Uid %s invalid uid or unknown user\n", suid); + goto error; + } + } + + gid = strtoll(sgid, &ptr, 10); + if(*ptr == '\0') { + if(gid < 0 || gid > ((1LL << 32) - 1)) { + ERROR("Gid %s out of range\n", sgid); + goto error; + } + } else { + struct group *grgid = getgrnam(sgid); + if(grgid) + gid = grgid->gr_gid; + else { + ERROR("Gid %s invalid uid or unknown user\n", sgid); + goto error; + } + } + + switch(type) { + case 'b': + mode |= S_IFBLK; + break; + case 'c': + mode |= S_IFCHR; + break; + case 'd': + mode |= S_IFDIR; + break; + case 'f': + mode |= S_IFREG; + break; + } + + dev = malloc(sizeof(struct pseudo_dev)); + if(dev == NULL) + BAD_ERROR("Failed to create pseudo_dev\n"); + + dev->type = type; + dev->mode = mode; + dev->uid = uid; + dev->gid = gid; + dev->major = major; + dev->minor = minor; + + if(type == 'f') { + int res; + + printf("Executing dynamic pseudo file\n"); + printf("\t\"%s\"\n", def); + res = exec_file(def + bytes, dev); + if(res == -1) { + ERROR("Failed to execute dynamic pseudo file definition" + " \"%s\"\n", def); + return FALSE; + } + add_pseudo_file(dev); + } + + *pseudo = add_pseudo(*pseudo, dev, filename, filename); + + return TRUE; + +error: + ERROR("Bad pseudo file definition \"%s\"\n", def); + return FALSE; +} + + + +#define MAX_LINE 2048 + +int read_pseudo_file(struct pseudo **pseudo, char *filename) +{ + FILE *fd; + char *line = NULL; + int size = 0; + int res = TRUE; + + fd = fopen(filename, "r"); + if(fd == NULL) { + ERROR("Could not open pseudo device file \"%s\" because %s\n", + filename, strerror(errno)); + return FALSE; + } + + while(1) { + int total = 0; + + while(1) { + int n, err; + + if(total + MAX_LINE > size) { + line = realloc(line, size += MAX_LINE); + if(line == NULL) { + ERROR("No space in read_pseudo_file\n"); + return FALSE; + } + } + + err = fscanf(fd, "%2047[^\n]%n\n", line + total, &n); + if(err <= 0) + goto done; + + if(line[total] == '#') + continue; + + if(line[total + n - 1] != '\\') + break; + + total += n - 1; + } + + res = read_pseudo_def(pseudo, line); + if(res == FALSE) + break; + } + +done: + fclose(fd); + free(line); + return res; +} diff --git a/squashfs-tools/pseudo.h b/squashfs-tools/pseudo.h new file mode 100644 index 0000000..37cfd26 --- /dev/null +++ b/squashfs-tools/pseudo.h @@ -0,0 +1,57 @@ +/* + * Create a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2009 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * pseudo.h + */ +struct pseudo_dev { + char type; + unsigned int mode; + unsigned int uid; + unsigned int gid; + unsigned int major; + unsigned int minor; + int pseudo_id; + int fd; + int child; +#ifdef USE_TMP_FILE + char *filename; +#endif +}; + +struct pseudo_entry { + char *name; + char *pathname; + struct pseudo *pseudo; + struct pseudo_dev *dev; +}; + +struct pseudo { + int names; + int count; + struct pseudo_entry *name; +}; + +extern int read_pseudo_def(struct pseudo **, char *); +extern int read_pseudo_file(struct pseudo **, char *); +extern struct pseudo *pseudo_subdir(char *, struct pseudo *); +extern struct pseudo_entry *pseudo_readdir(struct pseudo *); +extern struct pseudo_dev *get_pseudo_file(int); +extern void delete_pseudo_files(); diff --git a/squashfs-tools/read_fs.c b/squashfs-tools/read_fs.c new file mode 100644 index 0000000..2bb8685 --- /dev/null +++ b/squashfs-tools/read_fs.c @@ -0,0 +1,829 @@ +/* + * Read a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * read_fs.c + */ + +#define TRUE 1 +#define FALSE 0 +#include +#include +#include +#include +#include +#include +#include + +#ifndef linux +#define __BYTE_ORDER BYTE_ORDER +#define __BIG_ENDIAN BIG_ENDIAN +#define __LITTLE_ENDIAN LITTLE_ENDIAN +#else +#include +#endif + +#include + +#ifdef SQUASHFS_TRACE +#define TRACE(s, args...) \ + do { \ + printf("mksquashfs: "s, ## args); \ + } while(0) +#else +#define TRACE(s, args...) +#endif + +#define ERROR(s, args...) \ + do { \ + fprintf(stderr, s, ## args); \ + } while(0) + +#include "squashfs_fs.h" +#include "squashfs_swap.h" +#include "read_fs.h" +#include "compressor.h" +#include "xattr.h" + +extern int read_fs_bytes(int, long long, int, void *); +extern int add_file(long long, long long, long long, unsigned int *, int, + unsigned int, int, int); +extern void *create_id(unsigned int); +extern unsigned int get_uid(unsigned int); +extern unsigned int get_guid(unsigned int); + +static struct compressor *comp; + +int read_block(int fd, long long start, long long *next, void *block) +{ + unsigned short c_byte; + int res; + + res = read_fs_bytes(fd, start, 2, &c_byte); + if(res == 0) + return 0; + + SQUASHFS_INSWAP_SHORTS(&c_byte, 1); + + if(SQUASHFS_COMPRESSED(c_byte)) { + char buffer[SQUASHFS_METADATA_SIZE]; + int error, res; + + c_byte = SQUASHFS_COMPRESSED_SIZE(c_byte); + res = read_fs_bytes(fd, start + 2, c_byte, buffer); + if(res == 0) + return 0; + + res = compressor_uncompress(comp, block, buffer, c_byte, + SQUASHFS_METADATA_SIZE, &error); + if(res == -1) { + ERROR("%s uncompress failed with error code %d\n", + comp->name, error); + return 0; + } + if(next) + *next = start + 2 + c_byte; + return res; + } else { + c_byte = SQUASHFS_COMPRESSED_SIZE(c_byte); + res = read_fs_bytes(fd, start + 2, c_byte, block); + if(res == 0) + return 0; + + if(next) + *next = start + 2 + c_byte; + return c_byte; + } +} + + +int scan_inode_table(int fd, long long start, long long end, + long long root_inode_start, int root_inode_offset, + struct squashfs_super_block *sBlk, union squashfs_inode_header *dir_inode, + unsigned char **inode_table, unsigned int *root_inode_block, + unsigned int *root_inode_size, long long *uncompressed_file, + unsigned int *uncompressed_directory, int *file_count, int *sym_count, + int *dev_count, int *dir_count, int *fifo_count, int *sock_count, + unsigned int *id_table) +{ + unsigned char *cur_ptr; + int byte, bytes = 0, size = 0, files = 0; + struct squashfs_reg_inode_header inode; + unsigned int directory_start_block; + + TRACE("scan_inode_table: start 0x%llx, end 0x%llx, root_inode_start " + "0x%llx\n", start, end, root_inode_start); + + while(start < end) { + if(start == root_inode_start) { + TRACE("scan_inode_table: read compressed block 0x%llx " + "containing root inode\n", start); + *root_inode_block = bytes; + } + if(size - bytes < SQUASHFS_METADATA_SIZE) { + *inode_table = realloc(*inode_table, size + += SQUASHFS_METADATA_SIZE); + if(*inode_table == NULL) + return FALSE; + } + TRACE("scan_inode_table: reading block 0x%llx\n", start); + byte = read_block(fd, start, &start, *inode_table + bytes); + if(byte == 0) { + free(*inode_table); + return FALSE; + } + bytes += byte; + } + + /* + * Read last inode entry which is the root directory inode, and obtain + * the last directory start block index. This is used when calculating + * the total uncompressed directory size. The directory bytes in the + * last * block will be counted as normal. + * + * The root inode is ignored in the inode scan. This ensures there is + * always enough bytes left to read a regular file inode entry + */ + *root_inode_size = bytes - (*root_inode_block + root_inode_offset); + bytes = *root_inode_block + root_inode_offset; + SQUASHFS_SWAP_BASE_INODE_HEADER(&dir_inode->base, *inode_table + bytes); + if(dir_inode->base.inode_type == SQUASHFS_DIR_TYPE) { + SQUASHFS_SWAP_DIR_INODE_HEADER(&dir_inode->dir, + *inode_table + bytes); + directory_start_block = dir_inode->dir.start_block; + } else { + SQUASHFS_SWAP_LDIR_INODE_HEADER(&dir_inode->ldir, + *inode_table + bytes); + directory_start_block = dir_inode->ldir.start_block; + } + get_uid(id_table[dir_inode->base.uid]); + get_guid(id_table[dir_inode->base.guid]); + + for(cur_ptr = *inode_table; cur_ptr < *inode_table + bytes; files ++) { + SQUASHFS_SWAP_REG_INODE_HEADER(&inode, cur_ptr); + + TRACE("scan_inode_table: processing inode @ byte position " + "0x%x, type 0x%x\n", + (unsigned int) (cur_ptr - *inode_table), + inode.inode_type); + + get_uid(id_table[inode.uid]); + get_guid(id_table[inode.guid]); + + switch(inode.inode_type) { + case SQUASHFS_FILE_TYPE: { + int frag_bytes = inode.fragment == + SQUASHFS_INVALID_FRAG ? 0 : + inode.file_size % sBlk->block_size; + int blocks = inode.fragment == + SQUASHFS_INVALID_FRAG ? (inode.file_size + + sBlk->block_size - 1) >> + sBlk->block_log : inode.file_size >> + sBlk->block_log; + long long file_bytes = 0; + int i; + long long start = inode.start_block; + unsigned int *block_list; + + TRACE("scan_inode_table: regular file, " + "file_size %d, blocks %d\n", + inode.file_size, blocks); + + block_list = malloc(blocks * + sizeof(unsigned int)); + if(block_list == NULL) { + ERROR("Out of memory in block list " + "malloc\n"); + goto failed; + } + + cur_ptr += sizeof(inode); + SQUASHFS_SWAP_INTS(block_list, cur_ptr, blocks); + + *uncompressed_file += inode.file_size; + (*file_count) ++; + + for(i = 0; i < blocks; i++) + file_bytes += + SQUASHFS_COMPRESSED_SIZE_BLOCK + (block_list[i]); + + add_file(start, inode.file_size, file_bytes, + block_list, blocks, inode.fragment, + inode.offset, frag_bytes); + cur_ptr += blocks * sizeof(unsigned int); + break; + } + case SQUASHFS_LREG_TYPE: { + struct squashfs_lreg_inode_header inode; + int frag_bytes; + int blocks; + long long file_bytes = 0; + int i; + long long start; + unsigned int *block_list; + + SQUASHFS_SWAP_LREG_INODE_HEADER(&inode, cur_ptr); + + frag_bytes = inode.fragment == + SQUASHFS_INVALID_FRAG ? 0 : + inode.file_size % sBlk->block_size; + blocks = inode.fragment == SQUASHFS_INVALID_FRAG + ? (inode.file_size + + sBlk->block_size - 1) >> + sBlk->block_log : inode.file_size >> + sBlk->block_log; + start = inode.start_block; + + TRACE("scan_inode_table: extended regular " + "file, file_size %lld, blocks %d\n", + inode.file_size, blocks); + + block_list = malloc(blocks * + sizeof(unsigned int)); + if(block_list == NULL) { + ERROR("Out of memory in block list " + "malloc\n"); + goto failed; + } + + cur_ptr += sizeof(inode); + SQUASHFS_SWAP_INTS(block_list, cur_ptr, blocks); + + *uncompressed_file += inode.file_size; + (*file_count) ++; + + for(i = 0; i < blocks; i++) + file_bytes += + SQUASHFS_COMPRESSED_SIZE_BLOCK + (block_list[i]); + + add_file(start, inode.file_size, file_bytes, + block_list, blocks, inode.fragment, + inode.offset, frag_bytes); + cur_ptr += blocks * sizeof(unsigned int); + break; + } + case SQUASHFS_SYMLINK_TYPE: + case SQUASHFS_LSYMLINK_TYPE: { + struct squashfs_symlink_inode_header inodep; + + SQUASHFS_SWAP_SYMLINK_INODE_HEADER(&inodep, + cur_ptr); + (*sym_count) ++; + cur_ptr += sizeof(inodep) + inodep.symlink_size; + + if (inode.inode_type == SQUASHFS_LSYMLINK_TYPE) + cur_ptr += sizeof(unsigned int); + + break; + } + case SQUASHFS_DIR_TYPE: { + struct squashfs_dir_inode_header dir_inode; + + SQUASHFS_SWAP_DIR_INODE_HEADER(&dir_inode, + cur_ptr); + if(dir_inode.start_block < directory_start_block) + *uncompressed_directory += + dir_inode.file_size; + (*dir_count) ++; + cur_ptr += sizeof(struct squashfs_dir_inode_header); + break; + } + case SQUASHFS_LDIR_TYPE: { + struct squashfs_ldir_inode_header dir_inode; + int i; + + SQUASHFS_SWAP_LDIR_INODE_HEADER(&dir_inode, + cur_ptr); + if(dir_inode.start_block < directory_start_block) + *uncompressed_directory += + dir_inode.file_size; + (*dir_count) ++; + cur_ptr += sizeof(struct squashfs_ldir_inode_header); + for(i = 0; i < dir_inode.i_count; i++) { + struct squashfs_dir_index index; + + SQUASHFS_SWAP_DIR_INDEX(&index, + cur_ptr); + cur_ptr += sizeof(struct squashfs_dir_index) + + index.size + 1; + } + break; + } + case SQUASHFS_BLKDEV_TYPE: + case SQUASHFS_CHRDEV_TYPE: + (*dev_count) ++; + cur_ptr += sizeof(struct squashfs_dev_inode_header); + break; + case SQUASHFS_LBLKDEV_TYPE: + case SQUASHFS_LCHRDEV_TYPE: + (*dev_count) ++; + cur_ptr += sizeof(struct squashfs_ldev_inode_header); + break; + case SQUASHFS_FIFO_TYPE: + (*fifo_count) ++; + cur_ptr += sizeof(struct squashfs_ipc_inode_header); + break; + case SQUASHFS_LFIFO_TYPE: + (*fifo_count) ++; + cur_ptr += sizeof(struct squashfs_lipc_inode_header); + break; + case SQUASHFS_SOCKET_TYPE: + (*sock_count) ++; + cur_ptr += sizeof(struct squashfs_ipc_inode_header); + break; + case SQUASHFS_LSOCKET_TYPE: + (*sock_count) ++; + cur_ptr += sizeof(struct squashfs_lipc_inode_header); + break; + default: + ERROR("Unknown inode type %d in " + "scan_inode_table!\n", + inode.inode_type); + goto failed; + } + } + + printf("Read existing filesystem, %d inodes scanned\n", files); + return TRUE; + + +failed: + free(*inode_table); + return FALSE; +} + + +struct compressor *read_super(int fd, struct squashfs_super_block *sBlk, char *source) +{ + int res, bytes = 0; + char buffer[SQUASHFS_METADATA_SIZE] __attribute__ ((aligned)); + + res = read_fs_bytes(fd, SQUASHFS_START, sizeof(struct squashfs_super_block), + sBlk); + if(res == 0) + goto failed_mount; + + SQUASHFS_INSWAP_SUPER_BLOCK(sBlk); + + if(sBlk->s_magic != SQUASHFS_MAGIC) { + if(sBlk->s_magic == SQUASHFS_MAGIC_SWAP) + ERROR("Pre 4.0 big-endian filesystem on %s, appending" + " to this is unsupported\n", source); + else + ERROR("Can't find a SQUASHFS superblock on %s\n", + source); + goto failed_mount; + } + + /* Check the MAJOR & MINOR versions */ + if(sBlk->s_major != SQUASHFS_MAJOR || sBlk->s_minor > SQUASHFS_MINOR) { + if(sBlk->s_major < 4) + ERROR("Filesystem on %s is a SQUASHFS %d.%d filesystem." + " Appending\nto SQUASHFS %d.%d filesystems is " + "not supported. Please convert it to a " + "SQUASHFS 4 filesystem\n", source, + sBlk->s_major, + sBlk->s_minor, sBlk->s_major, sBlk->s_minor); + else + ERROR("Filesystem on %s is %d.%d, which is a later " + "filesystem version than I support\n", + source, sBlk->s_major, sBlk->s_minor); + goto failed_mount; + } + + /* Check the compression type */ + comp = lookup_compressor_id(sBlk->compression); + if(!comp->supported) { + ERROR("Filesystem on %s uses %s compression, this is" + "unsupported by this version\n", source, comp->name); + ERROR("Compressors available:\n"); + display_compressors("", ""); + goto failed_mount; + } + + /* + * Read extended superblock information from disk. + * + * Read compressor specific options from disk if present, and pass + * to compressor to set compressor options. + * + * Note, if there's no compressor options present, the compressor + * is still called to set the default options (the defaults may have + * been changed by the user specifying options on the command + * line which need to be over-ridden). + */ + if(SQUASHFS_COMP_OPTS(sBlk->flags)) { + bytes = read_block(fd, sizeof(*sBlk), NULL, buffer); + + if(bytes == 0) + goto failed_mount; + } + + res = compressor_extract_options(comp, sBlk->block_size, buffer, bytes); + if(res == -1) { + ERROR("Compressor failed to set compressor options\n"); + goto failed_mount; + } + + printf("Found a valid %sSQUASHFS superblock on %s.\n", + SQUASHFS_EXPORTABLE(sBlk->flags) ? "exportable " : "", source); + printf("\tCompression used %s\n", comp->name); + printf("\tInodes are %scompressed\n", + SQUASHFS_UNCOMPRESSED_INODES(sBlk->flags) ? "un" : ""); + printf("\tData is %scompressed\n", + SQUASHFS_UNCOMPRESSED_DATA(sBlk->flags) ? "un" : ""); + printf("\tFragments are %scompressed\n", + SQUASHFS_UNCOMPRESSED_FRAGMENTS(sBlk->flags) ? "un" : ""); + printf("\tXattrs are %scompressed\n", + SQUASHFS_UNCOMPRESSED_XATTRS(sBlk->flags) ? "un" : ""); + printf("\tFragments are %spresent in the filesystem\n", + SQUASHFS_NO_FRAGMENTS(sBlk->flags) ? "not " : ""); + printf("\tAlways_use_fragments option is %sspecified\n", + SQUASHFS_ALWAYS_FRAGMENTS(sBlk->flags) ? "" : "not "); + printf("\tDuplicates are %sremoved\n", + SQUASHFS_DUPLICATES(sBlk->flags) ? "" : "not "); + printf("\tXattrs are %sstored\n", + SQUASHFS_NO_XATTRS(sBlk->flags) ? "not " : ""); + printf("\tFilesystem size %.2f Kbytes (%.2f Mbytes)\n", + sBlk->bytes_used / 1024.0, sBlk->bytes_used + / (1024.0 * 1024.0)); + printf("\tBlock size %d\n", sBlk->block_size); + printf("\tNumber of fragments %d\n", sBlk->fragments); + printf("\tNumber of inodes %d\n", sBlk->inodes); + printf("\tNumber of ids %d\n", sBlk->no_ids); + TRACE("sBlk->inode_table_start %llx\n", sBlk->inode_table_start); + TRACE("sBlk->directory_table_start %llx\n", + sBlk->directory_table_start); + TRACE("sBlk->id_table_start %llx\n", sBlk->id_table_start); + TRACE("sBlk->fragment_table_start %llx\n", sBlk->fragment_table_start); + TRACE("sBlk->lookup_table_start %llx\n", sBlk->lookup_table_start); + TRACE("sBlk->xattr_id_table_start %llx\n", sBlk->xattr_id_table_start); + printf("\n"); + + return comp; + +failed_mount: + return NULL; +} + + +unsigned char *squashfs_readdir(int fd, int root_entries, + unsigned int directory_start_block, int offset, int size, + unsigned int *last_directory_block, struct squashfs_super_block *sBlk, + void (push_directory_entry)(char *, squashfs_inode, int, int)) +{ + struct squashfs_dir_header dirh; + char buffer[sizeof(struct squashfs_dir_entry) + SQUASHFS_NAME_LEN + 1] + __attribute__ ((aligned)); + struct squashfs_dir_entry *dire = (struct squashfs_dir_entry *) buffer; + unsigned char *directory_table = NULL; + int byte, bytes = 0, dir_count; + long long start = sBlk->directory_table_start + directory_start_block, + last_start_block = start; + + size += offset; + directory_table = malloc((size + SQUASHFS_METADATA_SIZE * 2 - 1) & + ~(SQUASHFS_METADATA_SIZE - 1)); + if(directory_table == NULL) + return NULL; + while(bytes < size) { + TRACE("squashfs_readdir: reading block 0x%llx, bytes read so " + "far %d\n", start, bytes); + last_start_block = start; + byte = read_block(fd, start, &start, directory_table + bytes); + if(byte == 0) { + free(directory_table); + return NULL; + } + bytes += byte; + } + + if(!root_entries) + goto all_done; + + bytes = offset; + while(bytes < size) { + SQUASHFS_SWAP_DIR_HEADER(&dirh, directory_table + bytes); + + dir_count = dirh.count + 1; + TRACE("squashfs_readdir: Read directory header @ byte position " + "0x%x, 0x%x directory entries\n", bytes, dir_count); + bytes += sizeof(dirh); + + while(dir_count--) { + SQUASHFS_SWAP_DIR_ENTRY(dire, directory_table + bytes); + bytes += sizeof(*dire); + + memcpy(dire->name, directory_table + bytes, + dire->size + 1); + dire->name[dire->size + 1] = '\0'; + TRACE("squashfs_readdir: pushing directory entry %s, " + "inode %x:%x, type 0x%x\n", dire->name, + dirh.start_block, dire->offset, dire->type); + push_directory_entry(dire->name, + SQUASHFS_MKINODE(dirh.start_block, + dire->offset), dirh.inode_number + + dire->inode_number, dire->type); + bytes += dire->size + 1; + } + } + +all_done: + *last_directory_block = (unsigned int) last_start_block - + sBlk->directory_table_start; + return directory_table; +} + + +unsigned int *read_id_table(int fd, struct squashfs_super_block *sBlk) +{ + int indexes = SQUASHFS_ID_BLOCKS(sBlk->no_ids); + long long index[indexes]; + int bytes = SQUASHFS_ID_BYTES(sBlk->no_ids); + unsigned int *id_table; + int res, i; + + id_table = malloc(bytes); + if(id_table == NULL) { + ERROR("Failed to allocate id table\n"); + return NULL; + } + + res = read_fs_bytes(fd, sBlk->id_table_start, + SQUASHFS_ID_BLOCK_BYTES(sBlk->no_ids), index); + if(res == 0) { + free(id_table); + return NULL; + } + + SQUASHFS_INSWAP_ID_BLOCKS(index, indexes); + + for(i = 0; i < indexes; i++) { + int length = read_block(fd, index[i], NULL, + ((unsigned char *) id_table) + + (i * SQUASHFS_METADATA_SIZE)); + TRACE("Read id table block %d, from 0x%llx, length %d\n", i, + index[i], length); + if(length == 0) { + ERROR("Failed to read id table block %d, from 0x%llx, " + "length %d\n", i, index[i], length); + free(id_table); + return NULL; + } + } + + SQUASHFS_INSWAP_INTS(id_table, sBlk->no_ids); + + for(i = 0; i < sBlk->no_ids; i++) { + TRACE("Adding id %d to id tables\n", id_table[i]); + create_id(id_table[i]); + } + + return id_table; +} + + +int read_fragment_table(int fd, struct squashfs_super_block *sBlk, + struct squashfs_fragment_entry **fragment_table) +{ + int res, i, indexes = SQUASHFS_FRAGMENT_INDEXES(sBlk->fragments); + long long fragment_table_index[indexes]; + + TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " + "from 0x%llx\n", sBlk->fragments, indexes, + sBlk->fragment_table_start); + if(sBlk->fragments == 0) + return 1; + + *fragment_table = malloc(sBlk->fragments * + sizeof(struct squashfs_fragment_entry)); + if(*fragment_table == NULL) { + ERROR("Failed to allocate fragment table\n"); + return 0; + } + + res = read_fs_bytes(fd, sBlk->fragment_table_start, + SQUASHFS_FRAGMENT_INDEX_BYTES(sBlk->fragments), + fragment_table_index); + if(res == 0) { + free(*fragment_table); + return 0; + } + + SQUASHFS_INSWAP_FRAGMENT_INDEXES(fragment_table_index, indexes); + + for(i = 0; i < indexes; i++) { + int length = read_block(fd, fragment_table_index[i], NULL, + ((unsigned char *) *fragment_table) + + (i * SQUASHFS_METADATA_SIZE)); + TRACE("Read fragment table block %d, from 0x%llx, length %d\n", + i, fragment_table_index[i], length); + if(length == 0) { + ERROR("Failed to read fragment table block %d, from " + "0x%llx, length %d\n", i, + fragment_table_index[i], length); + free(*fragment_table); + return 0; + } + } + + for(i = 0; i < sBlk->fragments; i++) + SQUASHFS_INSWAP_FRAGMENT_ENTRY(&(*fragment_table)[i]); + + return 1; +} + + +int read_inode_lookup_table(int fd, struct squashfs_super_block *sBlk, + squashfs_inode **inode_lookup_table) +{ + int lookup_bytes = SQUASHFS_LOOKUP_BYTES(sBlk->inodes); + int indexes = SQUASHFS_LOOKUP_BLOCKS(sBlk->inodes); + long long index[indexes]; + int res, i; + + if(sBlk->lookup_table_start == SQUASHFS_INVALID_BLK) + return 1; + + *inode_lookup_table = malloc(lookup_bytes); + if(*inode_lookup_table == NULL) { + ERROR("Failed to allocate inode lookup table\n"); + return 0; + } + + res = read_fs_bytes(fd, sBlk->lookup_table_start, + SQUASHFS_LOOKUP_BLOCK_BYTES(sBlk->inodes), index); + if(res == 0) { + free(*inode_lookup_table); + return 0; + } + + SQUASHFS_INSWAP_LONG_LONGS(index, indexes); + + for(i = 0; i < indexes; i++) { + int length = read_block(fd, index[i], NULL, + ((unsigned char *) *inode_lookup_table) + + (i * SQUASHFS_METADATA_SIZE)); + TRACE("Read inode lookup table block %d, from 0x%llx, length " + "%d\n", i, index[i], length); + if(length == 0) { + ERROR("Failed to read inode lookup table block %d, " + "from 0x%llx, length %d\n", i, index[i], + length); + free(*inode_lookup_table); + return 0; + } + } + + SQUASHFS_INSWAP_LONG_LONGS(*inode_lookup_table, sBlk->inodes); + + return 1; +} + + +long long read_filesystem(char *root_name, int fd, struct squashfs_super_block *sBlk, + char **cinode_table, char **data_cache, char **cdirectory_table, + char **directory_data_cache, unsigned int *last_directory_block, + unsigned int *inode_dir_offset, unsigned int *inode_dir_file_size, + unsigned int *root_inode_size, unsigned int *inode_dir_start_block, + int *file_count, int *sym_count, int *dev_count, int *dir_count, + int *fifo_count, int *sock_count, long long *uncompressed_file, + unsigned int *uncompressed_inode, unsigned int *uncompressed_directory, + unsigned int *inode_dir_inode_number, + unsigned int *inode_dir_parent_inode, + void (push_directory_entry)(char *, squashfs_inode, int, int), + struct squashfs_fragment_entry **fragment_table, + squashfs_inode **inode_lookup_table) +{ + unsigned char *inode_table = NULL, *directory_table; + long long start = sBlk->inode_table_start; + long long end = sBlk->directory_table_start; + long long root_inode_start = start + + SQUASHFS_INODE_BLK(sBlk->root_inode); + unsigned int root_inode_offset = + SQUASHFS_INODE_OFFSET(sBlk->root_inode); + unsigned int root_inode_block; + union squashfs_inode_header inode; + unsigned int *id_table; + int res; + + printf("Scanning existing filesystem...\n"); + + if(get_xattrs(fd, sBlk) == 0) + goto error; + + if(read_fragment_table(fd, sBlk, fragment_table) == 0) + goto error; + + if(read_inode_lookup_table(fd, sBlk, inode_lookup_table) == 0) + goto error; + + id_table = read_id_table(fd, sBlk); + if(id_table == NULL) + goto error; + + res = scan_inode_table(fd, start, end, root_inode_start, + root_inode_offset, sBlk, &inode, &inode_table, + &root_inode_block, root_inode_size, uncompressed_file, + uncompressed_directory, file_count, sym_count, dev_count, + dir_count, fifo_count, sock_count, id_table); + if(res == 0) { + ERROR("read_filesystem: inode table read failed\n"); + goto error; + } + + *uncompressed_inode = root_inode_block; + + if(inode.base.inode_type == SQUASHFS_DIR_TYPE || + inode.base.inode_type == SQUASHFS_LDIR_TYPE) { + if(inode.base.inode_type == SQUASHFS_DIR_TYPE) { + *inode_dir_start_block = inode.dir.start_block; + *inode_dir_offset = inode.dir.offset; + *inode_dir_file_size = inode.dir.file_size - 3; + *inode_dir_inode_number = inode.dir.inode_number; + *inode_dir_parent_inode = inode.dir.parent_inode; + } else { + *inode_dir_start_block = inode.ldir.start_block; + *inode_dir_offset = inode.ldir.offset; + *inode_dir_file_size = inode.ldir.file_size - 3; + *inode_dir_inode_number = inode.ldir.inode_number; + *inode_dir_parent_inode = inode.ldir.parent_inode; + } + + directory_table = squashfs_readdir(fd, !root_name, + *inode_dir_start_block, *inode_dir_offset, + *inode_dir_file_size, last_directory_block, sBlk, + push_directory_entry); + if(directory_table == NULL) { + ERROR("read_filesystem: Could not read root directory" + "\n"); + goto error; + } + + root_inode_start -= start; + *cinode_table = malloc(root_inode_start); + if(*cinode_table == NULL) { + ERROR("read_filesystem: failed to alloc space for " + "existing filesystem inode table\n"); + goto error; + } + res = read_fs_bytes(fd, start, root_inode_start, *cinode_table); + if(res == 0) + goto error; + + *cdirectory_table = malloc(*last_directory_block); + if(*cdirectory_table == NULL) { + ERROR("read_filesystem: failed to alloc space for " + "existing filesystem directory table\n"); + goto error; + } + res = read_fs_bytes(fd, sBlk->directory_table_start, + *last_directory_block, *cdirectory_table); + if(res == 0) + goto error; + + *data_cache = malloc(root_inode_offset + *root_inode_size); + if(*data_cache == NULL) { + ERROR("read_filesystem: failed to alloc inode cache\n"); + goto error; + } + memcpy(*data_cache, inode_table + root_inode_block, + root_inode_offset + *root_inode_size); + + *directory_data_cache = malloc(*inode_dir_offset + + *inode_dir_file_size); + if(*directory_data_cache == NULL) { + ERROR("read_filesystem: failed to alloc directory " + "cache\n"); + goto error; + } + memcpy(*directory_data_cache, directory_table, + *inode_dir_offset + *inode_dir_file_size); + + free(inode_table); + free(directory_table); + return sBlk->inode_table_start; + } + +error: + return 0; +} diff --git a/squashfs-tools/read_fs.h b/squashfs-tools/read_fs.h new file mode 100644 index 0000000..849f372 --- /dev/null +++ b/squashfs-tools/read_fs.h @@ -0,0 +1,42 @@ +#ifndef READ_FS_H +#define READ_FS_H +/* + * Squashfs + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * read_fs.h + * + */ + +#if __BYTE_ORDER == __BIG_ENDIAN +#define SQUASHFS_SWAP_SHORTS(d, s, n) swap_le16_num(s, d, n) +#define SQUASHFS_SWAP_INTS(d, s, n) swap_le32_num(s, d, n) +#define SQUASHFS_SWAP_LONG_LONGS(d, s, n) swap_le64_num(s, d, n) + +#define SWAP_LE16(d, s) swap_le16(s, d) +#define SWAP_LE32(d, s) swap_le32(s, d) +#define SWAP_LE64(d, s) swap_le64(s, d) +#else +#define SQUASHFS_MEMCPY(d, s, n) memcpy(d, s, n) +#define SQUASHFS_SWAP_SHORTS(d, s, n) memcpy(d, s, n * sizeof(short)) +#define SQUASHFS_SWAP_INTS(d, s, n) memcpy(d, s, n * sizeof(int)) +#define SQUASHFS_SWAP_LONG_LONGS(d, s, n) \ + memcpy(d, s, n * sizeof(long long)) +#endif +#endif diff --git a/squashfs-tools/read_xattrs.c b/squashfs-tools/read_xattrs.c new file mode 100644 index 0000000..6ef30f3 --- /dev/null +++ b/squashfs-tools/read_xattrs.c @@ -0,0 +1,364 @@ +/* + * Read a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * read_xattrs.c + */ + +/* + * Common xattr read code shared between mksquashfs and unsquashfs + */ + +#define TRUE 1 +#define FALSE 0 +#include +#include + +#ifndef linux +#define __BYTE_ORDER BYTE_ORDER +#define __BIG_ENDIAN BIG_ENDIAN +#define __LITTLE_ENDIAN LITTLE_ENDIAN +#else +#include +#endif + +#include "squashfs_fs.h" +#include "squashfs_swap.h" +#include "read_fs.h" +#include "xattr.h" + +#include + +#ifdef SQUASHFS_TRACE +#define TRACE(s, args...) \ + do { \ + printf("read_xattrs: "s, ## args); \ + } while(0) +#else +#define TRACE(s, args...) +#endif + +#define ERROR(s, args...) \ + do { \ + fprintf(stderr, s, ## args); \ + } while(0) + +extern int read_fs_bytes(int, long long, int, void *); +extern int read_block(int, long long, long long *, void *); + +static struct hash_entry { + long long start; + unsigned int offset; + struct hash_entry *next; +} *hash_table[65536]; + +static struct squashfs_xattr_id *xattr_ids; +static void *xattrs = NULL; +static long long xattr_table_start; + +/* + * Prefix lookup table, storing mapping to/from prefix string and prefix id + */ +struct prefix prefix_table[] = { + { "user.", SQUASHFS_XATTR_USER }, + { "trusted.", SQUASHFS_XATTR_TRUSTED }, + { "security.", SQUASHFS_XATTR_SECURITY }, + { "", -1 } +}; + +/* + * store mapping from location of compressed block in fs -> + * location of uncompressed block in memory + */ +static int save_xattr_block(long long start, int offset) +{ + struct hash_entry *hash_entry = malloc(sizeof(*hash_entry)); + int hash = start & 0xffff; + + TRACE("save_xattr_block: start %lld, offset %d\n", start, offset); + + if(hash_entry == NULL) { + ERROR("Failed to allocate hash entry\n"); + return -1; + } + + hash_entry->start = start; + hash_entry->offset = offset; + hash_entry->next = hash_table[hash]; + hash_table[hash] = hash_entry; + + return 1; +} + + +/* + * map from location of compressed block in fs -> + * location of uncompressed block in memory + */ +static int get_xattr_block(long long start) +{ + int hash = start & 0xffff; + struct hash_entry *hash_entry = hash_table[hash]; + + for(; hash_entry; hash_entry = hash_entry->next) + if(hash_entry->start == start) + break; + + TRACE("get_xattr_block: start %lld, offset %d\n", start, + hash_entry ? hash_entry->offset : -1); + + return hash_entry ? hash_entry->offset : -1; +} + + +/* + * construct the xattr_list entry from the fs xattr, including + * mapping name and prefix into a full name + */ +static int read_xattr_entry(struct xattr_list *xattr, + struct squashfs_xattr_entry *entry, void *name) +{ + int i, len, type = entry->type & XATTR_PREFIX_MASK; + + for(i = 0; prefix_table[i].type != -1; i++) + if(prefix_table[i].type == type) + break; + + if(prefix_table[i].type == -1) { + ERROR("Unrecognised type in read_xattr_entry\n"); + return 0; + } + + len = strlen(prefix_table[i].prefix); + xattr->full_name = malloc(len + entry->size + 1); + if(xattr->full_name == NULL) { + ERROR("Out of memory in read_xattr_entry\n"); + return -1; + } + memcpy(xattr->full_name, prefix_table[i].prefix, len); + memcpy(xattr->full_name + len, name, entry->size); + xattr->full_name[len + entry->size] = '\0'; + xattr->name = xattr->full_name + len; + xattr->size = entry->size; + xattr->type = type; + + return 1; +} + + +/* + * Read and decompress the xattr id table and the xattr metadata. + * This is cached in memory for later use by get_xattr() + */ +int read_xattrs_from_disk(int fd, struct squashfs_super_block *sBlk) +{ + int res, bytes, i, indexes, index_bytes, ids; + long long *index, start, end; + struct squashfs_xattr_table id_table; + + TRACE("read_xattrs_from_disk\n"); + + if(sBlk->xattr_id_table_start == SQUASHFS_INVALID_BLK) + return SQUASHFS_INVALID_BLK; + + /* + * Read xattr id table, containing start of xattr metadata and the + * number of xattrs in the file system + */ + res = read_fs_bytes(fd, sBlk->xattr_id_table_start, sizeof(id_table), + &id_table); + if(res == 0) + return 0; + + SQUASHFS_INSWAP_XATTR_TABLE(&id_table); + + /* + * Allocate and read the index to the xattr id table metadata + * blocks + */ + ids = id_table.xattr_ids; + xattr_table_start = id_table.xattr_table_start; + index_bytes = SQUASHFS_XATTR_BLOCK_BYTES(ids); + indexes = SQUASHFS_XATTR_BLOCKS(ids); + index = malloc(index_bytes); + if(index == NULL) { + ERROR("Failed to allocate index array\n"); + return 0; + } + + res = read_fs_bytes(fd, sBlk->xattr_id_table_start + sizeof(id_table), + index_bytes, index); + if(res ==0) + goto failed1; + + SQUASHFS_INSWAP_LONG_LONGS(index, indexes); + + /* + * Allocate enough space for the uncompressed xattr id table, and + * read and decompress it + */ + bytes = SQUASHFS_XATTR_BYTES(ids); + xattr_ids = malloc(bytes); + if(xattr_ids == NULL) { + ERROR("Failed to allocate xattr id table\n"); + goto failed1; + } + + for(i = 0; i < indexes; i++) { + int length = read_block(fd, index[i], NULL, + ((unsigned char *) xattr_ids) + + (i * SQUASHFS_METADATA_SIZE)); + TRACE("Read xattr id table block %d, from 0x%llx, length " + "%d\n", i, index[i], length); + if(length == 0) { + ERROR("Failed to read xattr id table block %d, " + "from 0x%llx, length %d\n", i, index[i], + length); + goto failed2; + } + } + + /* + * Read and decompress the xattr metadata + * + * Note the first xattr id table metadata block is immediately after + * the last xattr metadata block, so we can use index[0] to work out + * the end of the xattr metadata + */ + start = xattr_table_start; + end = index[0]; + for(i = 0; start < end; i++) { + int length; + void *x = realloc(xattrs, (i + 1) * SQUASHFS_METADATA_SIZE); + if(x == NULL) { + ERROR("Failed to realloc xattr data\n"); + goto failed3; + } + xattrs = x; + + /* store mapping from location of compressed block in fs -> + * location of uncompressed block in memory */ + res = save_xattr_block(start, i * SQUASHFS_METADATA_SIZE); + if(res == -1) + goto failed3; + + length = read_block(fd, start, &start, + ((unsigned char *) xattrs) + + (i * SQUASHFS_METADATA_SIZE)); + TRACE("Read xattr block %d, length %d\n", i, length); + if(length == 0) { + ERROR("Failed to read xattr block %d\n", i); + goto failed3; + } + } + + /* swap if necessary the xattr id entries */ + for(i = 0; i < ids; i++) + SQUASHFS_INSWAP_XATTR_ID(&xattr_ids[i]); + + free(index); + + return ids; + +failed3: + free(xattrs); +failed2: + free(xattr_ids); +failed1: + free(index); + + return 0; +} + + +/* + * Construct and return the list of xattr name:value pairs for the passed xattr + * id + */ +struct xattr_list *get_xattr(int i, unsigned int *count) +{ + long long start; + struct xattr_list *xattr_list = NULL; + unsigned int offset; + void *xptr; + int j; + + TRACE("get_xattr\n"); + + *count = xattr_ids[i].count; + start = SQUASHFS_XATTR_BLK(xattr_ids[i].xattr) + xattr_table_start; + offset = SQUASHFS_XATTR_OFFSET(xattr_ids[i].xattr); + xptr = xattrs + get_xattr_block(start) + offset; + + TRACE("get_xattr: xattr_id %d, count %d, start %lld, offset %d\n", i, + *count, start, offset); + + for(j = 0; j < *count; j++) { + struct squashfs_xattr_entry entry; + struct squashfs_xattr_val val; + int res; + + xattr_list = realloc(xattr_list, (j + 1) * + sizeof(struct xattr_list)); + if(xattr_list == NULL) { + ERROR("Out of memory in get_xattrs\n"); + goto failed; + } + + SQUASHFS_SWAP_XATTR_ENTRY(&entry, xptr); + xptr += sizeof(entry); + res = read_xattr_entry(&xattr_list[j], &entry, xptr); + if(res != 1) + goto failed; + xptr += entry.size; + + TRACE("get_xattr: xattr %d, type %d, size %d, name %s\n", j, + entry.type, entry.size, xattr_list[j].full_name); + + if(entry.type & SQUASHFS_XATTR_VALUE_OOL) { + long long xattr; + void *ool_xptr; + + xptr += sizeof(val); + SQUASHFS_SWAP_LONG_LONGS(&xattr, xptr, 1); + xptr += sizeof(xattr); + start = SQUASHFS_XATTR_BLK(xattr) + xattr_table_start; + offset = SQUASHFS_XATTR_OFFSET(xattr); + ool_xptr = xattrs + get_xattr_block(start) + offset; + SQUASHFS_SWAP_XATTR_VAL(&val, ool_xptr); + xattr_list[j].value = ool_xptr + sizeof(val); + } else { + SQUASHFS_SWAP_XATTR_VAL(&val, xptr); + xattr_list[j].value = xptr + sizeof(val); + xptr += sizeof(val) + val.vsize; + } + + TRACE("get_xattr: xattr %d, vsize %d\n", j, val.vsize); + + xattr_list[j].vsize = val.vsize; + } + + return xattr_list; + +failed: + free(xattr_list); + + return NULL; +} diff --git a/squashfs-tools/sort.c b/squashfs-tools/sort.c new file mode 100644 index 0000000..c02368c --- /dev/null +++ b/squashfs-tools/sort.c @@ -0,0 +1,287 @@ +/* + * Create a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * sort.c + */ + +#define TRUE 1 +#define FALSE 0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "squashfs_fs.h" +#include "mksquashfs.h" +#include "sort.h" + +#ifdef SQUASHFS_TRACE +#define TRACE(s, args...) \ + do { \ + printf("mksquashfs: "s, ## args); \ + } while(0) +#else +#define TRACE(s, args...) +#endif + +#define INFO(s, args...) \ + do { \ + if(!silent) printf("mksquashfs: "s, ## args); \ + } while(0) + +#define ERROR(s, args...) \ + do { \ + fprintf(stderr, s, ## args); \ + } while(0) + +int mkisofs_style = -1; + +struct sort_info { + dev_t st_dev; + ino_t st_ino; + int priority; + struct sort_info *next; +}; + +struct sort_info *sort_info_list[65536]; + +struct priority_entry *priority_list[65536]; + +extern int silent; +extern void write_file(squashfs_inode *inode, struct dir_ent *dir_ent, + int *c_size); + + +int add_priority_list(struct dir_ent *dir, int priority) +{ + struct priority_entry *new_priority_entry; + + priority += 32768; + if((new_priority_entry = malloc(sizeof(struct priority_entry))) == NULL) { + ERROR("Out of memory allocating priority entry\n"); + return FALSE; + } + + new_priority_entry->dir = dir;; + new_priority_entry->next = priority_list[priority]; + priority_list[priority] = new_priority_entry; + return TRUE; +} + + +int get_priority(char *filename, struct stat *buf, int priority) +{ + int hash = buf->st_ino & 0xffff; + struct sort_info *s; + + for(s = sort_info_list[hash]; s; s = s->next) + if((s->st_dev == buf->st_dev) && (s->st_ino == buf->st_ino)) { + TRACE("returning priority %d (%s)\n", s->priority, + filename); + return s->priority; + } + TRACE("returning priority %d (%s)\n", priority, filename); + return priority; +} + + +#define ADD_ENTRY(buf, priority) {\ + int hash = buf.st_ino & 0xffff;\ + struct sort_info *s;\ + if((s = malloc(sizeof(struct sort_info))) == NULL) {\ + ERROR("Out of memory allocating sort list entry\n");\ + return FALSE;\ + }\ + s->st_dev = buf.st_dev;\ + s->st_ino = buf.st_ino;\ + s->priority = priority;\ + s->next = sort_info_list[hash];\ + sort_info_list[hash] = s;\ + } +int add_sort_list(char *path, int priority, int source, char *source_path[]) +{ + int i, n; + char filename[4096]; + struct stat buf; + + TRACE("add_sort_list: filename %s, priority %d\n", path, priority); + if(strlen(path) > 1 && strcmp(path + strlen(path) - 2, "/*") == 0) + path[strlen(path) - 2] = '\0'; + + TRACE("add_sort_list: filename %s, priority %d\n", path, priority); +re_read: + if(path[0] == '/' || strncmp(path, "./", 2) == 0 || + strncmp(path, "../", 3) == 0 || mkisofs_style == 1) { + if(lstat(path, &buf) == -1) + goto error; + TRACE("adding filename %s, priority %d, st_dev %d, st_ino " + "%lld\n", path, priority, (int) buf.st_dev, + (long long) buf.st_ino); + ADD_ENTRY(buf, priority); + return TRUE; + } + + for(i = 0, n = 0; i < source; i++) { + strcat(strcat(strcpy(filename, source_path[i]), "/"), path); + if(lstat(filename, &buf) == -1) { + if(!(errno == ENOENT || errno == ENOTDIR)) + goto error; + continue; + } + ADD_ENTRY(buf, priority); + n ++; + } + + if(n == 0 && mkisofs_style == -1 && lstat(path, &buf) != -1) { + ERROR("WARNING: Mkisofs style sortlist detected! This is " + "supported but please\n"); + ERROR("convert to mksquashfs style sortlist! A sortlist entry"); + ERROR(" should be\neither absolute (starting with "); + ERROR("'/') start with './' or '../' (taken to be\nrelative to " + "$PWD), otherwise it "); + ERROR("is assumed the entry is relative to one\nof the source " + "directories, i.e. with "); + ERROR("\"mksquashfs test test.sqsh\",\nthe sortlist "); + ERROR("entry \"file\" is assumed to be inside the directory " + "test.\n\n"); + mkisofs_style = 1; + goto re_read; + } + + mkisofs_style = 0; + + if(n == 1) + return TRUE; + if(n > 1) { + ERROR(" Ambiguous sortlist entry \"%s\"\n\nIt maps to more " + "than one source entry! Please use an absolute path." + "\n", path); + return FALSE; + } + +error: + ERROR("Cannot stat sortlist entry \"%s\"\n", path); + ERROR("This is probably because you're using the wrong file\n"); + ERROR("path relative to the source directories\n"); + /* + * Historical note + * Failure to stat a sortlist entry is deliberately ignored, even + * though it is an error. Squashfs release 2.2 changed the behaviour + * to treat it as a fatal error, but it was changed back to + * the original behaviour to ignore it in release 2.2-r2 following + * feedback from users at the time. + */ + return TRUE; +} + + +int generate_file_priorities(struct dir_info *dir, int priority, + struct stat *buf) +{ + int res; + + priority = get_priority(dir->pathname, buf, priority); + + while(dir->current_count < dir->count) { + struct dir_ent *dir_ent = dir->list[dir->current_count++]; + struct stat *buf = &dir_ent->inode->buf; + if(dir_ent->inode->root_entry) + continue; + + switch(buf->st_mode & S_IFMT) { + case S_IFREG: + res = add_priority_list(dir_ent, + get_priority(dir_ent->pathname, buf, + priority)); + if(res == FALSE) + return FALSE; + break; + case S_IFDIR: + res = generate_file_priorities(dir_ent->dir, + priority, buf); + if(res == FALSE) + return FALSE; + break; + } + } + dir->current_count = 0; + + return TRUE; +} + + +int read_sort_file(char *filename, int source, char *source_path[]) +{ + FILE *fd; + char sort_filename[16385]; + int res, priority; + + if((fd = fopen(filename, "r")) == NULL) { + perror("Could not open sort_list file..."); + return FALSE; + } + while(fscanf(fd, "%s %d", sort_filename, &priority) != EOF) + if(priority >= -32768 && priority <= 32767) { + res = add_sort_list(sort_filename, priority, source, + source_path); + if(res == FALSE) + return FALSE; + } else + ERROR("Sort file %s, priority %d outside range of " + "-32767:32768 - skipping...\n", sort_filename, + priority); + fclose(fd); + return TRUE; +} + + +void sort_files_and_write(struct dir_info *dir) +{ + int i; + struct priority_entry *entry; + squashfs_inode inode; + int duplicate_file; + + for(i = 65535; i >= 0; i--) + for(entry = priority_list[i]; entry; entry = entry->next) { + TRACE("%d: %s\n", i - 32768, entry->dir->pathname); + if(entry->dir->inode->inode == SQUASHFS_INVALID_BLK) { + write_file(&inode, entry->dir, &duplicate_file); + INFO("file %s, uncompressed size %lld bytes %s" + "\n", entry->dir->pathname, + (long long) + entry->dir->inode->buf.st_size, + duplicate_file ? "DUPLICATE" : ""); + entry->dir->inode->inode = inode; + entry->dir->inode->type = SQUASHFS_FILE_TYPE; + } else + INFO("file %s, uncompressed size %lld bytes " + "LINK\n", entry->dir->pathname, + (long long) + entry->dir->inode->buf.st_size); + } +} diff --git a/squashfs-tools/sort.h b/squashfs-tools/sort.h new file mode 100644 index 0000000..6b7d9d8 --- /dev/null +++ b/squashfs-tools/sort.h @@ -0,0 +1,31 @@ +#ifndef SORT_H +#define SORT_H + +/* + * Squashfs + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * sort.h + */ + +struct priority_entry { + struct dir_ent *dir; + struct priority_entry *next; +}; +#endif diff --git a/squashfs-tools/squashfs_compat.h b/squashfs-tools/squashfs_compat.h new file mode 100644 index 0000000..4d7413e --- /dev/null +++ b/squashfs-tools/squashfs_compat.h @@ -0,0 +1,786 @@ +#ifndef SQUASHFS_COMPAT +#define SQUASHFS_COMPAT +/* + * Squashfs + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * squashfs_compat.h + */ + +/* + * definitions for structures on disk - layout 3.x + */ + +#define SQUASHFS_CHECK 2 +#define SQUASHFS_CHECK_DATA(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_CHECK) + +/* Max number of uids and gids */ +#define SQUASHFS_UIDS 256 +#define SQUASHFS_GUIDS 255 + +struct squashfs_super_block_3 { + unsigned int s_magic; + unsigned int inodes; + unsigned int bytes_used_2; + unsigned int uid_start_2; + unsigned int guid_start_2; + unsigned int inode_table_start_2; + unsigned int directory_table_start_2; + unsigned int s_major:16; + unsigned int s_minor:16; + unsigned int block_size_1:16; + unsigned int block_log:16; + unsigned int flags:8; + unsigned int no_uids:8; + unsigned int no_guids:8; + unsigned int mkfs_time /* time of filesystem creation */; + squashfs_inode root_inode; + unsigned int block_size; + unsigned int fragments; + unsigned int fragment_table_start_2; + long long bytes_used; + long long uid_start; + long long guid_start; + long long inode_table_start; + long long directory_table_start; + long long fragment_table_start; + long long lookup_table_start; +} __attribute__ ((packed)); + +struct squashfs_dir_index_3 { + unsigned int index; + unsigned int start_block; + unsigned char size; + unsigned char name[0]; +} __attribute__ ((packed)); + +#define SQUASHFS_BASE_INODE_HEADER_3 \ + unsigned int inode_type:4; \ + unsigned int mode:12; \ + unsigned int uid:8; \ + unsigned int guid:8; \ + unsigned int mtime; \ + unsigned int inode_number; + +struct squashfs_base_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; +} __attribute__ ((packed)); + +struct squashfs_ipc_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; + unsigned int nlink; +} __attribute__ ((packed)); + +struct squashfs_dev_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; + unsigned int nlink; + unsigned short rdev; +} __attribute__ ((packed)); + +struct squashfs_symlink_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; + unsigned int nlink; + unsigned short symlink_size; + char symlink[0]; +} __attribute__ ((packed)); + +struct squashfs_reg_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; + squashfs_block start_block; + unsigned int fragment; + unsigned int offset; + unsigned int file_size; + unsigned short block_list[0]; +} __attribute__ ((packed)); + +struct squashfs_lreg_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; + unsigned int nlink; + squashfs_block start_block; + unsigned int fragment; + unsigned int offset; + long long file_size; + unsigned short block_list[0]; +} __attribute__ ((packed)); + +struct squashfs_dir_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; + unsigned int nlink; + unsigned int file_size:19; + unsigned int offset:13; + unsigned int start_block; + unsigned int parent_inode; +} __attribute__ ((packed)); + +struct squashfs_ldir_inode_header_3 { + SQUASHFS_BASE_INODE_HEADER_3; + unsigned int nlink; + unsigned int file_size:27; + unsigned int offset:13; + unsigned int start_block; + unsigned int i_count:16; + unsigned int parent_inode; + struct squashfs_dir_index_3 index[0]; +} __attribute__ ((packed)); + +union squashfs_inode_header_3 { + struct squashfs_base_inode_header_3 base; + struct squashfs_dev_inode_header_3 dev; + struct squashfs_symlink_inode_header_3 symlink; + struct squashfs_reg_inode_header_3 reg; + struct squashfs_lreg_inode_header_3 lreg; + struct squashfs_dir_inode_header_3 dir; + struct squashfs_ldir_inode_header_3 ldir; + struct squashfs_ipc_inode_header_3 ipc; +}; + +struct squashfs_dir_entry_3 { + unsigned int offset:13; + unsigned int type:3; + unsigned int size:8; + int inode_number:16; + char name[0]; +} __attribute__ ((packed)); + +struct squashfs_dir_header_3 { + unsigned int count:8; + unsigned int start_block; + unsigned int inode_number; +} __attribute__ ((packed)); + +struct squashfs_fragment_entry_3 { + long long start_block; + unsigned int size; + unsigned int pending; +} __attribute__ ((packed)); + + +typedef struct squashfs_super_block_3 squashfs_super_block_3; +typedef struct squashfs_dir_index_3 squashfs_dir_index_3; +typedef struct squashfs_base_inode_header_3 squashfs_base_inode_header_3; +typedef struct squashfs_ipc_inode_header_3 squashfs_ipc_inode_header_3; +typedef struct squashfs_dev_inode_header_3 squashfs_dev_inode_header_3; +typedef struct squashfs_symlink_inode_header_3 squashfs_symlink_inode_header_3; +typedef struct squashfs_reg_inode_header_3 squashfs_reg_inode_header_3; +typedef struct squashfs_lreg_inode_header_3 squashfs_lreg_inode_header_3; +typedef struct squashfs_dir_inode_header_3 squashfs_dir_inode_header_3; +typedef struct squashfs_ldir_inode_header_3 squashfs_ldir_inode_header_3; +typedef struct squashfs_dir_entry_3 squashfs_dir_entry_3; +typedef struct squashfs_dir_header_3 squashfs_dir_header_3; +typedef struct squashfs_fragment_entry_3 squashfs_fragment_entry_3; + +/* + * macros to convert each packed bitfield structure from little endian to big + * endian and vice versa. These are needed when creating or using a filesystem + * on a machine with different byte ordering to the target architecture. + * + */ + +#define SQUASHFS_SWAP_START \ + int bits;\ + int b_pos;\ + unsigned long long val;\ + unsigned char *s;\ + unsigned char *d; + +#define SQUASHFS_SWAP_SUPER_BLOCK_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_super_block_3));\ + SQUASHFS_SWAP((s)->s_magic, d, 0, 32);\ + SQUASHFS_SWAP((s)->inodes, d, 32, 32);\ + SQUASHFS_SWAP((s)->bytes_used_2, d, 64, 32);\ + SQUASHFS_SWAP((s)->uid_start_2, d, 96, 32);\ + SQUASHFS_SWAP((s)->guid_start_2, d, 128, 32);\ + SQUASHFS_SWAP((s)->inode_table_start_2, d, 160, 32);\ + SQUASHFS_SWAP((s)->directory_table_start_2, d, 192, 32);\ + SQUASHFS_SWAP((s)->s_major, d, 224, 16);\ + SQUASHFS_SWAP((s)->s_minor, d, 240, 16);\ + SQUASHFS_SWAP((s)->block_size_1, d, 256, 16);\ + SQUASHFS_SWAP((s)->block_log, d, 272, 16);\ + SQUASHFS_SWAP((s)->flags, d, 288, 8);\ + SQUASHFS_SWAP((s)->no_uids, d, 296, 8);\ + SQUASHFS_SWAP((s)->no_guids, d, 304, 8);\ + SQUASHFS_SWAP((s)->mkfs_time, d, 312, 32);\ + SQUASHFS_SWAP((s)->root_inode, d, 344, 64);\ + SQUASHFS_SWAP((s)->block_size, d, 408, 32);\ + SQUASHFS_SWAP((s)->fragments, d, 440, 32);\ + SQUASHFS_SWAP((s)->fragment_table_start_2, d, 472, 32);\ + SQUASHFS_SWAP((s)->bytes_used, d, 504, 64);\ + SQUASHFS_SWAP((s)->uid_start, d, 568, 64);\ + SQUASHFS_SWAP((s)->guid_start, d, 632, 64);\ + SQUASHFS_SWAP((s)->inode_table_start, d, 696, 64);\ + SQUASHFS_SWAP((s)->directory_table_start, d, 760, 64);\ + SQUASHFS_SWAP((s)->fragment_table_start, d, 824, 64);\ + SQUASHFS_SWAP((s)->lookup_table_start, d, 888, 64);\ +} + +#define SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, n)\ + SQUASHFS_MEMSET(s, d, n);\ + SQUASHFS_SWAP((s)->inode_type, d, 0, 4);\ + SQUASHFS_SWAP((s)->mode, d, 4, 12);\ + SQUASHFS_SWAP((s)->uid, d, 16, 8);\ + SQUASHFS_SWAP((s)->guid, d, 24, 8);\ + SQUASHFS_SWAP((s)->mtime, d, 32, 32);\ + SQUASHFS_SWAP((s)->inode_number, d, 64, 32); + +#define SQUASHFS_SWAP_BASE_INODE_HEADER_3(s, d, n) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, n)\ +} + +#define SQUASHFS_SWAP_IPC_INODE_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ + sizeof(struct squashfs_ipc_inode_header_3))\ + SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ +} + +#define SQUASHFS_SWAP_DEV_INODE_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ + sizeof(struct squashfs_dev_inode_header_3)); \ + SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ + SQUASHFS_SWAP((s)->rdev, d, 128, 16);\ +} + +#define SQUASHFS_SWAP_SYMLINK_INODE_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ + sizeof(struct squashfs_symlink_inode_header_3));\ + SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ + SQUASHFS_SWAP((s)->symlink_size, d, 128, 16);\ +} + +#define SQUASHFS_SWAP_REG_INODE_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ + sizeof(struct squashfs_reg_inode_header_3));\ + SQUASHFS_SWAP((s)->start_block, d, 96, 64);\ + SQUASHFS_SWAP((s)->fragment, d, 160, 32);\ + SQUASHFS_SWAP((s)->offset, d, 192, 32);\ + SQUASHFS_SWAP((s)->file_size, d, 224, 32);\ +} + +#define SQUASHFS_SWAP_LREG_INODE_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ + sizeof(struct squashfs_lreg_inode_header_3));\ + SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ + SQUASHFS_SWAP((s)->start_block, d, 128, 64);\ + SQUASHFS_SWAP((s)->fragment, d, 192, 32);\ + SQUASHFS_SWAP((s)->offset, d, 224, 32);\ + SQUASHFS_SWAP((s)->file_size, d, 256, 64);\ +} + +#define SQUASHFS_SWAP_DIR_INODE_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ + sizeof(struct squashfs_dir_inode_header_3));\ + SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ + SQUASHFS_SWAP((s)->file_size, d, 128, 19);\ + SQUASHFS_SWAP((s)->offset, d, 147, 13);\ + SQUASHFS_SWAP((s)->start_block, d, 160, 32);\ + SQUASHFS_SWAP((s)->parent_inode, d, 192, 32);\ +} + +#define SQUASHFS_SWAP_LDIR_INODE_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ + sizeof(struct squashfs_ldir_inode_header_3));\ + SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ + SQUASHFS_SWAP((s)->file_size, d, 128, 27);\ + SQUASHFS_SWAP((s)->offset, d, 155, 13);\ + SQUASHFS_SWAP((s)->start_block, d, 168, 32);\ + SQUASHFS_SWAP((s)->i_count, d, 200, 16);\ + SQUASHFS_SWAP((s)->parent_inode, d, 216, 32);\ +} + +#define SQUASHFS_SWAP_DIR_INDEX_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_index_3));\ + SQUASHFS_SWAP((s)->index, d, 0, 32);\ + SQUASHFS_SWAP((s)->start_block, d, 32, 32);\ + SQUASHFS_SWAP((s)->size, d, 64, 8);\ +} + +#define SQUASHFS_SWAP_DIR_HEADER_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_header_3));\ + SQUASHFS_SWAP((s)->count, d, 0, 8);\ + SQUASHFS_SWAP((s)->start_block, d, 8, 32);\ + SQUASHFS_SWAP((s)->inode_number, d, 40, 32);\ +} + +#define SQUASHFS_SWAP_DIR_ENTRY_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_entry_3));\ + SQUASHFS_SWAP((s)->offset, d, 0, 13);\ + SQUASHFS_SWAP((s)->type, d, 13, 3);\ + SQUASHFS_SWAP((s)->size, d, 16, 8);\ + SQUASHFS_SWAP((s)->inode_number, d, 24, 16);\ +} + +#define SQUASHFS_SWAP_INODE_T_3(s, d) SQUASHFS_SWAP_LONG_LONGS_3(s, d, 1) + +#define SQUASHFS_SWAP_SHORTS_3(s, d, n) {\ + int entry;\ + int bit_position;\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, n * 2);\ + for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ + 16)\ + SQUASHFS_SWAP(s[entry], d, bit_position, 16);\ +} + +#define SQUASHFS_SWAP_INTS_3(s, d, n) {\ + int entry;\ + int bit_position;\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, n * 4);\ + for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ + 32)\ + SQUASHFS_SWAP(s[entry], d, bit_position, 32);\ +} + +#define SQUASHFS_SWAP_LONG_LONGS_3(s, d, n) {\ + int entry;\ + int bit_position;\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, n * 8);\ + for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ + 64)\ + SQUASHFS_SWAP(s[entry], d, bit_position, 64);\ +} + +#define SQUASHFS_SWAP_DATA(s, d, n, bits) {\ + int entry;\ + int bit_position;\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, n * bits / 8);\ + for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ + bits)\ + SQUASHFS_SWAP(s[entry], d, bit_position, bits);\ +} + +#define SQUASHFS_SWAP_FRAGMENT_INDEXES_3(s, d, n) SQUASHFS_SWAP_LONG_LONGS_3(s, d, n) +#define SQUASHFS_SWAP_LOOKUP_BLOCKS_3(s, d, n) SQUASHFS_SWAP_LONG_LONGS_3(s, d, n) + +#define SQUASHFS_SWAP_FRAGMENT_ENTRY_3(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_fragment_entry_3));\ + SQUASHFS_SWAP((s)->start_block, d, 0, 64);\ + SQUASHFS_SWAP((s)->size, d, 64, 32);\ +} + +/* fragment and fragment table defines */ +#define SQUASHFS_FRAGMENT_BYTES_3(A) ((A) * sizeof(struct squashfs_fragment_entry_3)) + +#define SQUASHFS_FRAGMENT_INDEX_3(A) (SQUASHFS_FRAGMENT_BYTES_3(A) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEX_OFFSET_3(A) (SQUASHFS_FRAGMENT_BYTES_3(A) % \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEXES_3(A) ((SQUASHFS_FRAGMENT_BYTES_3(A) + \ + SQUASHFS_METADATA_SIZE - 1) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEX_BYTES_3(A) (SQUASHFS_FRAGMENT_INDEXES_3(A) *\ + sizeof(long long)) + +/* + * definitions for structures on disk - layout 1.x + */ +#define SQUASHFS_TYPES 5 +#define SQUASHFS_IPC_TYPE 0 + +struct squashfs_base_inode_header_1 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:4; /* index into uid table */ + unsigned int guid:4; /* index into guid table */ +} __attribute__ ((packed)); + +struct squashfs_ipc_inode_header_1 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:4; /* index into uid table */ + unsigned int guid:4; /* index into guid table */ + unsigned int type:4; + unsigned int offset:4; +} __attribute__ ((packed)); + +struct squashfs_dev_inode_header_1 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:4; /* index into uid table */ + unsigned int guid:4; /* index into guid table */ + unsigned short rdev; +} __attribute__ ((packed)); + +struct squashfs_symlink_inode_header_1 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:4; /* index into uid table */ + unsigned int guid:4; /* index into guid table */ + unsigned short symlink_size; + char symlink[0]; +} __attribute__ ((packed)); + +struct squashfs_reg_inode_header_1 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:4; /* index into uid table */ + unsigned int guid:4; /* index into guid table */ + unsigned int mtime; + unsigned int start_block; + unsigned int file_size:32; + unsigned short block_list[0]; +} __attribute__ ((packed)); + +struct squashfs_dir_inode_header_1 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:4; /* index into uid table */ + unsigned int guid:4; /* index into guid table */ + unsigned int file_size:19; + unsigned int offset:13; + unsigned int mtime; + unsigned int start_block:24; +} __attribute__ ((packed)); + +union squashfs_inode_header_1 { + struct squashfs_base_inode_header_1 base; + struct squashfs_dev_inode_header_1 dev; + struct squashfs_symlink_inode_header_1 symlink; + struct squashfs_reg_inode_header_1 reg; + struct squashfs_dir_inode_header_1 dir; + struct squashfs_ipc_inode_header_1 ipc; +}; + +typedef struct squashfs_dir_index_1 squashfs_dir_index_1; +typedef struct squashfs_base_inode_header_1 squashfs_base_inode_header_1; +typedef struct squashfs_ipc_inode_header_1 squashfs_ipc_inode_header_1; +typedef struct squashfs_dev_inode_header_1 squashfs_dev_inode_header_1; +typedef struct squashfs_symlink_inode_header_1 squashfs_symlink_inode_header_1; +typedef struct squashfs_reg_inode_header_1 squashfs_reg_inode_header_1; +typedef struct squashfs_dir_inode_header_1 squashfs_dir_inode_header_1; + +#define SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, n) \ + SQUASHFS_MEMSET(s, d, n);\ + SQUASHFS_SWAP((s)->inode_type, d, 0, 4);\ + SQUASHFS_SWAP((s)->mode, d, 4, 12);\ + SQUASHFS_SWAP((s)->uid, d, 16, 4);\ + SQUASHFS_SWAP((s)->guid, d, 20, 4); + +#define SQUASHFS_SWAP_BASE_INODE_HEADER_1(s, d, n) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, n)\ +} + +#define SQUASHFS_SWAP_IPC_INODE_HEADER_1(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ + sizeof(struct squashfs_ipc_inode_header_1));\ + SQUASHFS_SWAP((s)->type, d, 24, 4);\ + SQUASHFS_SWAP((s)->offset, d, 28, 4);\ +} + +#define SQUASHFS_SWAP_DEV_INODE_HEADER_1(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ + sizeof(struct squashfs_dev_inode_header_1));\ + SQUASHFS_SWAP((s)->rdev, d, 24, 16);\ +} + +#define SQUASHFS_SWAP_SYMLINK_INODE_HEADER_1(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ + sizeof(struct squashfs_symlink_inode_header_1));\ + SQUASHFS_SWAP((s)->symlink_size, d, 24, 16);\ +} + +#define SQUASHFS_SWAP_REG_INODE_HEADER_1(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ + sizeof(struct squashfs_reg_inode_header_1));\ + SQUASHFS_SWAP((s)->mtime, d, 24, 32);\ + SQUASHFS_SWAP((s)->start_block, d, 56, 32);\ + SQUASHFS_SWAP((s)->file_size, d, 88, 32);\ +} + +#define SQUASHFS_SWAP_DIR_INODE_HEADER_1(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ + sizeof(struct squashfs_dir_inode_header_1));\ + SQUASHFS_SWAP((s)->file_size, d, 24, 19);\ + SQUASHFS_SWAP((s)->offset, d, 43, 13);\ + SQUASHFS_SWAP((s)->mtime, d, 56, 32);\ + SQUASHFS_SWAP((s)->start_block, d, 88, 24);\ +} + +/* + * definitions for structures on disk - layout 2.x + */ +struct squashfs_dir_index_2 { + unsigned int index:27; + unsigned int start_block:29; + unsigned char size; + unsigned char name[0]; +} __attribute__ ((packed)); + +struct squashfs_base_inode_header_2 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:8; /* index into uid table */ + unsigned int guid:8; /* index into guid table */ +} __attribute__ ((packed)); + +struct squashfs_ipc_inode_header_2 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:8; /* index into uid table */ + unsigned int guid:8; /* index into guid table */ +} __attribute__ ((packed)); + +struct squashfs_dev_inode_header_2 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:8; /* index into uid table */ + unsigned int guid:8; /* index into guid table */ + unsigned short rdev; +} __attribute__ ((packed)); + +struct squashfs_symlink_inode_header_2 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:8; /* index into uid table */ + unsigned int guid:8; /* index into guid table */ + unsigned short symlink_size; + char symlink[0]; +} __attribute__ ((packed)); + +struct squashfs_reg_inode_header_2 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:8; /* index into uid table */ + unsigned int guid:8; /* index into guid table */ + unsigned int mtime; + unsigned int start_block; + unsigned int fragment; + unsigned int offset; + unsigned int file_size:32; + unsigned short block_list[0]; +} __attribute__ ((packed)); + +struct squashfs_dir_inode_header_2 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:8; /* index into uid table */ + unsigned int guid:8; /* index into guid table */ + unsigned int file_size:19; + unsigned int offset:13; + unsigned int mtime; + unsigned int start_block:24; +} __attribute__ ((packed)); + +struct squashfs_ldir_inode_header_2 { + unsigned int inode_type:4; + unsigned int mode:12; /* protection */ + unsigned int uid:8; /* index into uid table */ + unsigned int guid:8; /* index into guid table */ + unsigned int file_size:27; + unsigned int offset:13; + unsigned int mtime; + unsigned int start_block:24; + unsigned int i_count:16; + struct squashfs_dir_index_2 index[0]; +} __attribute__ ((packed)); + +union squashfs_inode_header_2 { + struct squashfs_base_inode_header_2 base; + struct squashfs_dev_inode_header_2 dev; + struct squashfs_symlink_inode_header_2 symlink; + struct squashfs_reg_inode_header_2 reg; + struct squashfs_dir_inode_header_2 dir; + struct squashfs_ldir_inode_header_2 ldir; + struct squashfs_ipc_inode_header_2 ipc; +}; + +struct squashfs_dir_header_2 { + unsigned int count:8; + unsigned int start_block:24; +} __attribute__ ((packed)); + +struct squashfs_dir_entry_2 { + unsigned int offset:13; + unsigned int type:3; + unsigned int size:8; + char name[0]; +} __attribute__ ((packed)); + +struct squashfs_fragment_entry_2 { + unsigned int start_block; + unsigned int size; +} __attribute__ ((packed)); + +typedef struct squashfs_dir_index_2 squashfs_dir_index_2; +typedef struct squashfs_base_inode_header_2 squashfs_base_inode_header_2; +typedef struct squashfs_ipc_inode_header_2 squashfs_ipc_inode_header_2; +typedef struct squashfs_dev_inode_header_2 squashfs_dev_inode_header_2; +typedef struct squashfs_symlink_inode_header_2 squashfs_symlink_inode_header_2; +typedef struct squashfs_reg_inode_header_2 squashfs_reg_inode_header_2; +typedef struct squashfs_lreg_inode_header_2 squashfs_lreg_inode_header_2; +typedef struct squashfs_dir_inode_header_2 squashfs_dir_inode_header_2; +typedef struct squashfs_ldir_inode_header_2 squashfs_ldir_inode_header_2; +typedef struct squashfs_dir_entry_2 squashfs_dir_entry_2; +typedef struct squashfs_dir_header_2 squashfs_dir_header_2; +typedef struct squashfs_fragment_entry_2 squashfs_fragment_entry_2; + +#define SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, n)\ + SQUASHFS_MEMSET(s, d, n);\ + SQUASHFS_SWAP((s)->inode_type, d, 0, 4);\ + SQUASHFS_SWAP((s)->mode, d, 4, 12);\ + SQUASHFS_SWAP((s)->uid, d, 16, 8);\ + SQUASHFS_SWAP((s)->guid, d, 24, 8);\ + +#define SQUASHFS_SWAP_BASE_INODE_HEADER_2(s, d, n) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, n)\ +} + +#define SQUASHFS_SWAP_IPC_INODE_HEADER_2(s, d) \ + SQUASHFS_SWAP_BASE_INODE_HEADER_2(s, d, sizeof(struct squashfs_ipc_inode_header_2)) + +#define SQUASHFS_SWAP_DEV_INODE_HEADER_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ + sizeof(struct squashfs_dev_inode_header_2)); \ + SQUASHFS_SWAP((s)->rdev, d, 32, 16);\ +} + +#define SQUASHFS_SWAP_SYMLINK_INODE_HEADER_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ + sizeof(struct squashfs_symlink_inode_header_2));\ + SQUASHFS_SWAP((s)->symlink_size, d, 32, 16);\ +} + +#define SQUASHFS_SWAP_REG_INODE_HEADER_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ + sizeof(struct squashfs_reg_inode_header_2));\ + SQUASHFS_SWAP((s)->mtime, d, 32, 32);\ + SQUASHFS_SWAP((s)->start_block, d, 64, 32);\ + SQUASHFS_SWAP((s)->fragment, d, 96, 32);\ + SQUASHFS_SWAP((s)->offset, d, 128, 32);\ + SQUASHFS_SWAP((s)->file_size, d, 160, 32);\ +} + +#define SQUASHFS_SWAP_DIR_INODE_HEADER_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ + sizeof(struct squashfs_dir_inode_header_2));\ + SQUASHFS_SWAP((s)->file_size, d, 32, 19);\ + SQUASHFS_SWAP((s)->offset, d, 51, 13);\ + SQUASHFS_SWAP((s)->mtime, d, 64, 32);\ + SQUASHFS_SWAP((s)->start_block, d, 96, 24);\ +} + +#define SQUASHFS_SWAP_LDIR_INODE_HEADER_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ + sizeof(struct squashfs_ldir_inode_header_2));\ + SQUASHFS_SWAP((s)->file_size, d, 32, 27);\ + SQUASHFS_SWAP((s)->offset, d, 59, 13);\ + SQUASHFS_SWAP((s)->mtime, d, 72, 32);\ + SQUASHFS_SWAP((s)->start_block, d, 104, 24);\ + SQUASHFS_SWAP((s)->i_count, d, 128, 16);\ +} + +#define SQUASHFS_SWAP_DIR_INDEX_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_index_2));\ + SQUASHFS_SWAP((s)->index, d, 0, 27);\ + SQUASHFS_SWAP((s)->start_block, d, 27, 29);\ + SQUASHFS_SWAP((s)->size, d, 56, 8);\ +} +#define SQUASHFS_SWAP_DIR_HEADER_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_header_2));\ + SQUASHFS_SWAP((s)->count, d, 0, 8);\ + SQUASHFS_SWAP((s)->start_block, d, 8, 24);\ +} + +#define SQUASHFS_SWAP_DIR_ENTRY_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_entry_2));\ + SQUASHFS_SWAP((s)->offset, d, 0, 13);\ + SQUASHFS_SWAP((s)->type, d, 13, 3);\ + SQUASHFS_SWAP((s)->size, d, 16, 8);\ +} + +#define SQUASHFS_SWAP_FRAGMENT_ENTRY_2(s, d) {\ + SQUASHFS_SWAP_START\ + SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_fragment_entry_2));\ + SQUASHFS_SWAP((s)->start_block, d, 0, 32);\ + SQUASHFS_SWAP((s)->size, d, 32, 32);\ +} + +#define SQUASHFS_SWAP_FRAGMENT_INDEXES_2(s, d, n) SQUASHFS_SWAP_INTS_3(s, d, n) + +/* fragment and fragment table defines */ +#define SQUASHFS_FRAGMENT_BYTES_2(A) ((A) * sizeof(struct squashfs_fragment_entry_2)) + +#define SQUASHFS_FRAGMENT_INDEX_2(A) (SQUASHFS_FRAGMENT_BYTES_2(A) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEX_OFFSET_2(A) (SQUASHFS_FRAGMENT_BYTES_2(A) % \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEXES_2(A) ((SQUASHFS_FRAGMENT_BYTES_2(A) + \ + SQUASHFS_METADATA_SIZE - 1) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEX_BYTES_2(A) (SQUASHFS_FRAGMENT_INDEXES_2(A) *\ + sizeof(int)) +/* + * macros used to swap each structure entry, taking into account + * bitfields and different bitfield placing conventions on differing architectures + */ +#if __BYTE_ORDER == __BIG_ENDIAN + /* convert from big endian to little endian */ +#define SQUASHFS_SWAP(value, p, pos, tbits) _SQUASHFS_SWAP(value, p, pos, tbits, b_pos) +#else + /* convert from little endian to big endian */ +#define SQUASHFS_SWAP(value, p, pos, tbits) _SQUASHFS_SWAP(value, p, pos, tbits, 64 - tbits - b_pos) +#endif + +#define _SQUASHFS_SWAP(value, p, pos, tbits, SHIFT) {\ + b_pos = pos % 8;\ + val = 0;\ + s = (unsigned char *)p + (pos / 8);\ + d = ((unsigned char *) &val) + 7;\ + for(bits = 0; bits < (tbits + b_pos); bits += 8) \ + *d-- = *s++;\ + value = (val >> (SHIFT))/* & ((1 << tbits) - 1)*/;\ +} +#define SQUASHFS_MEMSET(s, d, n) memset(s, 0, n); +#endif diff --git a/squashfs-tools/squashfs_fs.h b/squashfs-tools/squashfs_fs.h new file mode 100644 index 0000000..d4fba1b --- /dev/null +++ b/squashfs-tools/squashfs_fs.h @@ -0,0 +1,490 @@ +#ifndef SQUASHFS_FS +#define SQUASHFS_FS +/* + * Squashfs + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * squashfs_fs.h + */ + +#define SQUASHFS_CACHED_FRAGMENTS CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE +#define SQUASHFS_MAJOR 4 +#define SQUASHFS_MINOR 0 +#define SQUASHFS_MAGIC 0x73717368 +#define SQUASHFS_MAGIC_SWAP 0x68737173 +#define SQUASHFS_START 0 + +/* size of metadata (inode and directory) blocks */ +#define SQUASHFS_METADATA_SIZE 8192 +#define SQUASHFS_METADATA_LOG 13 + +/* default size of data blocks */ +#define SQUASHFS_FILE_SIZE 131072 +#define SQUASHFS_FILE_LOG 17 + +#define SQUASHFS_FILE_MAX_SIZE 1048576 + +/* Max number of uids and gids */ +#define SQUASHFS_IDS 65536 + +/* Max length of filename (not 255) */ +#define SQUASHFS_NAME_LEN 256 + +#define SQUASHFS_INVALID ((long long) 0xffffffffffff) +#define SQUASHFS_INVALID_FRAG ((unsigned int) 0xffffffff) +#define SQUASHFS_INVALID_XATTR ((unsigned int) 0xffffffff) +#define SQUASHFS_INVALID_BLK ((long long) -1) +#define SQUASHFS_USED_BLK ((long long) -2) + +/* Filesystem flags */ +#define SQUASHFS_NOI 0 +#define SQUASHFS_NOD 1 +#define SQUASHFS_CHECK 2 +#define SQUASHFS_NOF 3 +#define SQUASHFS_NO_FRAG 4 +#define SQUASHFS_ALWAYS_FRAG 5 +#define SQUASHFS_DUPLICATE 6 +#define SQUASHFS_EXPORT 7 +#define SQUASHFS_NOX 8 +#define SQUASHFS_NO_XATTR 9 +#define SQUASHFS_COMP_OPT 10 + +#define SQUASHFS_BIT(flag, bit) ((flag >> bit) & 1) + +#define SQUASHFS_UNCOMPRESSED_INODES(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_NOI) + +#define SQUASHFS_UNCOMPRESSED_DATA(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_NOD) + +#define SQUASHFS_UNCOMPRESSED_FRAGMENTS(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_NOF) + +#define SQUASHFS_NO_FRAGMENTS(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_NO_FRAG) + +#define SQUASHFS_ALWAYS_FRAGMENTS(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_ALWAYS_FRAG) + +#define SQUASHFS_DUPLICATES(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_DUPLICATE) + +#define SQUASHFS_EXPORTABLE(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_EXPORT) + +#define SQUASHFS_UNCOMPRESSED_XATTRS(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_NOX) + +#define SQUASHFS_NO_XATTRS(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_NO_XATTR) + +#define SQUASHFS_COMP_OPTS(flags) SQUASHFS_BIT(flags, \ + SQUASHFS_COMP_OPT) + +#define SQUASHFS_MKFLAGS(noi, nod, nof, nox, no_frag, always_frag, \ + duplicate_checking, exportable, no_xattr, comp_opt) (noi | \ + (nod << 1) | (nof << 3) | (no_frag << 4) | \ + (always_frag << 5) | (duplicate_checking << 6) | \ + (exportable << 7) | (nox << 8) | (no_xattr << 9) | \ + (comp_opt << 10)) + +/* Max number of types and file types */ +#define SQUASHFS_DIR_TYPE 1 +#define SQUASHFS_FILE_TYPE 2 +#define SQUASHFS_SYMLINK_TYPE 3 +#define SQUASHFS_BLKDEV_TYPE 4 +#define SQUASHFS_CHRDEV_TYPE 5 +#define SQUASHFS_FIFO_TYPE 6 +#define SQUASHFS_SOCKET_TYPE 7 +#define SQUASHFS_LDIR_TYPE 8 +#define SQUASHFS_LREG_TYPE 9 +#define SQUASHFS_LSYMLINK_TYPE 10 +#define SQUASHFS_LBLKDEV_TYPE 11 +#define SQUASHFS_LCHRDEV_TYPE 12 +#define SQUASHFS_LFIFO_TYPE 13 +#define SQUASHFS_LSOCKET_TYPE 14 + +/* Xattr types */ +#define SQUASHFS_XATTR_USER 0 +#define SQUASHFS_XATTR_TRUSTED 1 +#define SQUASHFS_XATTR_SECURITY 2 +#define SQUASHFS_XATTR_VALUE_OOL 256 +#define SQUASHFS_XATTR_PREFIX_MASK 0xff + +/* Flag whether block is compressed or uncompressed, bit is set if block is + * uncompressed */ +#define SQUASHFS_COMPRESSED_BIT (1 << 15) + +#define SQUASHFS_COMPRESSED_SIZE(B) (((B) & ~SQUASHFS_COMPRESSED_BIT) ? \ + (B) & ~SQUASHFS_COMPRESSED_BIT : SQUASHFS_COMPRESSED_BIT) + +#define SQUASHFS_COMPRESSED(B) (!((B) & SQUASHFS_COMPRESSED_BIT)) + +#define SQUASHFS_COMPRESSED_BIT_BLOCK (1 << 24) + +#define SQUASHFS_COMPRESSED_SIZE_BLOCK(B) ((B) & \ + ~SQUASHFS_COMPRESSED_BIT_BLOCK) + +#define SQUASHFS_COMPRESSED_BLOCK(B) (!((B) & SQUASHFS_COMPRESSED_BIT_BLOCK)) + +/* + * Inode number ops. Inodes consist of a compressed block number, and an + * uncompressed offset within that block + */ +#define SQUASHFS_INODE_BLK(a) ((unsigned int) ((a) >> 16)) + +#define SQUASHFS_INODE_OFFSET(a) ((unsigned int) ((a) & 0xffff)) + +#define SQUASHFS_MKINODE(A, B) ((squashfs_inode)(((squashfs_inode) (A)\ + << 16) + (B))) + +/* Compute 32 bit VFS inode number from squashfs inode number */ +#define SQUASHFS_MK_VFS_INODE(a, b) ((unsigned int) (((a) << 8) + \ + ((b) >> 2) + 1)) + +/* Translate between VFS mode and squashfs mode */ +#define SQUASHFS_MODE(a) ((a) & 0xfff) + +/* fragment and fragment table defines */ +#define SQUASHFS_FRAGMENT_BYTES(A) ((A) * sizeof(struct squashfs_fragment_entry)) + +#define SQUASHFS_FRAGMENT_INDEX(A) (SQUASHFS_FRAGMENT_BYTES(A) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEX_OFFSET(A) (SQUASHFS_FRAGMENT_BYTES(A) % \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEXES(A) ((SQUASHFS_FRAGMENT_BYTES(A) + \ + SQUASHFS_METADATA_SIZE - 1) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_FRAGMENT_INDEX_BYTES(A) (SQUASHFS_FRAGMENT_INDEXES(A) *\ + sizeof(long long)) + +/* inode lookup table defines */ +#define SQUASHFS_LOOKUP_BYTES(A) ((A) * sizeof(squashfs_inode)) + +#define SQUASHFS_LOOKUP_BLOCK(A) (SQUASHFS_LOOKUP_BYTES(A) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_LOOKUP_BLOCK_OFFSET(A) (SQUASHFS_LOOKUP_BYTES(A) % \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_LOOKUP_BLOCKS(A) ((SQUASHFS_LOOKUP_BYTES(A) + \ + SQUASHFS_METADATA_SIZE - 1) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_LOOKUP_BLOCK_BYTES(A) (SQUASHFS_LOOKUP_BLOCKS(A) *\ + sizeof(long long)) + +/* uid lookup table defines */ +#define SQUASHFS_ID_BYTES(A) ((A) * sizeof(unsigned int)) + +#define SQUASHFS_ID_BLOCK(A) (SQUASHFS_ID_BYTES(A) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_ID_BLOCK_OFFSET(A) (SQUASHFS_ID_BYTES(A) % \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_ID_BLOCKS(A) ((SQUASHFS_ID_BYTES(A) + \ + SQUASHFS_METADATA_SIZE - 1) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_ID_BLOCK_BYTES(A) (SQUASHFS_ID_BLOCKS(A) *\ + sizeof(long long)) + +/* xattr id lookup table defines */ +#define SQUASHFS_XATTR_BYTES(A) ((A) * sizeof(struct squashfs_xattr_id)) + +#define SQUASHFS_XATTR_BLOCK(A) (SQUASHFS_XATTR_BYTES(A) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_XATTR_BLOCK_OFFSET(A) (SQUASHFS_XATTR_BYTES(A) % \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_XATTR_BLOCKS(A) ((SQUASHFS_XATTR_BYTES(A) + \ + SQUASHFS_METADATA_SIZE - 1) / \ + SQUASHFS_METADATA_SIZE) + +#define SQUASHFS_XATTR_BLOCK_BYTES(A) (SQUASHFS_XATTR_BLOCKS(A) *\ + sizeof(long long)) + +#define SQUASHFS_XATTR_BLK(A) ((unsigned int) ((A) >> 16)) + +#define SQUASHFS_XATTR_OFFSET(A) ((unsigned int) ((A) & 0xffff)) + +/* cached data constants for filesystem */ +#define SQUASHFS_CACHED_BLKS 8 + +#define SQUASHFS_MAX_FILE_SIZE_LOG 64 + +#define SQUASHFS_MAX_FILE_SIZE ((long long) 1 << \ + (SQUASHFS_MAX_FILE_SIZE_LOG - 2)) + +#define SQUASHFS_MARKER_BYTE 0xff + +/* meta index cache */ +#define SQUASHFS_META_INDEXES (SQUASHFS_METADATA_SIZE / sizeof(unsigned int)) +#define SQUASHFS_META_ENTRIES 31 +#define SQUASHFS_META_NUMBER 8 +#define SQUASHFS_SLOTS 4 + +struct meta_entry { + long long data_block; + unsigned int index_block; + unsigned short offset; + unsigned short pad; +}; + +struct meta_index { + unsigned int inode_number; + unsigned int offset; + unsigned short entries; + unsigned short skip; + unsigned short locked; + unsigned short pad; + struct meta_entry meta_entry[SQUASHFS_META_ENTRIES]; +}; + + +/* + * definitions for structures on disk + */ + +typedef long long squashfs_block; +typedef long long squashfs_inode; + +#define ZLIB_COMPRESSION 1 +#define LZMA_COMPRESSION 2 +#define LZO_COMPRESSION 3 +#define XZ_COMPRESSION 4 + +struct squashfs_super_block { + unsigned int s_magic; + unsigned int inodes; + unsigned int mkfs_time /* time of filesystem creation */; + unsigned int block_size; + unsigned int fragments; + unsigned short compression; + unsigned short block_log; + unsigned short flags; + unsigned short no_ids; + unsigned short s_major; + unsigned short s_minor; + squashfs_inode root_inode; + long long bytes_used; + long long id_table_start; + long long xattr_id_table_start; + long long inode_table_start; + long long directory_table_start; + long long fragment_table_start; + long long lookup_table_start; +}; + +struct squashfs_dir_index { + unsigned int index; + unsigned int start_block; + unsigned int size; + unsigned char name[0]; +}; + +struct squashfs_base_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; +}; + +struct squashfs_ipc_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int nlink; +}; + +struct squashfs_lipc_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int nlink; + unsigned int xattr; +}; + +struct squashfs_dev_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int nlink; + unsigned int rdev; +}; + +struct squashfs_ldev_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int nlink; + unsigned int rdev; + unsigned int xattr; +}; + +struct squashfs_symlink_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int nlink; + unsigned int symlink_size; + char symlink[0]; +}; + +struct squashfs_reg_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int start_block; + unsigned int fragment; + unsigned int offset; + unsigned int file_size; + unsigned int block_list[0]; +}; + +struct squashfs_lreg_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + squashfs_block start_block; + long long file_size; + long long sparse; + unsigned int nlink; + unsigned int fragment; + unsigned int offset; + unsigned int xattr; + unsigned int block_list[0]; +}; + +struct squashfs_dir_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int start_block; + unsigned int nlink; + unsigned short file_size; + unsigned short offset; + unsigned int parent_inode; +}; + +struct squashfs_ldir_inode_header { + unsigned short inode_type; + unsigned short mode; + unsigned short uid; + unsigned short guid; + unsigned int mtime; + unsigned int inode_number; + unsigned int nlink; + unsigned int file_size; + unsigned int start_block; + unsigned int parent_inode; + unsigned short i_count; + unsigned short offset; + unsigned int xattr; + struct squashfs_dir_index index[0]; +}; + +union squashfs_inode_header { + struct squashfs_base_inode_header base; + struct squashfs_dev_inode_header dev; + struct squashfs_ldev_inode_header ldev; + struct squashfs_symlink_inode_header symlink; + struct squashfs_reg_inode_header reg; + struct squashfs_lreg_inode_header lreg; + struct squashfs_dir_inode_header dir; + struct squashfs_ldir_inode_header ldir; + struct squashfs_ipc_inode_header ipc; + struct squashfs_lipc_inode_header lipc; +}; + +struct squashfs_dir_entry { + unsigned short offset; + short inode_number; + unsigned short type; + unsigned short size; + char name[0]; +}; + +struct squashfs_dir_header { + unsigned int count; + unsigned int start_block; + unsigned int inode_number; +}; + +struct squashfs_fragment_entry { + long long start_block; + unsigned int size; + unsigned int unused; +}; + +struct squashfs_xattr_entry { + unsigned short type; + unsigned short size; + char data[0]; +}; + +struct squashfs_xattr_val { + unsigned int vsize; + char value[0]; +}; + +struct squashfs_xattr_id { + long long xattr; + unsigned int count; + unsigned int size; +}; + +struct squashfs_xattr_table { + long long xattr_table_start; + unsigned int xattr_ids; + unsigned int unused; +}; + +#endif diff --git a/squashfs-tools/squashfs_swap.h b/squashfs-tools/squashfs_swap.h new file mode 100644 index 0000000..eed7d6e --- /dev/null +++ b/squashfs-tools/squashfs_swap.h @@ -0,0 +1,409 @@ +#ifndef SQUASHFS_SWAP_H +#define SQUASHFS_SWAP_H +/* + * Squashfs + * + * Copyright (c) 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * squashfs_swap.h + */ + +/* + * macros to convert each stucture from big endian to little endian + */ + +#if __BYTE_ORDER == __BIG_ENDIAN +#include +extern void swap_le16(void *, void *); +extern void swap_le32(void *, void *); +extern void swap_le64(void *, void *); +extern void swap_le16_num(void *, void *, int); +extern void swap_le32_num(void *, void *, int); +extern void swap_le64_num(void *, void *, int); +extern unsigned short inswap_le16(unsigned short); +extern unsigned int inswap_le32(unsigned int); +extern long long inswap_le64(long long); +extern void inswap_le16_num(unsigned short *, int); +extern void inswap_le32_num(unsigned int *, int); +extern void inswap_le64_num(long long *, int); + +#define _SQUASHFS_SWAP_SUPER_BLOCK(s, d, SWAP_FUNC) {\ + SWAP_FUNC(32, s, d, s_magic, struct squashfs_super_block);\ + SWAP_FUNC(32, s, d, inodes, struct squashfs_super_block);\ + SWAP_FUNC(32, s, d, mkfs_time, struct squashfs_super_block);\ + SWAP_FUNC(32, s, d, block_size, struct squashfs_super_block);\ + SWAP_FUNC(32, s, d, fragments, struct squashfs_super_block);\ + SWAP_FUNC(16, s, d, compression, struct squashfs_super_block);\ + SWAP_FUNC(16, s, d, block_log, struct squashfs_super_block);\ + SWAP_FUNC(16, s, d, flags, struct squashfs_super_block);\ + SWAP_FUNC(16, s, d, no_ids, struct squashfs_super_block);\ + SWAP_FUNC(16, s, d, s_major, struct squashfs_super_block);\ + SWAP_FUNC(16, s, d, s_minor, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, root_inode, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, bytes_used, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, id_table_start, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, xattr_id_table_start, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, inode_table_start, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, directory_table_start, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, fragment_table_start, struct squashfs_super_block);\ + SWAP_FUNC(64, s, d, lookup_table_start, struct squashfs_super_block);\ +} + +#define _SQUASHFS_SWAP_DIR_INDEX(s, d, SWAP_FUNC) {\ + SWAP_FUNC(32, s, d, index, struct squashfs_dir_index);\ + SWAP_FUNC(32, s, d, start_block, struct squashfs_dir_index);\ + SWAP_FUNC(32, s, d, size, struct squashfs_dir_index);\ +} + +#define _SQUASHFS_SWAP_BASE_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_base_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_base_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_base_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_base_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_base_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_base_inode_header);\ +} + +#define _SQUASHFS_SWAP_IPC_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_ipc_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_ipc_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_ipc_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_ipc_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_ipc_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_ipc_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_ipc_inode_header);\ +} + +#define _SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_lipc_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_lipc_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_lipc_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_lipc_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_lipc_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_lipc_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_lipc_inode_header);\ + SWAP_FUNC(32, s, d, xattr, struct squashfs_lipc_inode_header);\ +} + +#define _SQUASHFS_SWAP_DEV_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_dev_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_dev_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_dev_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_dev_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_dev_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_dev_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_dev_inode_header);\ + SWAP_FUNC(32, s, d, rdev, struct squashfs_dev_inode_header);\ +} + +#define _SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(32, s, d, rdev, struct squashfs_ldev_inode_header);\ + SWAP_FUNC(32, s, d, xattr, struct squashfs_ldev_inode_header);\ +} + +#define _SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_symlink_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_symlink_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_symlink_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_symlink_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_symlink_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_symlink_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_symlink_inode_header);\ + SWAP_FUNC(32, s, d, symlink_size, struct squashfs_symlink_inode_header);\ +} + +#define _SQUASHFS_SWAP_REG_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_reg_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_reg_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_reg_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_reg_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_reg_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_reg_inode_header);\ + SWAP_FUNC(32, s, d, start_block, struct squashfs_reg_inode_header);\ + SWAP_FUNC(32, s, d, fragment, struct squashfs_reg_inode_header);\ + SWAP_FUNC(32, s, d, offset, struct squashfs_reg_inode_header);\ + SWAP_FUNC(32, s, d, file_size, struct squashfs_reg_inode_header);\ +} + +#define _SQUASHFS_SWAP_LREG_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(64, s, d, start_block, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(64, s, d, file_size, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(64, s, d, sparse, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(32, s, d, fragment, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(32, s, d, offset, struct squashfs_lreg_inode_header);\ + SWAP_FUNC(32, s, d, xattr, struct squashfs_lreg_inode_header);\ +} + +#define _SQUASHFS_SWAP_DIR_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_dir_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_dir_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_dir_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_dir_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_dir_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_dir_inode_header);\ + SWAP_FUNC(32, s, d, start_block, struct squashfs_dir_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_dir_inode_header);\ + SWAP_FUNC(16, s, d, file_size, struct squashfs_dir_inode_header);\ + SWAP_FUNC(16, s, d, offset, struct squashfs_dir_inode_header);\ + SWAP_FUNC(32, s, d, parent_inode, struct squashfs_dir_inode_header);\ +} + +#define _SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, inode_type, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(16, s, d, mode, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(16, s, d, uid, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(16, s, d, guid, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(32, s, d, mtime, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(32, s, d, nlink, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(32, s, d, file_size, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(32, s, d, start_block, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(32, s, d, parent_inode, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(16, s, d, i_count, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(16, s, d, offset, struct squashfs_ldir_inode_header);\ + SWAP_FUNC(32, s, d, xattr, struct squashfs_ldir_inode_header);\ +} + +#define _SQUASHFS_SWAP_DIR_ENTRY(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, offset, struct squashfs_dir_entry);\ + SWAP_FUNC##S(16, s, d, inode_number, struct squashfs_dir_entry);\ + SWAP_FUNC(16, s, d, type, struct squashfs_dir_entry);\ + SWAP_FUNC(16, s, d, size, struct squashfs_dir_entry);\ +} + +#define _SQUASHFS_SWAP_DIR_HEADER(s, d, SWAP_FUNC) {\ + SWAP_FUNC(32, s, d, count, struct squashfs_dir_header);\ + SWAP_FUNC(32, s, d, start_block, struct squashfs_dir_header);\ + SWAP_FUNC(32, s, d, inode_number, struct squashfs_dir_header);\ +} + +#define _SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d, SWAP_FUNC) {\ + SWAP_FUNC(64, s, d, start_block, struct squashfs_fragment_entry);\ + SWAP_FUNC(32, s, d, size, struct squashfs_fragment_entry);\ +} + +#define _SQUASHFS_SWAP_XATTR_ENTRY(s, d, SWAP_FUNC) {\ + SWAP_FUNC(16, s, d, type, struct squashfs_xattr_entry);\ + SWAP_FUNC(16, s, d, size, struct squashfs_xattr_entry);\ +} + +#define _SQUASHFS_SWAP_XATTR_VAL(s, d, SWAP_FUNC) {\ + SWAP_FUNC(32, s, d, vsize, struct squashfs_xattr_val);\ +} + +#define _SQUASHFS_SWAP_XATTR_ID(s, d, SWAP_FUNC) {\ + SWAP_FUNC(64, s, d, xattr, struct squashfs_xattr_id);\ + SWAP_FUNC(32, s, d, count, struct squashfs_xattr_id);\ + SWAP_FUNC(32, s, d, size, struct squashfs_xattr_id);\ +} + +#define _SQUASHFS_SWAP_XATTR_TABLE(s, d, SWAP_FUNC) {\ + SWAP_FUNC(64, s, d, xattr_table_start, struct squashfs_xattr_table);\ + SWAP_FUNC(32, s, d, xattr_ids, struct squashfs_xattr_table);\ +} + +/* big endian architecture copy and swap macros */ +#define SQUASHFS_SWAP_SUPER_BLOCK(s, d) \ + _SQUASHFS_SWAP_SUPER_BLOCK(s, d, SWAP_LE) +#define SQUASHFS_SWAP_DIR_INDEX(s, d) \ + _SQUASHFS_SWAP_DIR_INDEX(s, d, SWAP_LE) +#define SQUASHFS_SWAP_BASE_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_BASE_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_IPC_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_IPC_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_DEV_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_DEV_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_REG_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_REG_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_LREG_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_LREG_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_DIR_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_DIR_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d) \ + _SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_DIR_ENTRY(s, d) \ + _SQUASHFS_SWAP_DIR_ENTRY(s, d, SWAP_LE) +#define SQUASHFS_SWAP_DIR_HEADER(s, d) \ + _SQUASHFS_SWAP_DIR_HEADER(s, d, SWAP_LE) +#define SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d) \ + _SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d, SWAP_LE) +#define SQUASHFS_SWAP_XATTR_ENTRY(s, d) \ + _SQUASHFS_SWAP_XATTR_ENTRY(s, d, SWAP_LE) +#define SQUASHFS_SWAP_XATTR_VAL(s, d) \ + _SQUASHFS_SWAP_XATTR_VAL(s, d, SWAP_LE) +#define SQUASHFS_SWAP_XATTR_ID(s, d) \ + _SQUASHFS_SWAP_XATTR_ID(s, d, SWAP_LE) +#define SQUASHFS_SWAP_XATTR_TABLE(s, d) \ + _SQUASHFS_SWAP_XATTR_TABLE(s, d, SWAP_LE) +#define SWAP_LE(bits, s, d, field, type) \ + SWAP_LE##bits(((void *)(s)) + offsetof(type, field), \ + ((void *)(d)) + offsetof(type, field)) +#define SWAP_LES(bits, s, d, field, type) \ + SWAP_LE(bits, s, d, field, type) +#define SQUASHFS_SWAP_INODE_T(s, d) SQUASHFS_SWAP_LONG_LONGS(s, d, 1) +#define SQUASHFS_SWAP_FRAGMENT_INDEXES(s, d, n) \ + SQUASHFS_SWAP_LONG_LONGS(s, d, n) +#define SQUASHFS_SWAP_LOOKUP_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) +#define SQUASHFS_SWAP_ID_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) + +/* big endian architecture swap in-place macros */ +#define SQUASHFS_INSWAP_SUPER_BLOCK(s) \ + _SQUASHFS_SWAP_SUPER_BLOCK(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_DIR_INDEX(s) \ + _SQUASHFS_SWAP_DIR_INDEX(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_BASE_INODE_HEADER(s) \ + _SQUASHFS_SWAP_BASE_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_IPC_INODE_HEADER(s) \ + _SQUASHFS_SWAP_IPC_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_LIPC_INODE_HEADER(s) \ + _SQUASHFS_SWAP_LIPC_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_DEV_INODE_HEADER(s) \ + _SQUASHFS_SWAP_DEV_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_LDEV_INODE_HEADER(s) \ + _SQUASHFS_SWAP_LDEV_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_SYMLINK_INODE_HEADER(s) \ + _SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_REG_INODE_HEADER(s) \ + _SQUASHFS_SWAP_REG_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_LREG_INODE_HEADER(s) \ + _SQUASHFS_SWAP_LREG_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_DIR_INODE_HEADER(s) \ + _SQUASHFS_SWAP_DIR_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_LDIR_INODE_HEADER(s) \ + _SQUASHFS_SWAP_LDIR_INODE_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_DIR_ENTRY(s) \ + _SQUASHFS_SWAP_DIR_ENTRY(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_DIR_HEADER(s) \ + _SQUASHFS_SWAP_DIR_HEADER(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_FRAGMENT_ENTRY(s) \ + _SQUASHFS_SWAP_FRAGMENT_ENTRY(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_XATTR_ENTRY(s) \ + _SQUASHFS_SWAP_XATTR_ENTRY(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_XATTR_VAL(s) \ + _SQUASHFS_SWAP_XATTR_VAL(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_XATTR_ID(s) \ + _SQUASHFS_SWAP_XATTR_ID(s, s, INSWAP_LE) +#define SQUASHFS_INSWAP_XATTR_TABLE(s) \ + _SQUASHFS_SWAP_XATTR_TABLE(s, s, INSWAP_LE) +#define INSWAP_LE(bits, s, d, field, type) \ + (s)->field = inswap_le##bits((s)->field) +#define INSWAP_LES(bits, s, d, field, type) \ + (s)->field = (short) inswap_le##bits((unsigned short) \ + (s)->field) +#define SQUASHFS_INSWAP_INODE_T(s) s = inswap_le64(s) +#define SQUASHFS_INSWAP_FRAGMENT_INDEXES(s, n) inswap_le64_num(s, n) +#define SQUASHFS_INSWAP_LOOKUP_BLOCKS(s, n) inswap_le64_num(s, n) +#define SQUASHFS_INSWAP_ID_BLOCKS(s, n) inswap_le64_num(s, n) +#define SQUASHFS_INSWAP_SHORTS(s, n) inswap_le16_num(s, n) +#define SQUASHFS_INSWAP_INTS(s, n) inswap_le32_num(s, n) +#define SQUASHFS_INSWAP_LONG_LONGS(s, n) inswap_le64_num(s, n) +#else +/* little endian architecture, just copy */ +#define SQUASHFS_SWAP_SUPER_BLOCK(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_DIR_INDEX(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_BASE_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_IPC_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_DEV_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_REG_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_LREG_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_DIR_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_DIR_ENTRY(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_DIR_HEADER(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_XATTR_ENTRY(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_XATTR_VAL(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_XATTR_ID(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_XATTR_TABLE(s, d) \ + SQUASHFS_MEMCPY(s, d, sizeof(*(s))) +#define SQUASHFS_SWAP_INODE_T(s, d) SQUASHFS_SWAP_LONG_LONGS(s, d, 1) +#define SQUASHFS_SWAP_FRAGMENT_INDEXES(s, d, n) \ + SQUASHFS_SWAP_LONG_LONGS(s, d, n) +#define SQUASHFS_SWAP_LOOKUP_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) +#define SQUASHFS_SWAP_ID_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) + +/* little endian architecture, data already in place so do nothing */ +#define SQUASHFS_INSWAP_SUPER_BLOCK(s) +#define SQUASHFS_INSWAP_DIR_INDEX(s) +#define SQUASHFS_INSWAP_BASE_INODE_HEADER(s) +#define SQUASHFS_INSWAP_IPC_INODE_HEADER(s) +#define SQUASHFS_INSWAP_LIPC_INODE_HEADER(s) +#define SQUASHFS_INSWAP_DEV_INODE_HEADER(s) +#define SQUASHFS_INSWAP_LDEV_INODE_HEADER(s) +#define SQUASHFS_INSWAP_SYMLINK_INODE_HEADER(s) +#define SQUASHFS_INSWAP_REG_INODE_HEADER(s) +#define SQUASHFS_INSWAP_LREG_INODE_HEADER(s) +#define SQUASHFS_INSWAP_DIR_INODE_HEADER(s) +#define SQUASHFS_INSWAP_LDIR_INODE_HEADER(s) +#define SQUASHFS_INSWAP_DIR_ENTRY(s) +#define SQUASHFS_INSWAP_DIR_HEADER(s) +#define SQUASHFS_INSWAP_FRAGMENT_ENTRY(s) +#define SQUASHFS_INSWAP_XATTR_ENTRY(s) +#define SQUASHFS_INSWAP_XATTR_VAL(s) +#define SQUASHFS_INSWAP_XATTR_ID(s) +#define SQUASHFS_INSWAP_XATTR_TABLE(s) +#define SQUASHFS_INSWAP_INODE_T(s) +#define SQUASHFS_INSWAP_FRAGMENT_INDEXES(s, n) +#define SQUASHFS_INSWAP_LOOKUP_BLOCKS(s, n) +#define SQUASHFS_INSWAP_ID_BLOCKS(s, n) +#define SQUASHFS_INSWAP_SHORTS(s, n) +#define SQUASHFS_INSWAP_INTS(s, n) +#define SQUASHFS_INSWAP_LONG_LONGS(s, n) +#endif +#endif diff --git a/squashfs-tools/swap.c b/squashfs-tools/swap.c new file mode 100644 index 0000000..6774bdc --- /dev/null +++ b/squashfs-tools/swap.c @@ -0,0 +1,123 @@ +/* + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * swap.c + */ + +#ifndef linux +#define __BYTE_ORDER BYTE_ORDER +#define __BIG_ENDIAN BIG_ENDIAN +#define __LITTLE_ENDIAN LITTLE_ENDIAN +#else +#include +#endif + +#if __BYTE_ORDER == __BIG_ENDIAN +void swap_le16(void *src, void *dest) +{ + unsigned char *s = src; + unsigned char *d = dest; + + d[0] = s[1]; + d[1] = s[0]; +} + + +void swap_le32(void *src, void *dest) +{ + unsigned char *s = src; + unsigned char *d = dest; + + d[0] = s[3]; + d[1] = s[2]; + d[2] = s[1]; + d[3] = s[0]; +} + + +void swap_le64(void *src, void *dest) +{ + unsigned char *s = src; + unsigned char *d = dest; + + d[0] = s[7]; + d[1] = s[6]; + d[2] = s[5]; + d[3] = s[4]; + d[4] = s[3]; + d[5] = s[2]; + d[6] = s[1]; + d[7] = s[0]; +} + + +unsigned short inswap_le16(unsigned short num) +{ + return (num >> 8) | + ((num & 0xff) << 8); +} + + +unsigned int inswap_le32(unsigned int num) +{ + return (num >> 24) | + ((num & 0xff0000) >> 8) | + ((num & 0xff00) << 8) | + ((num & 0xff) << 24); +} + + +long long inswap_le64(long long n) +{ + unsigned long long num = n; + + return (num >> 56) | + ((num & 0xff000000000000LL) >> 40) | + ((num & 0xff0000000000LL) >> 24) | + ((num & 0xff00000000LL) >> 8) | + ((num & 0xff000000) << 8) | + ((num & 0xff0000) << 24) | + ((num & 0xff00) << 40) | + ((num & 0xff) << 56); +} + + +#define SWAP_LE_NUM(BITS) \ +void swap_le##BITS##_num(void *s, void *d, int n) \ +{\ + int i;\ + for(i = 0; i < n; i++, s += BITS / 8, d += BITS / 8)\ + swap_le##BITS(s, d);\ +} + +SWAP_LE_NUM(16) +SWAP_LE_NUM(32) +SWAP_LE_NUM(64) + +#define INSWAP_LE_NUM(BITS, TYPE) \ +void inswap_le##BITS##_num(TYPE *s, int n) \ +{\ + int i;\ + for(i = 0; i < n; i++)\ + s[i] = inswap_le##BITS(s[i]);\ +} + +INSWAP_LE_NUM(16, unsigned short) +INSWAP_LE_NUM(32, unsigned int) +INSWAP_LE_NUM(64, long long) +#endif diff --git a/squashfs-tools/unsquash-1.c b/squashfs-tools/unsquash-1.c new file mode 100644 index 0000000..a403a79 --- /dev/null +++ b/squashfs-tools/unsquash-1.c @@ -0,0 +1,333 @@ +/* + * Unsquash a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * unsquash-1.c + */ + +#include "unsquashfs.h" +#include "squashfs_compat.h" + +void read_block_list_1(unsigned int *block_list, char *block_ptr, int blocks) +{ + unsigned short block_size; + int i; + + TRACE("read_block_list: blocks %d\n", blocks); + + for(i = 0; i < blocks; i++, block_ptr += 2) { + if(swap) { + unsigned short sblock_size; + memcpy(&sblock_size, block_ptr, sizeof(unsigned short)); + SQUASHFS_SWAP_SHORTS_3((&block_size), &sblock_size, 1); + } else + memcpy(&block_size, block_ptr, sizeof(unsigned short)); + block_list[i] = SQUASHFS_COMPRESSED_SIZE(block_size) | + (SQUASHFS_COMPRESSED(block_size) ? 0 : + SQUASHFS_COMPRESSED_BIT_BLOCK); + } +} + + +int read_fragment_table_1() +{ + TRACE("read_fragment_table\n"); + return TRUE; +} + + +struct inode *read_inode_1(unsigned int start_block, unsigned int offset) +{ + static union squashfs_inode_header_1 header; + long long start = sBlk.s.inode_table_start + start_block; + int bytes = lookup_entry(inode_table_hash, start); + char *block_ptr = inode_table + bytes + offset; + static struct inode i; + + TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); + + if(bytes == -1) + EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", + start); + + if(swap) { + squashfs_base_inode_header_1 sinode; + memcpy(&sinode, block_ptr, sizeof(header.base)); + SQUASHFS_SWAP_BASE_INODE_HEADER_1(&header.base, &sinode, + sizeof(squashfs_base_inode_header_1)); + } else + memcpy(&header.base, block_ptr, sizeof(header.base)); + + i.uid = (uid_t) uid_table[(header.base.inode_type - 1) / + SQUASHFS_TYPES * 16 + header.base.uid]; + if(header.base.inode_type == SQUASHFS_IPC_TYPE) { + squashfs_ipc_inode_header_1 *inodep = &header.ipc; + + if(swap) { + squashfs_ipc_inode_header_1 sinodep; + memcpy(&sinodep, block_ptr, sizeof(sinodep)); + SQUASHFS_SWAP_IPC_INODE_HEADER_1(inodep, &sinodep); + } else + memcpy(inodep, block_ptr, sizeof(*inodep)); + + if(inodep->type == SQUASHFS_SOCKET_TYPE) { + i.mode = S_IFSOCK | header.base.mode; + i.type = SQUASHFS_SOCKET_TYPE; + } else { + i.mode = S_IFIFO | header.base.mode; + i.type = SQUASHFS_FIFO_TYPE; + } + i.uid = (uid_t) uid_table[inodep->offset * 16 + inodep->uid]; + } else { + i.mode = lookup_type[(header.base.inode_type - 1) % + SQUASHFS_TYPES + 1] | header.base.mode; + i.type = (header.base.inode_type - 1) % SQUASHFS_TYPES + 1; + } + + i.xattr = SQUASHFS_INVALID_XATTR; + i.gid = header.base.guid == 15 ? i.uid : + (uid_t) guid_table[header.base.guid]; + i.time = sBlk.s.mkfs_time; + i.inode_number = inode_number ++; + + switch(i.type) { + case SQUASHFS_DIR_TYPE: { + squashfs_dir_inode_header_1 *inode = &header.dir; + + if(swap) { + squashfs_dir_inode_header_1 sinode; + memcpy(&sinode, block_ptr, sizeof(header.dir)); + SQUASHFS_SWAP_DIR_INODE_HEADER_1(inode, + &sinode); + } else + memcpy(inode, block_ptr, sizeof(header.dir)); + + i.data = inode->file_size; + i.offset = inode->offset; + i.start = inode->start_block; + i.time = inode->mtime; + break; + } + case SQUASHFS_FILE_TYPE: { + squashfs_reg_inode_header_1 *inode = &header.reg; + + if(swap) { + squashfs_reg_inode_header_1 sinode; + memcpy(&sinode, block_ptr, sizeof(sinode)); + SQUASHFS_SWAP_REG_INODE_HEADER_1(inode, + &sinode); + } else + memcpy(inode, block_ptr, sizeof(*inode)); + + i.data = inode->file_size; + i.time = inode->mtime; + i.blocks = (i.data + sBlk.s.block_size - 1) >> + sBlk.s.block_log; + i.start = inode->start_block; + i.block_ptr = block_ptr + sizeof(*inode); + i.fragment = 0; + i.frag_bytes = 0; + i.offset = 0; + i.sparse = 0; + break; + } + case SQUASHFS_SYMLINK_TYPE: { + squashfs_symlink_inode_header_1 *inodep = + &header.symlink; + + if(swap) { + squashfs_symlink_inode_header_1 sinodep; + memcpy(&sinodep, block_ptr, sizeof(sinodep)); + SQUASHFS_SWAP_SYMLINK_INODE_HEADER_1(inodep, + &sinodep); + } else + memcpy(inodep, block_ptr, sizeof(*inodep)); + + i.symlink = malloc(inodep->symlink_size + 1); + if(i.symlink == NULL) + EXIT_UNSQUASH("read_inode: failed to malloc " + "symlink data\n"); + strncpy(i.symlink, block_ptr + + sizeof(squashfs_symlink_inode_header_1), + inodep->symlink_size); + i.symlink[inodep->symlink_size] = '\0'; + i.data = inodep->symlink_size; + break; + } + case SQUASHFS_BLKDEV_TYPE: + case SQUASHFS_CHRDEV_TYPE: { + squashfs_dev_inode_header_1 *inodep = &header.dev; + + if(swap) { + squashfs_dev_inode_header_1 sinodep; + memcpy(&sinodep, block_ptr, sizeof(sinodep)); + SQUASHFS_SWAP_DEV_INODE_HEADER_1(inodep, + &sinodep); + } else + memcpy(inodep, block_ptr, sizeof(*inodep)); + + i.data = inodep->rdev; + break; + } + case SQUASHFS_FIFO_TYPE: + case SQUASHFS_SOCKET_TYPE: { + i.data = 0; + break; + } + default: + EXIT_UNSQUASH("Unknown inode type %d in " + " read_inode_header_1!\n", + header.base.inode_type); + } + return &i; +} + + +struct dir *squashfs_opendir_1(unsigned int block_start, unsigned int offset, + struct inode **i) +{ + squashfs_dir_header_2 dirh; + char buffer[sizeof(squashfs_dir_entry_2) + SQUASHFS_NAME_LEN + 1] + __attribute__((aligned)); + squashfs_dir_entry_2 *dire = (squashfs_dir_entry_2 *) buffer; + long long start; + int bytes; + int dir_count, size; + struct dir_ent *new_dir; + struct dir *dir; + + TRACE("squashfs_opendir: inode start block %d, offset %d\n", + block_start, offset); + + *i = s_ops.read_inode(block_start, offset); + start = sBlk.s.directory_table_start + (*i)->start; + bytes = lookup_entry(directory_table_hash, start); + if(bytes == -1) + EXIT_UNSQUASH("squashfs_opendir: directory block %d not " + "found!\n", block_start); + + bytes += (*i)->offset; + size = (*i)->data + bytes; + + dir = malloc(sizeof(struct dir)); + if(dir == NULL) + EXIT_UNSQUASH("squashfs_opendir: malloc failed!\n"); + + dir->dir_count = 0; + dir->cur_entry = 0; + dir->mode = (*i)->mode; + dir->uid = (*i)->uid; + dir->guid = (*i)->gid; + dir->mtime = (*i)->time; + dir->xattr = (*i)->xattr; + dir->dirs = NULL; + + while(bytes < size) { + if(swap) { + squashfs_dir_header_2 sdirh; + memcpy(&sdirh, directory_table + bytes, sizeof(sdirh)); + SQUASHFS_SWAP_DIR_HEADER_2(&dirh, &sdirh); + } else + memcpy(&dirh, directory_table + bytes, sizeof(dirh)); + + dir_count = dirh.count + 1; + TRACE("squashfs_opendir: Read directory header @ byte position " + "%d, %d directory entries\n", bytes, dir_count); + bytes += sizeof(dirh); + + while(dir_count--) { + if(swap) { + squashfs_dir_entry_2 sdire; + memcpy(&sdire, directory_table + bytes, + sizeof(sdire)); + SQUASHFS_SWAP_DIR_ENTRY_2(dire, &sdire); + } else + memcpy(dire, directory_table + bytes, + sizeof(*dire)); + bytes += sizeof(*dire); + + memcpy(dire->name, directory_table + bytes, + dire->size + 1); + dire->name[dire->size + 1] = '\0'; + TRACE("squashfs_opendir: directory entry %s, inode " + "%d:%d, type %d\n", dire->name, + dirh.start_block, dire->offset, dire->type); + if((dir->dir_count % DIR_ENT_SIZE) == 0) { + new_dir = realloc(dir->dirs, (dir->dir_count + + DIR_ENT_SIZE) * sizeof(struct dir_ent)); + if(new_dir == NULL) + EXIT_UNSQUASH("squashfs_opendir: " + "realloc failed!\n"); + dir->dirs = new_dir; + } + strcpy(dir->dirs[dir->dir_count].name, dire->name); + dir->dirs[dir->dir_count].start_block = + dirh.start_block; + dir->dirs[dir->dir_count].offset = dire->offset; + dir->dirs[dir->dir_count].type = dire->type; + dir->dir_count ++; + bytes += dire->size + 1; + } + } + + return dir; +} + + +int read_uids_guids_1() +{ + int res; + + TRACE("read_uids_guids: no_uids %d, no_guids %d\n", sBlk.no_uids, + sBlk.no_guids); + + uid_table = malloc((sBlk.no_uids + sBlk.no_guids) * + sizeof(unsigned int)); + if(uid_table == NULL) { + ERROR("read_uids_guids: failed to allocate uid/gid table\n"); + return FALSE; + } + + guid_table = uid_table + sBlk.no_uids; + + if(swap) { + unsigned int suid_table[sBlk.no_uids + sBlk.no_guids]; + + res = read_fs_bytes(fd, sBlk.uid_start, (sBlk.no_uids + + sBlk.no_guids) * sizeof(unsigned int), suid_table); + if(res == FALSE) { + ERROR("read_uids_guids: failed to read uid/gid table" + "\n"); + return FALSE; + } + SQUASHFS_SWAP_INTS_3(uid_table, suid_table, + sBlk.no_uids + sBlk.no_guids); + } else { + res = read_fs_bytes(fd, sBlk.uid_start, (sBlk.no_uids + + sBlk.no_guids) * sizeof(unsigned int), uid_table); + if(res == FALSE) { + ERROR("read_uids_guids: failed to read uid/gid table" + "\n"); + return FALSE; + } + } + + return TRUE; +} diff --git a/squashfs-tools/unsquash-2.c b/squashfs-tools/unsquash-2.c new file mode 100644 index 0000000..62c1db2 --- /dev/null +++ b/squashfs-tools/unsquash-2.c @@ -0,0 +1,264 @@ +/* + * Unsquash a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * unsquash-2.c + */ + +#include "unsquashfs.h" +#include "squashfs_compat.h" + +static squashfs_fragment_entry_2 *fragment_table; + +void read_block_list_2(unsigned int *block_list, char *block_ptr, int blocks) +{ + TRACE("read_block_list: blocks %d\n", blocks); + + if(swap) { + unsigned int sblock_list[blocks]; + memcpy(sblock_list, block_ptr, blocks * sizeof(unsigned int)); + SQUASHFS_SWAP_INTS_3(block_list, sblock_list, blocks); + } else + memcpy(block_list, block_ptr, blocks * sizeof(unsigned int)); +} + + +int read_fragment_table_2() +{ + int res, i, indexes = SQUASHFS_FRAGMENT_INDEXES_2(sBlk.s.fragments); + unsigned int fragment_table_index[indexes]; + + TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " + "from 0x%llx\n", sBlk.s.fragments, indexes, + sBlk.s.fragment_table_start); + + if(sBlk.s.fragments == 0) + return TRUE; + + fragment_table = malloc(sBlk.s.fragments * + sizeof(squashfs_fragment_entry_2)); + if(fragment_table == NULL) + EXIT_UNSQUASH("read_fragment_table: failed to allocate " + "fragment table\n"); + + if(swap) { + unsigned int sfragment_table_index[indexes]; + + res = read_fs_bytes(fd, sBlk.s.fragment_table_start, + SQUASHFS_FRAGMENT_INDEX_BYTES_2(sBlk.s.fragments), + sfragment_table_index); + if(res == FALSE) { + ERROR("read_fragment_table: failed to read fragment " + "table index\n"); + return FALSE; + } + SQUASHFS_SWAP_FRAGMENT_INDEXES_2(fragment_table_index, + sfragment_table_index, indexes); + } else { + res = read_fs_bytes(fd, sBlk.s.fragment_table_start, + SQUASHFS_FRAGMENT_INDEX_BYTES_2(sBlk.s.fragments), + fragment_table_index); + if(res == FALSE) { + ERROR("read_fragment_table: failed to read fragment " + "table index\n"); + return FALSE; + } + } + + for(i = 0; i < indexes; i++) { + int length = read_block(fd, fragment_table_index[i], NULL, + ((char *) fragment_table) + (i * + SQUASHFS_METADATA_SIZE)); + TRACE("Read fragment table block %d, from 0x%x, length %d\n", i, + fragment_table_index[i], length); + if(length == FALSE) { + ERROR("read_fragment_table: failed to read fragment " + "table block\n"); + return FALSE; + } + } + + if(swap) { + squashfs_fragment_entry_2 sfragment; + for(i = 0; i < sBlk.s.fragments; i++) { + SQUASHFS_SWAP_FRAGMENT_ENTRY_2((&sfragment), + (&fragment_table[i])); + memcpy((char *) &fragment_table[i], (char *) &sfragment, + sizeof(squashfs_fragment_entry_2)); + } + } + + return TRUE; +} + + +void read_fragment_2(unsigned int fragment, long long *start_block, int *size) +{ + TRACE("read_fragment: reading fragment %d\n", fragment); + + squashfs_fragment_entry_2 *fragment_entry = &fragment_table[fragment]; + *start_block = fragment_entry->start_block; + *size = fragment_entry->size; +} + + +struct inode *read_inode_2(unsigned int start_block, unsigned int offset) +{ + static union squashfs_inode_header_2 header; + long long start = sBlk.s.inode_table_start + start_block; + int bytes = lookup_entry(inode_table_hash, start); + char *block_ptr = inode_table + bytes + offset; + static struct inode i; + + TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); + + if(bytes == -1) + EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", + start); + + if(swap) { + squashfs_base_inode_header_2 sinode; + memcpy(&sinode, block_ptr, sizeof(header.base)); + SQUASHFS_SWAP_BASE_INODE_HEADER_2(&header.base, &sinode, + sizeof(squashfs_base_inode_header_2)); + } else + memcpy(&header.base, block_ptr, sizeof(header.base)); + + i.xattr = SQUASHFS_INVALID_XATTR; + i.uid = (uid_t) uid_table[header.base.uid]; + i.gid = header.base.guid == SQUASHFS_GUIDS ? i.uid : + (uid_t) guid_table[header.base.guid]; + i.mode = lookup_type[header.base.inode_type] | header.base.mode; + i.type = header.base.inode_type; + i.time = sBlk.s.mkfs_time; + i.inode_number = inode_number++; + + switch(header.base.inode_type) { + case SQUASHFS_DIR_TYPE: { + squashfs_dir_inode_header_2 *inode = &header.dir; + + if(swap) { + squashfs_dir_inode_header_2 sinode; + memcpy(&sinode, block_ptr, sizeof(header.dir)); + SQUASHFS_SWAP_DIR_INODE_HEADER_2(&header.dir, + &sinode); + } else + memcpy(&header.dir, block_ptr, + sizeof(header.dir)); + + i.data = inode->file_size; + i.offset = inode->offset; + i.start = inode->start_block; + i.time = inode->mtime; + break; + } + case SQUASHFS_LDIR_TYPE: { + squashfs_ldir_inode_header_2 *inode = &header.ldir; + + if(swap) { + squashfs_ldir_inode_header_2 sinode; + memcpy(&sinode, block_ptr, sizeof(header.ldir)); + SQUASHFS_SWAP_LDIR_INODE_HEADER_2(&header.ldir, + &sinode); + } else + memcpy(&header.ldir, block_ptr, + sizeof(header.ldir)); + + i.data = inode->file_size; + i.offset = inode->offset; + i.start = inode->start_block; + i.time = inode->mtime; + break; + } + case SQUASHFS_FILE_TYPE: { + squashfs_reg_inode_header_2 *inode = &header.reg; + + if(swap) { + squashfs_reg_inode_header_2 sinode; + memcpy(&sinode, block_ptr, sizeof(sinode)); + SQUASHFS_SWAP_REG_INODE_HEADER_2(inode, + &sinode); + } else + memcpy(inode, block_ptr, sizeof(*inode)); + + i.data = inode->file_size; + i.time = inode->mtime; + i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG + ? 0 : inode->file_size % sBlk.s.block_size; + i.fragment = inode->fragment; + i.offset = inode->offset; + i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? + (i.data + sBlk.s.block_size - 1) >> + sBlk.s.block_log : i.data >> + sBlk.s.block_log; + i.start = inode->start_block; + i.sparse = 0; + i.block_ptr = block_ptr + sizeof(*inode); + break; + } + case SQUASHFS_SYMLINK_TYPE: { + squashfs_symlink_inode_header_2 *inodep = + &header.symlink; + + if(swap) { + squashfs_symlink_inode_header_2 sinodep; + memcpy(&sinodep, block_ptr, sizeof(sinodep)); + SQUASHFS_SWAP_SYMLINK_INODE_HEADER_2(inodep, + &sinodep); + } else + memcpy(inodep, block_ptr, sizeof(*inodep)); + + i.symlink = malloc(inodep->symlink_size + 1); + if(i.symlink == NULL) + EXIT_UNSQUASH("read_inode: failed to malloc " + "symlink data\n"); + strncpy(i.symlink, block_ptr + + sizeof(squashfs_symlink_inode_header_2), + inodep->symlink_size); + i.symlink[inodep->symlink_size] = '\0'; + i.data = inodep->symlink_size; + break; + } + case SQUASHFS_BLKDEV_TYPE: + case SQUASHFS_CHRDEV_TYPE: { + squashfs_dev_inode_header_2 *inodep = &header.dev; + + if(swap) { + squashfs_dev_inode_header_2 sinodep; + memcpy(&sinodep, block_ptr, sizeof(sinodep)); + SQUASHFS_SWAP_DEV_INODE_HEADER_2(inodep, + &sinodep); + } else + memcpy(inodep, block_ptr, sizeof(*inodep)); + + i.data = inodep->rdev; + break; + } + case SQUASHFS_FIFO_TYPE: + case SQUASHFS_SOCKET_TYPE: + i.data = 0; + break; + default: + EXIT_UNSQUASH("Unknown inode type %d in " + "read_inode_header_2!\n", + header.base.inode_type); + } + return &i; +} diff --git a/squashfs-tools/unsquash-3.c b/squashfs-tools/unsquash-3.c new file mode 100644 index 0000000..86b5079 --- /dev/null +++ b/squashfs-tools/unsquash-3.c @@ -0,0 +1,364 @@ +/* + * Unsquash a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * unsquash-3.c + */ + +#include "unsquashfs.h" +#include "squashfs_compat.h" + +static squashfs_fragment_entry_3 *fragment_table; + +int read_fragment_table_3() +{ + int res, i, indexes = SQUASHFS_FRAGMENT_INDEXES_3(sBlk.s.fragments); + long long fragment_table_index[indexes]; + + TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " + "from 0x%llx\n", sBlk.s.fragments, indexes, + sBlk.s.fragment_table_start); + + if(sBlk.s.fragments == 0) + return TRUE; + + fragment_table = malloc(sBlk.s.fragments * + sizeof(squashfs_fragment_entry_3)); + if(fragment_table == NULL) + EXIT_UNSQUASH("read_fragment_table: failed to allocate " + "fragment table\n"); + + if(swap) { + long long sfragment_table_index[indexes]; + + res = read_fs_bytes(fd, sBlk.s.fragment_table_start, + SQUASHFS_FRAGMENT_INDEX_BYTES_3(sBlk.s.fragments), + sfragment_table_index); + if(res == FALSE) { + ERROR("read_fragment_table: failed to read fragment " + "table index\n"); + return FALSE; + } + SQUASHFS_SWAP_FRAGMENT_INDEXES_3(fragment_table_index, + sfragment_table_index, indexes); + } else { + res = read_fs_bytes(fd, sBlk.s.fragment_table_start, + SQUASHFS_FRAGMENT_INDEX_BYTES_3(sBlk.s.fragments), + fragment_table_index); + if(res == FALSE) { + ERROR("read_fragment_table: failed to read fragment " + "table index\n"); + return FALSE; + } + } + + for(i = 0; i < indexes; i++) { + int length = read_block(fd, fragment_table_index[i], NULL, + ((char *) fragment_table) + (i * + SQUASHFS_METADATA_SIZE)); + TRACE("Read fragment table block %d, from 0x%llx, length %d\n", + i, fragment_table_index[i], length); + if(length == FALSE) { + ERROR("read_fragment_table: failed to read fragment " + "table block\n"); + return FALSE; + } + } + + if(swap) { + squashfs_fragment_entry_3 sfragment; + for(i = 0; i < sBlk.s.fragments; i++) { + SQUASHFS_SWAP_FRAGMENT_ENTRY_3((&sfragment), + (&fragment_table[i])); + memcpy((char *) &fragment_table[i], (char *) &sfragment, + sizeof(squashfs_fragment_entry_3)); + } + } + + return TRUE; +} + + +void read_fragment_3(unsigned int fragment, long long *start_block, int *size) +{ + TRACE("read_fragment: reading fragment %d\n", fragment); + + squashfs_fragment_entry_3 *fragment_entry = &fragment_table[fragment]; + *start_block = fragment_entry->start_block; + *size = fragment_entry->size; +} + + +struct inode *read_inode_3(unsigned int start_block, unsigned int offset) +{ + static union squashfs_inode_header_3 header; + long long start = sBlk.s.inode_table_start + start_block; + int bytes = lookup_entry(inode_table_hash, start); + char *block_ptr = inode_table + bytes + offset; + static struct inode i; + + TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); + + if(bytes == -1) + EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", + start); + + if(swap) { + squashfs_base_inode_header_3 sinode; + memcpy(&sinode, block_ptr, sizeof(header.base)); + SQUASHFS_SWAP_BASE_INODE_HEADER_3(&header.base, &sinode, + sizeof(squashfs_base_inode_header_3)); + } else + memcpy(&header.base, block_ptr, sizeof(header.base)); + + i.xattr = SQUASHFS_INVALID_XATTR; + i.uid = (uid_t) uid_table[header.base.uid]; + i.gid = header.base.guid == SQUASHFS_GUIDS ? i.uid : + (uid_t) guid_table[header.base.guid]; + i.mode = lookup_type[header.base.inode_type] | header.base.mode; + i.type = header.base.inode_type; + i.time = header.base.mtime; + i.inode_number = header.base.inode_number; + + switch(header.base.inode_type) { + case SQUASHFS_DIR_TYPE: { + squashfs_dir_inode_header_3 *inode = &header.dir; + + if(swap) { + squashfs_dir_inode_header_3 sinode; + memcpy(&sinode, block_ptr, sizeof(header.dir)); + SQUASHFS_SWAP_DIR_INODE_HEADER_3(&header.dir, + &sinode); + } else + memcpy(&header.dir, block_ptr, + sizeof(header.dir)); + + i.data = inode->file_size; + i.offset = inode->offset; + i.start = inode->start_block; + break; + } + case SQUASHFS_LDIR_TYPE: { + squashfs_ldir_inode_header_3 *inode = &header.ldir; + + if(swap) { + squashfs_ldir_inode_header_3 sinode; + memcpy(&sinode, block_ptr, sizeof(header.ldir)); + SQUASHFS_SWAP_LDIR_INODE_HEADER_3(&header.ldir, + &sinode); + } else + memcpy(&header.ldir, block_ptr, + sizeof(header.ldir)); + + i.data = inode->file_size; + i.offset = inode->offset; + i.start = inode->start_block; + break; + } + case SQUASHFS_FILE_TYPE: { + squashfs_reg_inode_header_3 *inode = &header.reg; + + if(swap) { + squashfs_reg_inode_header_3 sinode; + memcpy(&sinode, block_ptr, sizeof(sinode)); + SQUASHFS_SWAP_REG_INODE_HEADER_3(inode, + &sinode); + } else + memcpy(inode, block_ptr, sizeof(*inode)); + + i.data = inode->file_size; + i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG + ? 0 : inode->file_size % sBlk.s.block_size; + i.fragment = inode->fragment; + i.offset = inode->offset; + i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? + (i.data + sBlk.s.block_size - 1) >> + sBlk.s.block_log : + i.data >> sBlk.s.block_log; + i.start = inode->start_block; + i.sparse = 1; + i.block_ptr = block_ptr + sizeof(*inode); + break; + } + case SQUASHFS_LREG_TYPE: { + squashfs_lreg_inode_header_3 *inode = &header.lreg; + + if(swap) { + squashfs_lreg_inode_header_3 sinode; + memcpy(&sinode, block_ptr, sizeof(sinode)); + SQUASHFS_SWAP_LREG_INODE_HEADER_3(inode, + &sinode); + } else + memcpy(inode, block_ptr, sizeof(*inode)); + + i.data = inode->file_size; + i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG + ? 0 : inode->file_size % sBlk.s.block_size; + i.fragment = inode->fragment; + i.offset = inode->offset; + i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? + (inode->file_size + sBlk.s.block_size - 1) >> + sBlk.s.block_log : + inode->file_size >> sBlk.s.block_log; + i.start = inode->start_block; + i.sparse = 1; + i.block_ptr = block_ptr + sizeof(*inode); + break; + } + case SQUASHFS_SYMLINK_TYPE: { + squashfs_symlink_inode_header_3 *inodep = + &header.symlink; + + if(swap) { + squashfs_symlink_inode_header_3 sinodep; + memcpy(&sinodep, block_ptr, sizeof(sinodep)); + SQUASHFS_SWAP_SYMLINK_INODE_HEADER_3(inodep, + &sinodep); + } else + memcpy(inodep, block_ptr, sizeof(*inodep)); + + i.symlink = malloc(inodep->symlink_size + 1); + if(i.symlink == NULL) + EXIT_UNSQUASH("read_inode: failed to malloc " + "symlink data\n"); + strncpy(i.symlink, block_ptr + + sizeof(squashfs_symlink_inode_header_3), + inodep->symlink_size); + i.symlink[inodep->symlink_size] = '\0'; + i.data = inodep->symlink_size; + break; + } + case SQUASHFS_BLKDEV_TYPE: + case SQUASHFS_CHRDEV_TYPE: { + squashfs_dev_inode_header_3 *inodep = &header.dev; + + if(swap) { + squashfs_dev_inode_header_3 sinodep; + memcpy(&sinodep, block_ptr, sizeof(sinodep)); + SQUASHFS_SWAP_DEV_INODE_HEADER_3(inodep, + &sinodep); + } else + memcpy(inodep, block_ptr, sizeof(*inodep)); + + i.data = inodep->rdev; + break; + } + case SQUASHFS_FIFO_TYPE: + case SQUASHFS_SOCKET_TYPE: + i.data = 0; + break; + default: + EXIT_UNSQUASH("Unknown inode type %d in read_inode!\n", + header.base.inode_type); + } + return &i; +} + + +struct dir *squashfs_opendir_3(unsigned int block_start, unsigned int offset, + struct inode **i) +{ + squashfs_dir_header_3 dirh; + char buffer[sizeof(squashfs_dir_entry_3) + SQUASHFS_NAME_LEN + 1] + __attribute__((aligned)); + squashfs_dir_entry_3 *dire = (squashfs_dir_entry_3 *) buffer; + long long start; + int bytes; + int dir_count, size; + struct dir_ent *new_dir; + struct dir *dir; + + TRACE("squashfs_opendir: inode start block %d, offset %d\n", + block_start, offset); + + *i = s_ops.read_inode(block_start, offset); + start = sBlk.s.directory_table_start + (*i)->start; + bytes = lookup_entry(directory_table_hash, start); + + if(bytes == -1) + EXIT_UNSQUASH("squashfs_opendir: directory block %d not " + "found!\n", block_start); + + bytes += (*i)->offset; + size = (*i)->data + bytes - 3; + + dir = malloc(sizeof(struct dir)); + if(dir == NULL) + EXIT_UNSQUASH("squashfs_opendir: malloc failed!\n"); + + dir->dir_count = 0; + dir->cur_entry = 0; + dir->mode = (*i)->mode; + dir->uid = (*i)->uid; + dir->guid = (*i)->gid; + dir->mtime = (*i)->time; + dir->xattr = (*i)->xattr; + dir->dirs = NULL; + + while(bytes < size) { + if(swap) { + squashfs_dir_header_3 sdirh; + memcpy(&sdirh, directory_table + bytes, sizeof(sdirh)); + SQUASHFS_SWAP_DIR_HEADER_3(&dirh, &sdirh); + } else + memcpy(&dirh, directory_table + bytes, sizeof(dirh)); + + dir_count = dirh.count + 1; + TRACE("squashfs_opendir: Read directory header @ byte position " + "%d, %d directory entries\n", bytes, dir_count); + bytes += sizeof(dirh); + + while(dir_count--) { + if(swap) { + squashfs_dir_entry_3 sdire; + memcpy(&sdire, directory_table + bytes, + sizeof(sdire)); + SQUASHFS_SWAP_DIR_ENTRY_3(dire, &sdire); + } else + memcpy(dire, directory_table + bytes, + sizeof(*dire)); + bytes += sizeof(*dire); + + memcpy(dire->name, directory_table + bytes, + dire->size + 1); + dire->name[dire->size + 1] = '\0'; + TRACE("squashfs_opendir: directory entry %s, inode " + "%d:%d, type %d\n", dire->name, + dirh.start_block, dire->offset, dire->type); + if((dir->dir_count % DIR_ENT_SIZE) == 0) { + new_dir = realloc(dir->dirs, (dir->dir_count + + DIR_ENT_SIZE) * sizeof(struct dir_ent)); + if(new_dir == NULL) + EXIT_UNSQUASH("squashfs_opendir: " + "realloc failed!\n"); + dir->dirs = new_dir; + } + strcpy(dir->dirs[dir->dir_count].name, dire->name); + dir->dirs[dir->dir_count].start_block = + dirh.start_block; + dir->dirs[dir->dir_count].offset = dire->offset; + dir->dirs[dir->dir_count].type = dire->type; + dir->dir_count ++; + bytes += dire->size + 1; + } + } + + return dir; +} diff --git a/squashfs-tools/unsquash-4.c b/squashfs-tools/unsquash-4.c new file mode 100644 index 0000000..49ab9de --- /dev/null +++ b/squashfs-tools/unsquash-4.c @@ -0,0 +1,360 @@ +/* + * Unsquash a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * unsquash-4.c + */ + +#include "unsquashfs.h" +#include "squashfs_swap.h" +#include "read_fs.h" + +static struct squashfs_fragment_entry *fragment_table; +static unsigned int *id_table; + +int read_fragment_table_4() +{ + int res, i, indexes = SQUASHFS_FRAGMENT_INDEXES(sBlk.s.fragments); + long long fragment_table_index[indexes]; + + TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " + "from 0x%llx\n", sBlk.s.fragments, indexes, + sBlk.s.fragment_table_start); + + if(sBlk.s.fragments == 0) + return TRUE; + + fragment_table = malloc(sBlk.s.fragments * + sizeof(struct squashfs_fragment_entry)); + if(fragment_table == NULL) + EXIT_UNSQUASH("read_fragment_table: failed to allocate " + "fragment table\n"); + + res = read_fs_bytes(fd, sBlk.s.fragment_table_start, + SQUASHFS_FRAGMENT_INDEX_BYTES(sBlk.s.fragments), + fragment_table_index); + if(res == FALSE) { + ERROR("read_fragment_table: failed to read fragment table " + "index\n"); + return FALSE; + } + SQUASHFS_INSWAP_FRAGMENT_INDEXES(fragment_table_index, indexes); + + for(i = 0; i < indexes; i++) { + int length = read_block(fd, fragment_table_index[i], NULL, + ((char *) fragment_table) + (i * + SQUASHFS_METADATA_SIZE)); + TRACE("Read fragment table block %d, from 0x%llx, length %d\n", + i, fragment_table_index[i], length); + if(length == FALSE) { + ERROR("read_fragment_table: failed to read fragment " + "table index\n"); + return FALSE; + } + } + + for(i = 0; i < sBlk.s.fragments; i++) + SQUASHFS_INSWAP_FRAGMENT_ENTRY(&fragment_table[i]); + + return TRUE; +} + + +void read_fragment_4(unsigned int fragment, long long *start_block, int *size) +{ + TRACE("read_fragment: reading fragment %d\n", fragment); + + struct squashfs_fragment_entry *fragment_entry; + + fragment_entry = &fragment_table[fragment]; + *start_block = fragment_entry->start_block; + *size = fragment_entry->size; +} + + +struct inode *read_inode_4(unsigned int start_block, unsigned int offset) +{ + static union squashfs_inode_header header; + long long start = sBlk.s.inode_table_start + start_block; + int bytes = lookup_entry(inode_table_hash, start); + char *block_ptr = inode_table + bytes + offset; + static struct inode i; + + TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); + + if(bytes == -1) + EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", + start); + + SQUASHFS_SWAP_BASE_INODE_HEADER(&header.base, block_ptr); + + i.uid = (uid_t) id_table[header.base.uid]; + i.gid = (uid_t) id_table[header.base.guid]; + i.mode = lookup_type[header.base.inode_type] | header.base.mode; + i.type = header.base.inode_type; + i.time = header.base.mtime; + i.inode_number = header.base.inode_number; + + switch(header.base.inode_type) { + case SQUASHFS_DIR_TYPE: { + struct squashfs_dir_inode_header *inode = &header.dir; + + SQUASHFS_SWAP_DIR_INODE_HEADER(inode, block_ptr); + + i.data = inode->file_size; + i.offset = inode->offset; + i.start = inode->start_block; + i.xattr = SQUASHFS_INVALID_XATTR; + break; + } + case SQUASHFS_LDIR_TYPE: { + struct squashfs_ldir_inode_header *inode = &header.ldir; + + SQUASHFS_SWAP_LDIR_INODE_HEADER(inode, block_ptr); + + i.data = inode->file_size; + i.offset = inode->offset; + i.start = inode->start_block; + i.xattr = inode->xattr; + break; + } + case SQUASHFS_FILE_TYPE: { + struct squashfs_reg_inode_header *inode = &header.reg; + + SQUASHFS_SWAP_REG_INODE_HEADER(inode, block_ptr); + + i.data = inode->file_size; + i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG + ? 0 : inode->file_size % sBlk.s.block_size; + i.fragment = inode->fragment; + i.offset = inode->offset; + i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? + (i.data + sBlk.s.block_size - 1) >> + sBlk.s.block_log : + i.data >> sBlk.s.block_log; + i.start = inode->start_block; + i.sparse = 0; + i.block_ptr = block_ptr + sizeof(*inode); + i.xattr = SQUASHFS_INVALID_XATTR; + break; + } + case SQUASHFS_LREG_TYPE: { + struct squashfs_lreg_inode_header *inode = &header.lreg; + + SQUASHFS_SWAP_LREG_INODE_HEADER(inode, block_ptr); + + i.data = inode->file_size; + i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG + ? 0 : inode->file_size % sBlk.s.block_size; + i.fragment = inode->fragment; + i.offset = inode->offset; + i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? + (inode->file_size + sBlk.s.block_size - 1) >> + sBlk.s.block_log : + inode->file_size >> sBlk.s.block_log; + i.start = inode->start_block; + i.sparse = inode->sparse != 0; + i.block_ptr = block_ptr + sizeof(*inode); + i.xattr = inode->xattr; + break; + } + case SQUASHFS_SYMLINK_TYPE: + case SQUASHFS_LSYMLINK_TYPE: { + struct squashfs_symlink_inode_header *inode = &header.symlink; + + SQUASHFS_SWAP_SYMLINK_INODE_HEADER(inode, block_ptr); + + i.symlink = malloc(inode->symlink_size + 1); + if(i.symlink == NULL) + EXIT_UNSQUASH("read_inode: failed to malloc " + "symlink data\n"); + strncpy(i.symlink, block_ptr + + sizeof(struct squashfs_symlink_inode_header), + inode->symlink_size); + i.symlink[inode->symlink_size] = '\0'; + i.data = inode->symlink_size; + + if(header.base.inode_type == SQUASHFS_LSYMLINK_TYPE) + SQUASHFS_SWAP_INTS(&i.xattr, block_ptr + + sizeof(struct squashfs_symlink_inode_header) + + inode->symlink_size, 1); + else + i.xattr = SQUASHFS_INVALID_XATTR; + break; + } + case SQUASHFS_BLKDEV_TYPE: + case SQUASHFS_CHRDEV_TYPE: { + struct squashfs_dev_inode_header *inode = &header.dev; + + SQUASHFS_SWAP_DEV_INODE_HEADER(inode, block_ptr); + + i.data = inode->rdev; + i.xattr = SQUASHFS_INVALID_XATTR; + break; + } + case SQUASHFS_LBLKDEV_TYPE: + case SQUASHFS_LCHRDEV_TYPE: { + struct squashfs_ldev_inode_header *inode = &header.ldev; + + SQUASHFS_SWAP_LDEV_INODE_HEADER(inode, block_ptr); + + i.data = inode->rdev; + i.xattr = inode->xattr; + break; + } + case SQUASHFS_FIFO_TYPE: + case SQUASHFS_SOCKET_TYPE: + i.data = 0; + i.xattr = SQUASHFS_INVALID_XATTR; + break; + case SQUASHFS_LFIFO_TYPE: + case SQUASHFS_LSOCKET_TYPE: { + struct squashfs_lipc_inode_header *inode = &header.lipc; + + SQUASHFS_SWAP_LIPC_INODE_HEADER(inode, block_ptr); + + i.data = 0; + i.xattr = inode->xattr; + break; + } + default: + EXIT_UNSQUASH("Unknown inode type %d in read_inode!\n", + header.base.inode_type); + } + return &i; +} + + +struct dir *squashfs_opendir_4(unsigned int block_start, unsigned int offset, + struct inode **i) +{ + struct squashfs_dir_header dirh; + char buffer[sizeof(struct squashfs_dir_entry) + SQUASHFS_NAME_LEN + 1] + __attribute__((aligned)); + struct squashfs_dir_entry *dire = (struct squashfs_dir_entry *) buffer; + long long start; + int bytes; + int dir_count, size; + struct dir_ent *new_dir; + struct dir *dir; + + TRACE("squashfs_opendir: inode start block %d, offset %d\n", + block_start, offset); + + *i = s_ops.read_inode(block_start, offset); + start = sBlk.s.directory_table_start + (*i)->start; + bytes = lookup_entry(directory_table_hash, start); + + if(bytes == -1) + EXIT_UNSQUASH("squashfs_opendir: directory block %d not " + "found!\n", block_start); + + bytes += (*i)->offset; + size = (*i)->data + bytes - 3; + + dir = malloc(sizeof(struct dir)); + if(dir == NULL) + EXIT_UNSQUASH("squashfs_opendir: malloc failed!\n"); + + dir->dir_count = 0; + dir->cur_entry = 0; + dir->mode = (*i)->mode; + dir->uid = (*i)->uid; + dir->guid = (*i)->gid; + dir->mtime = (*i)->time; + dir->xattr = (*i)->xattr; + dir->dirs = NULL; + + while(bytes < size) { + SQUASHFS_SWAP_DIR_HEADER(&dirh, directory_table + bytes); + + dir_count = dirh.count + 1; + TRACE("squashfs_opendir: Read directory header @ byte position " + "%d, %d directory entries\n", bytes, dir_count); + bytes += sizeof(dirh); + + while(dir_count--) { + SQUASHFS_SWAP_DIR_ENTRY(dire, directory_table + bytes); + + bytes += sizeof(*dire); + + memcpy(dire->name, directory_table + bytes, + dire->size + 1); + dire->name[dire->size + 1] = '\0'; + TRACE("squashfs_opendir: directory entry %s, inode " + "%d:%d, type %d\n", dire->name, + dirh.start_block, dire->offset, dire->type); + if((dir->dir_count % DIR_ENT_SIZE) == 0) { + new_dir = realloc(dir->dirs, (dir->dir_count + + DIR_ENT_SIZE) * sizeof(struct dir_ent)); + if(new_dir == NULL) + EXIT_UNSQUASH("squashfs_opendir: " + "realloc failed!\n"); + dir->dirs = new_dir; + } + strcpy(dir->dirs[dir->dir_count].name, dire->name); + dir->dirs[dir->dir_count].start_block = + dirh.start_block; + dir->dirs[dir->dir_count].offset = dire->offset; + dir->dirs[dir->dir_count].type = dire->type; + dir->dir_count ++; + bytes += dire->size + 1; + } + } + + return dir; +} + + +int read_uids_guids_4() +{ + int res, i, indexes = SQUASHFS_ID_BLOCKS(sBlk.s.no_ids); + long long id_index_table[indexes]; + + TRACE("read_uids_guids: no_ids %d\n", sBlk.s.no_ids); + + id_table = malloc(SQUASHFS_ID_BYTES(sBlk.s.no_ids)); + if(id_table == NULL) { + ERROR("read_uids_guids: failed to allocate id table\n"); + return FALSE; + } + + res = read_fs_bytes(fd, sBlk.s.id_table_start, + SQUASHFS_ID_BLOCK_BYTES(sBlk.s.no_ids), id_index_table); + if(res == FALSE) { + ERROR("read_uids_guids: failed to read id index table\n"); + return FALSE; + } + SQUASHFS_INSWAP_ID_BLOCKS(id_index_table, indexes); + + for(i = 0; i < indexes; i++) { + res = read_block(fd, id_index_table[i], NULL, + ((char *) id_table) + i * SQUASHFS_METADATA_SIZE); + if(res == FALSE) { + ERROR("read_uids_guids: failed to read id table block" + "\n"); + return FALSE; + } + } + + SQUASHFS_INSWAP_INTS(id_table, sBlk.s.no_ids); + + return TRUE; +} diff --git a/squashfs-tools/unsquashfs.c b/squashfs-tools/unsquashfs.c new file mode 100644 index 0000000..529dfac --- /dev/null +++ b/squashfs-tools/unsquashfs.c @@ -0,0 +1,2238 @@ +/* + * Unsquash a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * unsquashfs.c + */ + +#include "unsquashfs.h" +#include "squashfs_swap.h" +#include "squashfs_compat.h" +#include "read_fs.h" +#include "compressor.h" +#include "xattr.h" + +#include +#include + +struct cache *fragment_cache, *data_cache; +struct queue *to_reader, *to_deflate, *to_writer, *from_writer; +pthread_t *thread, *deflator_thread; +pthread_mutex_t fragment_mutex; + +/* user options that control parallelisation */ +int processors = -1; + +struct super_block sBlk; +squashfs_operations s_ops; +struct compressor *comp; + +int bytes = 0, swap, file_count = 0, dir_count = 0, sym_count = 0, + dev_count = 0, fifo_count = 0; +char *inode_table = NULL, *directory_table = NULL; +struct hash_table_entry *inode_table_hash[65536], *directory_table_hash[65536]; +int fd; +unsigned int *uid_table, *guid_table; +unsigned int cached_frag = SQUASHFS_INVALID_FRAG; +char *fragment_data; +char *file_data; +char *data; +unsigned int block_size; +unsigned int block_log; +int lsonly = FALSE, info = FALSE, force = FALSE, short_ls = TRUE; +int use_regex = FALSE; +char **created_inode; +int root_process; +int columns; +int rotate = 0; +pthread_mutex_t screen_mutex; +pthread_cond_t progress_wait; +int progress = TRUE, progress_enabled = FALSE; +unsigned int total_blocks = 0, total_files = 0, total_inodes = 0; +unsigned int cur_blocks = 0; +int inode_number = 1; +int no_xattrs = XATTR_DEF; + +int lookup_type[] = { + 0, + S_IFDIR, + S_IFREG, + S_IFLNK, + S_IFBLK, + S_IFCHR, + S_IFIFO, + S_IFSOCK, + S_IFDIR, + S_IFREG, + S_IFLNK, + S_IFBLK, + S_IFCHR, + S_IFIFO, + S_IFSOCK +}; + +struct test table[] = { + { S_IFMT, S_IFSOCK, 0, 's' }, + { S_IFMT, S_IFLNK, 0, 'l' }, + { S_IFMT, S_IFBLK, 0, 'b' }, + { S_IFMT, S_IFDIR, 0, 'd' }, + { S_IFMT, S_IFCHR, 0, 'c' }, + { S_IFMT, S_IFIFO, 0, 'p' }, + { S_IRUSR, S_IRUSR, 1, 'r' }, + { S_IWUSR, S_IWUSR, 2, 'w' }, + { S_IRGRP, S_IRGRP, 4, 'r' }, + { S_IWGRP, S_IWGRP, 5, 'w' }, + { S_IROTH, S_IROTH, 7, 'r' }, + { S_IWOTH, S_IWOTH, 8, 'w' }, + { S_IXUSR | S_ISUID, S_IXUSR | S_ISUID, 3, 's' }, + { S_IXUSR | S_ISUID, S_ISUID, 3, 'S' }, + { S_IXUSR | S_ISUID, S_IXUSR, 3, 'x' }, + { S_IXGRP | S_ISGID, S_IXGRP | S_ISGID, 6, 's' }, + { S_IXGRP | S_ISGID, S_ISGID, 6, 'S' }, + { S_IXGRP | S_ISGID, S_IXGRP, 6, 'x' }, + { S_IXOTH | S_ISVTX, S_IXOTH | S_ISVTX, 9, 't' }, + { S_IXOTH | S_ISVTX, S_ISVTX, 9, 'T' }, + { S_IXOTH | S_ISVTX, S_IXOTH, 9, 'x' }, + { 0, 0, 0, 0} +}; + +void progress_bar(long long current, long long max, int columns); +void update_progress_bar(); + +void sigwinch_handler() +{ + struct winsize winsize; + + if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { + if(isatty(STDOUT_FILENO)) + ERROR("TIOCGWINSZ ioctl failed, defaulting to 80 " + "columns\n"); + columns = 80; + } else + columns = winsize.ws_col; +} + + +void sigalrm_handler() +{ + rotate = (rotate + 1) % 4; +} + + +struct queue *queue_init(int size) +{ + struct queue *queue = malloc(sizeof(struct queue)); + + if(queue == NULL) + EXIT_UNSQUASH("Out of memory in queue_init\n"); + + queue->data = malloc(sizeof(void *) * (size + 1)); + if(queue->data == NULL) + EXIT_UNSQUASH("Out of memory in queue_init\n"); + + queue->size = size + 1; + queue->readp = queue->writep = 0; + pthread_mutex_init(&queue->mutex, NULL); + pthread_cond_init(&queue->empty, NULL); + pthread_cond_init(&queue->full, NULL); + + return queue; +} + + +void queue_put(struct queue *queue, void *data) +{ + int nextp; + + pthread_mutex_lock(&queue->mutex); + + while((nextp = (queue->writep + 1) % queue->size) == queue->readp) + pthread_cond_wait(&queue->full, &queue->mutex); + + queue->data[queue->writep] = data; + queue->writep = nextp; + pthread_cond_signal(&queue->empty); + pthread_mutex_unlock(&queue->mutex); +} + + +void *queue_get(struct queue *queue) +{ + void *data; + pthread_mutex_lock(&queue->mutex); + + while(queue->readp == queue->writep) + pthread_cond_wait(&queue->empty, &queue->mutex); + + data = queue->data[queue->readp]; + queue->readp = (queue->readp + 1) % queue->size; + pthread_cond_signal(&queue->full); + pthread_mutex_unlock(&queue->mutex); + + return data; +} + + +/* Called with the cache mutex held */ +void insert_hash_table(struct cache *cache, struct cache_entry *entry) +{ + int hash = CALCULATE_HASH(entry->block); + + entry->hash_next = cache->hash_table[hash]; + cache->hash_table[hash] = entry; + entry->hash_prev = NULL; + if(entry->hash_next) + entry->hash_next->hash_prev = entry; +} + + +/* Called with the cache mutex held */ +void remove_hash_table(struct cache *cache, struct cache_entry *entry) +{ + if(entry->hash_prev) + entry->hash_prev->hash_next = entry->hash_next; + else + cache->hash_table[CALCULATE_HASH(entry->block)] = + entry->hash_next; + if(entry->hash_next) + entry->hash_next->hash_prev = entry->hash_prev; + + entry->hash_prev = entry->hash_next = NULL; +} + + +/* Called with the cache mutex held */ +void insert_free_list(struct cache *cache, struct cache_entry *entry) +{ + if(cache->free_list) { + entry->free_next = cache->free_list; + entry->free_prev = cache->free_list->free_prev; + cache->free_list->free_prev->free_next = entry; + cache->free_list->free_prev = entry; + } else { + cache->free_list = entry; + entry->free_prev = entry->free_next = entry; + } +} + + +/* Called with the cache mutex held */ +void remove_free_list(struct cache *cache, struct cache_entry *entry) +{ + if(entry->free_prev == NULL && entry->free_next == NULL) + /* not in free list */ + return; + else if(entry->free_prev == entry && entry->free_next == entry) { + /* only this entry in the free list */ + cache->free_list = NULL; + } else { + /* more than one entry in the free list */ + entry->free_next->free_prev = entry->free_prev; + entry->free_prev->free_next = entry->free_next; + if(cache->free_list == entry) + cache->free_list = entry->free_next; + } + + entry->free_prev = entry->free_next = NULL; +} + + +struct cache *cache_init(int buffer_size, int max_buffers) +{ + struct cache *cache = malloc(sizeof(struct cache)); + + if(cache == NULL) + EXIT_UNSQUASH("Out of memory in cache_init\n"); + + cache->max_buffers = max_buffers; + cache->buffer_size = buffer_size; + cache->count = 0; + cache->free_list = NULL; + memset(cache->hash_table, 0, sizeof(struct cache_entry *) * 65536); + cache->wait_free = FALSE; + cache->wait_pending = FALSE; + pthread_mutex_init(&cache->mutex, NULL); + pthread_cond_init(&cache->wait_for_free, NULL); + pthread_cond_init(&cache->wait_for_pending, NULL); + + return cache; +} + + +struct cache_entry *cache_get(struct cache *cache, long long block, int size) +{ + /* + * Get a block out of the cache. If the block isn't in the cache + * it is added and queued to the reader() and deflate() threads for + * reading off disk and decompression. The cache grows until max_blocks + * is reached, once this occurs existing discarded blocks on the free + * list are reused + */ + int hash = CALCULATE_HASH(block); + struct cache_entry *entry; + + pthread_mutex_lock(&cache->mutex); + + for(entry = cache->hash_table[hash]; entry; entry = entry->hash_next) + if(entry->block == block) + break; + + if(entry) { + /* + * found the block in the cache, increment used count and + * if necessary remove from free list so it won't disappear + */ + entry->used ++; + remove_free_list(cache, entry); + pthread_mutex_unlock(&cache->mutex); + } else { + /* + * not in the cache + * + * first try to allocate new block + */ + if(cache->count < cache->max_buffers) { + entry = malloc(sizeof(struct cache_entry)); + if(entry == NULL) + EXIT_UNSQUASH("Out of memory in cache_get\n"); + entry->data = malloc(cache->buffer_size); + if(entry->data == NULL) + EXIT_UNSQUASH("Out of memory in cache_get\n"); + entry->cache = cache; + entry->free_prev = entry->free_next = NULL; + cache->count ++; + } else { + /* + * try to get from free list + */ + while(cache->free_list == NULL) { + cache->wait_free = TRUE; + pthread_cond_wait(&cache->wait_for_free, + &cache->mutex); + } + entry = cache->free_list; + remove_free_list(cache, entry); + remove_hash_table(cache, entry); + } + + /* + * initialise block and insert into the hash table + */ + entry->block = block; + entry->size = size; + entry->used = 1; + entry->error = FALSE; + entry->pending = TRUE; + insert_hash_table(cache, entry); + + /* + * queue to read thread to read and ultimately (via the + * decompress threads) decompress the buffer + */ + pthread_mutex_unlock(&cache->mutex); + queue_put(to_reader, entry); + } + + return entry; +} + + +void cache_block_ready(struct cache_entry *entry, int error) +{ + /* + * mark cache entry as being complete, reading and (if necessary) + * decompression has taken place, and the buffer is valid for use. + * If an error occurs reading or decompressing, the buffer also + * becomes ready but with an error... + */ + pthread_mutex_lock(&entry->cache->mutex); + entry->pending = FALSE; + entry->error = error; + + /* + * if the wait_pending flag is set, one or more threads may be waiting + * on this buffer + */ + if(entry->cache->wait_pending) { + entry->cache->wait_pending = FALSE; + pthread_cond_broadcast(&entry->cache->wait_for_pending); + } + + pthread_mutex_unlock(&entry->cache->mutex); +} + + +void cache_block_wait(struct cache_entry *entry) +{ + /* + * wait for this cache entry to become ready, when reading and (if + * necessary) decompression has taken place + */ + pthread_mutex_lock(&entry->cache->mutex); + + while(entry->pending) { + entry->cache->wait_pending = TRUE; + pthread_cond_wait(&entry->cache->wait_for_pending, + &entry->cache->mutex); + } + + pthread_mutex_unlock(&entry->cache->mutex); +} + + +void cache_block_put(struct cache_entry *entry) +{ + /* + * finished with this cache entry, once the usage count reaches zero it + * can be reused and is put onto the free list. As it remains + * accessible via the hash table it can be found getting a new lease of + * life before it is reused. + */ + pthread_mutex_lock(&entry->cache->mutex); + + entry->used --; + if(entry->used == 0) { + insert_free_list(entry->cache, entry); + + /* + * if the wait_free flag is set, one or more threads may be + * waiting on this buffer + */ + if(entry->cache->wait_free) { + entry->cache->wait_free = FALSE; + pthread_cond_broadcast(&entry->cache->wait_for_free); + } + } + + pthread_mutex_unlock(&entry->cache->mutex); +} + + +char *modestr(char *str, int mode) +{ + int i; + + strcpy(str, "----------"); + + for(i = 0; table[i].mask != 0; i++) { + if((mode & table[i].mask) == table[i].value) + str[table[i].position] = table[i].mode; + } + + return str; +} + + +#define TOTALCHARS 25 +int print_filename(char *pathname, struct inode *inode) +{ + char str[11], dummy[100], dummy2[100], *userstr, *groupstr; + int padchars; + struct passwd *user; + struct group *group; + struct tm *t; + + if(short_ls) { + printf("%s\n", pathname); + return 1; + } + + user = getpwuid(inode->uid); + if(user == NULL) { + sprintf(dummy, "%d", inode->uid); + userstr = dummy; + } else + userstr = user->pw_name; + + group = getgrgid(inode->gid); + if(group == NULL) { + sprintf(dummy2, "%d", inode->gid); + groupstr = dummy2; + } else + groupstr = group->gr_name; + + printf("%s %s/%s ", modestr(str, inode->mode), userstr, groupstr); + + switch(inode->mode & S_IFMT) { + case S_IFREG: + case S_IFDIR: + case S_IFSOCK: + case S_IFIFO: + case S_IFLNK: + padchars = TOTALCHARS - strlen(userstr) - + strlen(groupstr); + + printf("%*lld ", padchars > 0 ? padchars : 0, + inode->data); + break; + case S_IFCHR: + case S_IFBLK: + padchars = TOTALCHARS - strlen(userstr) - + strlen(groupstr) - 7; + + printf("%*s%3d,%3d ", padchars > 0 ? padchars : 0, " ", + (int) inode->data >> 8, (int) inode->data & + 0xff); + break; + } + + t = localtime(&inode->time); + + printf("%d-%02d-%02d %02d:%02d %s", t->tm_year + 1900, t->tm_mon + 1, + t->tm_mday, t->tm_hour, t->tm_min, pathname); + if((inode->mode & S_IFMT) == S_IFLNK) + printf(" -> %s", inode->symlink); + printf("\n"); + + return 1; +} + + +void add_entry(struct hash_table_entry *hash_table[], long long start, + int bytes) +{ + int hash = CALCULATE_HASH(start); + struct hash_table_entry *hash_table_entry; + + hash_table_entry = malloc(sizeof(struct hash_table_entry)); + if(hash_table_entry == NULL) + EXIT_UNSQUASH("Out of memory in add_entry\n"); + + hash_table_entry->start = start; + hash_table_entry->bytes = bytes; + hash_table_entry->next = hash_table[hash]; + hash_table[hash] = hash_table_entry; +} + + +int lookup_entry(struct hash_table_entry *hash_table[], long long start) +{ + int hash = CALCULATE_HASH(start); + struct hash_table_entry *hash_table_entry; + + for(hash_table_entry = hash_table[hash]; hash_table_entry; + hash_table_entry = hash_table_entry->next) + + if(hash_table_entry->start == start) + return hash_table_entry->bytes; + + return -1; +} + + +int read_fs_bytes(int fd, long long byte, int bytes, void *buff) +{ + off_t off = byte; + int res, count; + + TRACE("read_bytes: reading from position 0x%llx, bytes %d\n", byte, + bytes); + + if(lseek(fd, off, SEEK_SET) == -1) { + ERROR("Lseek failed because %s\n", strerror(errno)); + return FALSE; + } + + for(count = 0; count < bytes; count += res) { + res = read(fd, buff + count, bytes - count); + if(res < 1) { + if(res == 0) { + ERROR("Read on filesystem failed because " + "EOF\n"); + return FALSE; + } else if(errno != EINTR) { + ERROR("Read on filesystem failed because %s\n", + strerror(errno)); + return FALSE; + } else + res = 0; + } + } + + return TRUE; +} + + +int read_block(int fd, long long start, long long *next, void *block) +{ + unsigned short c_byte; + int offset = 2; + + if(swap) { + if(read_fs_bytes(fd, start, 2, &c_byte) == FALSE) + goto failed; + c_byte = (c_byte >> 8) | ((c_byte & 0xff) << 8); + } else + if(read_fs_bytes(fd, start, 2, &c_byte) == FALSE) + goto failed; + + TRACE("read_block: block @0x%llx, %d %s bytes\n", start, + SQUASHFS_COMPRESSED_SIZE(c_byte), SQUASHFS_COMPRESSED(c_byte) ? + "compressed" : "uncompressed"); + + if(SQUASHFS_CHECK_DATA(sBlk.s.flags)) + offset = 3; + if(SQUASHFS_COMPRESSED(c_byte)) { + char buffer[SQUASHFS_METADATA_SIZE]; + int error, res; + + c_byte = SQUASHFS_COMPRESSED_SIZE(c_byte); + if(read_fs_bytes(fd, start + offset, c_byte, buffer) == FALSE) + goto failed; + + res = compressor_uncompress(comp, block, buffer, c_byte, + SQUASHFS_METADATA_SIZE, &error); + + if(res == -1) { + ERROR("%s uncompress failed with error code %d\n", + comp->name, error); + goto failed; + } + if(next) + *next = start + offset + c_byte; + return res; + } else { + c_byte = SQUASHFS_COMPRESSED_SIZE(c_byte); + if(read_fs_bytes(fd, start + offset, c_byte, block) == FALSE) + goto failed; + if(next) + *next = start + offset + c_byte; + return c_byte; + } + +failed: + ERROR("read_block: failed to read block @0x%llx\n", start); + return FALSE; +} + + +int read_data_block(long long start, unsigned int size, char *block) +{ + int error, res; + int c_byte = SQUASHFS_COMPRESSED_SIZE_BLOCK(size); + + TRACE("read_data_block: block @0x%llx, %d %s bytes\n", start, + c_byte, SQUASHFS_COMPRESSED_BLOCK(size) ? "compressed" : + "uncompressed"); + + if(SQUASHFS_COMPRESSED_BLOCK(size)) { + if(read_fs_bytes(fd, start, c_byte, data) == FALSE) + goto failed; + + res = compressor_uncompress(comp, block, data, c_byte, + block_size, &error); + + if(res == -1) { + ERROR("%s uncompress failed with error code %d\n", + comp->name, error); + goto failed; + } + + return res; + } else { + if(read_fs_bytes(fd, start, c_byte, block) == FALSE) + goto failed; + + return c_byte; + } + +failed: + ERROR("read_data_block: failed to read block @0x%llx, size %d\n", start, + c_byte); + return FALSE; +} + + +void uncompress_inode_table(long long start, long long end) +{ + int size = 0, bytes = 0, res; + + TRACE("uncompress_inode_table: start %lld, end %lld\n", start, end); + while(start < end) { + if(size - bytes < SQUASHFS_METADATA_SIZE) { + inode_table = realloc(inode_table, size += + SQUASHFS_METADATA_SIZE); + if(inode_table == NULL) + EXIT_UNSQUASH("Out of memory in " + "uncompress_inode_table"); + } + TRACE("uncompress_inode_table: reading block 0x%llx\n", start); + add_entry(inode_table_hash, start, bytes); + res = read_block(fd, start, &start, inode_table + bytes); + if(res == 0) { + free(inode_table); + EXIT_UNSQUASH("uncompress_inode_table: failed to read " + "block \n"); + } + bytes += res; + } +} + + +int set_attributes(char *pathname, int mode, uid_t uid, gid_t guid, time_t time, + unsigned int xattr, unsigned int set_mode) +{ + struct utimbuf times = { time, time }; + + write_xattr(pathname, xattr); + + if(utime(pathname, ×) == -1) { + ERROR("set_attributes: failed to set time on %s, because %s\n", + pathname, strerror(errno)); + return FALSE; + } + + if(root_process) { + if(chown(pathname, uid, guid) == -1) { + ERROR("set_attributes: failed to change uid and gids " + "on %s, because %s\n", pathname, + strerror(errno)); + return FALSE; + } + } else + mode &= ~07000; + + if((set_mode || (mode & 07000)) && chmod(pathname, (mode_t) mode) == -1) { + ERROR("set_attributes: failed to change mode %s, because %s\n", + pathname, strerror(errno)); + return FALSE; + } + + return TRUE; +} + + +int write_bytes(int fd, char *buff, int bytes) +{ + int res, count; + + for(count = 0; count < bytes; count += res) { + res = write(fd, buff + count, bytes - count); + if(res == -1) { + if(errno != EINTR) { + ERROR("Write on output file failed because " + "%s\n", strerror(errno)); + return -1; + } + res = 0; + } + } + + return 0; +} + + +int lseek_broken = FALSE; +char *zero_data = NULL; + +int write_block(int file_fd, char *buffer, int size, long long hole, int sparse) +{ + off_t off = hole; + + if(hole) { + if(sparse && lseek_broken == FALSE) { + int error = lseek(file_fd, off, SEEK_CUR); + if(error == -1) + /* failed to seek beyond end of file */ + lseek_broken = TRUE; + } + + if((sparse == FALSE || lseek_broken) && zero_data == NULL) { + if((zero_data = malloc(block_size)) == NULL) + EXIT_UNSQUASH("write_block: failed to alloc " + "zero data block\n"); + memset(zero_data, 0, block_size); + } + + if(sparse == FALSE || lseek_broken) { + int blocks = (hole + block_size -1) / block_size; + int avail_bytes, i; + for(i = 0; i < blocks; i++, hole -= avail_bytes) { + avail_bytes = hole > block_size ? block_size : + hole; + if(write_bytes(file_fd, zero_data, avail_bytes) + == -1) + goto failure; + } + } + } + + if(write_bytes(file_fd, buffer, size) == -1) + goto failure; + + return TRUE; + +failure: + return FALSE; +} + + +int write_file(struct inode *inode, char *pathname) +{ + unsigned int file_fd, i; + unsigned int *block_list; + int file_end = inode->data / block_size; + long long start = inode->start; + struct squashfs_file *file; + + TRACE("write_file: regular file, blocks %d\n", inode->blocks); + + file_fd = open(pathname, O_CREAT | O_WRONLY | (force ? O_TRUNC : 0), + (mode_t) inode->mode & 0777); + if(file_fd == -1) { + ERROR("write_file: failed to create file %s, because %s\n", + pathname, strerror(errno)); + return FALSE; + } + + block_list = malloc(inode->blocks * sizeof(unsigned int)); + if(block_list == NULL) + EXIT_UNSQUASH("write_file: unable to malloc block list\n"); + + s_ops.read_block_list(block_list, inode->block_ptr, inode->blocks); + + file = malloc(sizeof(struct squashfs_file)); + if(file == NULL) + EXIT_UNSQUASH("write_file: unable to malloc file\n"); + + /* + * the writer thread is queued a squashfs_file structure describing the + * file. If the file has one or more blocks or a fragments they are + * queued separately (references to blocks in the cache). + */ + file->fd = file_fd; + file->file_size = inode->data; + file->mode = inode->mode; + file->gid = inode->gid; + file->uid = inode->uid; + file->time = inode->time; + file->pathname = strdup(pathname); + file->blocks = inode->blocks + (inode->frag_bytes > 0); + file->sparse = inode->sparse; + file->xattr = inode->xattr; + queue_put(to_writer, file); + + for(i = 0; i < inode->blocks; i++) { + int c_byte = SQUASHFS_COMPRESSED_SIZE_BLOCK(block_list[i]); + struct file_entry *block = malloc(sizeof(struct file_entry)); + + if(block == NULL) + EXIT_UNSQUASH("write_file: unable to malloc file\n"); + block->offset = 0; + block->size = i == file_end ? inode->data & (block_size - 1) : + block_size; + if(block_list[i] == 0) /* sparse file */ + block->buffer = NULL; + else { + block->buffer = cache_get(data_cache, start, + block_list[i]); + start += c_byte; + } + queue_put(to_writer, block); + } + + if(inode->frag_bytes) { + int size; + long long start; + struct file_entry *block = malloc(sizeof(struct file_entry)); + + if(block == NULL) + EXIT_UNSQUASH("write_file: unable to malloc file\n"); + s_ops.read_fragment(inode->fragment, &start, &size); + block->buffer = cache_get(fragment_cache, start, size); + block->offset = inode->offset; + block->size = inode->frag_bytes; + queue_put(to_writer, block); + } + + free(block_list); + return TRUE; +} + + +int create_inode(char *pathname, struct inode *i) +{ + TRACE("create_inode: pathname %s\n", pathname); + + if(created_inode[i->inode_number - 1]) { + TRACE("create_inode: hard link\n"); + if(force) + unlink(pathname); + + if(link(created_inode[i->inode_number - 1], pathname) == -1) { + ERROR("create_inode: failed to create hardlink, " + "because %s\n", strerror(errno)); + return FALSE; + } + + return TRUE; + } + + switch(i->type) { + case SQUASHFS_FILE_TYPE: + case SQUASHFS_LREG_TYPE: + TRACE("create_inode: regular file, file_size %lld, " + "blocks %d\n", i->data, i->blocks); + + if(write_file(i, pathname)) + file_count ++; + break; + case SQUASHFS_SYMLINK_TYPE: + case SQUASHFS_LSYMLINK_TYPE: + TRACE("create_inode: symlink, symlink_size %lld\n", + i->data); + + if(force) + unlink(pathname); + + if(symlink(i->symlink, pathname) == -1) { + ERROR("create_inode: failed to create symlink " + "%s, because %s\n", pathname, + strerror(errno)); + break; + } + + write_xattr(pathname, i->xattr); + + if(root_process) { + if(lchown(pathname, i->uid, i->gid) == -1) + ERROR("create_inode: failed to change " + "uid and gids on %s, because " + "%s\n", pathname, + strerror(errno)); + } + + sym_count ++; + break; + case SQUASHFS_BLKDEV_TYPE: + case SQUASHFS_CHRDEV_TYPE: + case SQUASHFS_LBLKDEV_TYPE: + case SQUASHFS_LCHRDEV_TYPE: { + int chrdev = i->type == SQUASHFS_CHRDEV_TYPE; + TRACE("create_inode: dev, rdev 0x%llx\n", i->data); + + if(root_process) { + if(force) + unlink(pathname); + + if(mknod(pathname, chrdev ? S_IFCHR : S_IFBLK, + makedev((i->data >> 8) & 0xff, + i->data & 0xff)) == -1) { + ERROR("create_inode: failed to create " + "%s device %s, because %s\n", + chrdev ? "character" : "block", + pathname, strerror(errno)); + break; + } + set_attributes(pathname, i->mode, i->uid, + i->gid, i->time, i->xattr, TRUE); + dev_count ++; + } else + ERROR("create_inode: could not create %s " + "device %s, because you're not " + "superuser!\n", chrdev ? "character" : + "block", pathname); + break; + } + case SQUASHFS_FIFO_TYPE: + case SQUASHFS_LFIFO_TYPE: + TRACE("create_inode: fifo\n"); + + if(force) + unlink(pathname); + + if(mknod(pathname, S_IFIFO, 0) == -1) { + ERROR("create_inode: failed to create fifo %s, " + "because %s\n", pathname, + strerror(errno)); + break; + } + set_attributes(pathname, i->mode, i->uid, i->gid, + i->time, i->xattr, TRUE); + fifo_count ++; + break; + case SQUASHFS_SOCKET_TYPE: + case SQUASHFS_LSOCKET_TYPE: + TRACE("create_inode: socket\n"); + ERROR("create_inode: socket %s ignored\n", pathname); + break; + default: + ERROR("Unknown inode type %d in create_inode_table!\n", + i->type); + return FALSE; + } + + created_inode[i->inode_number - 1] = strdup(pathname); + + return TRUE; +} + + +void uncompress_directory_table(long long start, long long end) +{ + int bytes = 0, size = 0, res; + + TRACE("uncompress_directory_table: start %lld, end %lld\n", start, end); + + while(start < end) { + if(size - bytes < SQUASHFS_METADATA_SIZE) { + directory_table = realloc(directory_table, size += + SQUASHFS_METADATA_SIZE); + if(directory_table == NULL) + EXIT_UNSQUASH("Out of memory in " + "uncompress_directory_table\n"); + } + TRACE("uncompress_directory_table: reading block 0x%llx\n", + start); + add_entry(directory_table_hash, start, bytes); + res = read_block(fd, start, &start, directory_table + bytes); + if(res == 0) + EXIT_UNSQUASH("uncompress_directory_table: failed to " + "read block\n"); + bytes += res; + } +} + + +int squashfs_readdir(struct dir *dir, char **name, unsigned int *start_block, +unsigned int *offset, unsigned int *type) +{ + if(dir->cur_entry == dir->dir_count) + return FALSE; + + *name = dir->dirs[dir->cur_entry].name; + *start_block = dir->dirs[dir->cur_entry].start_block; + *offset = dir->dirs[dir->cur_entry].offset; + *type = dir->dirs[dir->cur_entry].type; + dir->cur_entry ++; + + return TRUE; +} + + +void squashfs_closedir(struct dir *dir) +{ + free(dir->dirs); + free(dir); +} + + +char *get_component(char *target, char *targname) +{ + while(*target == '/') + target ++; + + while(*target != '/' && *target!= '\0') + *targname ++ = *target ++; + + *targname = '\0'; + + return target; +} + + +void free_path(struct pathname *paths) +{ + int i; + + for(i = 0; i < paths->names; i++) { + if(paths->name[i].paths) + free_path(paths->name[i].paths); + free(paths->name[i].name); + if(paths->name[i].preg) { + regfree(paths->name[i].preg); + free(paths->name[i].preg); + } + } + + free(paths); +} + + +struct pathname *add_path(struct pathname *paths, char *target, char *alltarget) +{ + char targname[1024]; + int i, error; + + TRACE("add_path: adding \"%s\" extract file\n", target); + + target = get_component(target, targname); + + if(paths == NULL) { + paths = malloc(sizeof(struct pathname)); + if(paths == NULL) + EXIT_UNSQUASH("failed to allocate paths\n"); + + paths->names = 0; + paths->name = NULL; + } + + for(i = 0; i < paths->names; i++) + if(strcmp(paths->name[i].name, targname) == 0) + break; + + if(i == paths->names) { + /* + * allocate new name entry + */ + paths->names ++; + paths->name = realloc(paths->name, (i + 1) * + sizeof(struct path_entry)); + if(paths->name == NULL) + EXIT_UNSQUASH("Out of memory in add_path\n"); + paths->name[i].name = strdup(targname); + paths->name[i].paths = NULL; + if(use_regex) { + paths->name[i].preg = malloc(sizeof(regex_t)); + if(paths->name[i].preg == NULL) + EXIT_UNSQUASH("Out of memory in add_path\n"); + error = regcomp(paths->name[i].preg, targname, + REG_EXTENDED|REG_NOSUB); + if(error) { + char str[1024]; + + regerror(error, paths->name[i].preg, str, 1024); + EXIT_UNSQUASH("invalid regex %s in export %s, " + "because %s\n", targname, alltarget, + str); + } + } else + paths->name[i].preg = NULL; + + if(target[0] == '\0') + /* + * at leaf pathname component + */ + paths->name[i].paths = NULL; + else + /* + * recurse adding child components + */ + paths->name[i].paths = add_path(NULL, target, alltarget); + } else { + /* + * existing matching entry + */ + if(paths->name[i].paths == NULL) { + /* + * No sub-directory which means this is the leaf + * component of a pre-existing extract which subsumes + * the extract currently being added, in which case stop + * adding components + */ + } else if(target[0] == '\0') { + /* + * at leaf pathname component and child components exist + * from more specific extracts, delete as they're + * subsumed by this extract + */ + free_path(paths->name[i].paths); + paths->name[i].paths = NULL; + } else + /* + * recurse adding child components + */ + add_path(paths->name[i].paths, target, alltarget); + } + + return paths; +} + + +struct pathnames *init_subdir() +{ + struct pathnames *new = malloc(sizeof(struct pathnames)); + if(new == NULL) + EXIT_UNSQUASH("Out of memory in init_subdir\n"); + new->count = 0; + return new; +} + + +struct pathnames *add_subdir(struct pathnames *paths, struct pathname *path) +{ + if(paths->count % PATHS_ALLOC_SIZE == 0) { + paths = realloc(paths, sizeof(struct pathnames *) + + (paths->count + PATHS_ALLOC_SIZE) * + sizeof(struct pathname *)); + if(paths == NULL) + EXIT_UNSQUASH("Out of memory in add_subdir\n"); + } + + paths->path[paths->count++] = path; + return paths; +} + + +void free_subdir(struct pathnames *paths) +{ + free(paths); +} + + +int matches(struct pathnames *paths, char *name, struct pathnames **new) +{ + int i, n; + + if(paths == NULL) { + *new = NULL; + return TRUE; + } + + *new = init_subdir(); + + for(n = 0; n < paths->count; n++) { + struct pathname *path = paths->path[n]; + for(i = 0; i < path->names; i++) { + int match = use_regex ? + regexec(path->name[i].preg, name, (size_t) 0, + NULL, 0) == 0 : fnmatch(path->name[i].name, + name, FNM_PATHNAME|FNM_PERIOD|FNM_EXTMATCH) == + 0; + if(match && path->name[i].paths == NULL) + /* + * match on a leaf component, any subdirectories + * will implicitly match, therefore return an + * empty new search set + */ + goto empty_set; + + if(match) + /* + * match on a non-leaf component, add any + * subdirectories to the new set of + * subdirectories to scan for this name + */ + *new = add_subdir(*new, path->name[i].paths); + } + } + + if((*new)->count == 0) { + /* + * no matching names found, delete empty search set, and return + * FALSE + */ + free_subdir(*new); + *new = NULL; + return FALSE; + } + + /* + * one or more matches with sub-directories found (no leaf matches), + * return new search set and return TRUE + */ + return TRUE; + +empty_set: + /* + * found matching leaf exclude, return empty search set and return TRUE + */ + free_subdir(*new); + *new = NULL; + return TRUE; +} + + +void pre_scan(char *parent_name, unsigned int start_block, unsigned int offset, + struct pathnames *paths) +{ + unsigned int type; + char *name, pathname[1024]; + struct pathnames *new; + struct inode *i; + struct dir *dir = s_ops.squashfs_opendir(start_block, offset, &i); + + while(squashfs_readdir(dir, &name, &start_block, &offset, &type)) { + struct inode *i; + + TRACE("pre_scan: name %s, start_block %d, offset %d, type %d\n", + name, start_block, offset, type); + + if(!matches(paths, name, &new)) + continue; + + strcat(strcat(strcpy(pathname, parent_name), "/"), name); + + if(type == SQUASHFS_DIR_TYPE) + pre_scan(parent_name, start_block, offset, new); + else if(new == NULL) { + if(type == SQUASHFS_FILE_TYPE || + type == SQUASHFS_LREG_TYPE) { + i = s_ops.read_inode(start_block, offset); + if(created_inode[i->inode_number - 1] == NULL) { + created_inode[i->inode_number - 1] = + (char *) i; + total_blocks += (i->data + + (block_size - 1)) >> block_log; + } + total_files ++; + } + total_inodes ++; + } + + free_subdir(new); + } + + squashfs_closedir(dir); +} + + +void dir_scan(char *parent_name, unsigned int start_block, unsigned int offset, + struct pathnames *paths) +{ + unsigned int type; + char *name, pathname[1024]; + struct pathnames *new; + struct inode *i; + struct dir *dir = s_ops.squashfs_opendir(start_block, offset, &i); + + if(lsonly || info) + print_filename(parent_name, i); + + if(!lsonly && mkdir(parent_name, (mode_t) dir->mode) == -1 && + (!force || errno != EEXIST)) { + ERROR("dir_scan: failed to make directory %s, because %s\n", + parent_name, strerror(errno)); + squashfs_closedir(dir); + return; + } + + while(squashfs_readdir(dir, &name, &start_block, &offset, &type)) { + TRACE("dir_scan: name %s, start_block %d, offset %d, type %d\n", + name, start_block, offset, type); + + + if(!matches(paths, name, &new)) + continue; + + strcat(strcat(strcpy(pathname, parent_name), "/"), name); + + if(type == SQUASHFS_DIR_TYPE) + dir_scan(pathname, start_block, offset, new); + else if(new == NULL) { + i = s_ops.read_inode(start_block, offset); + + if(lsonly || info) + print_filename(pathname, i); + + if(!lsonly) { + create_inode(pathname, i); + update_progress_bar(); + } + + if(i->type == SQUASHFS_SYMLINK_TYPE || + i->type == SQUASHFS_LSYMLINK_TYPE) + free(i->symlink); + } + + free_subdir(new); + } + + if(!lsonly) + set_attributes(parent_name, dir->mode, dir->uid, dir->guid, + dir->mtime, dir->xattr, force); + + squashfs_closedir(dir); + dir_count ++; +} + + +void squashfs_stat(char *source) +{ + time_t mkfs_time = (time_t) sBlk.s.mkfs_time; + char *mkfs_str = ctime(&mkfs_time); + +#if __BYTE_ORDER == __BIG_ENDIAN + printf("Found a valid %sSQUASHFS %d:%d superblock on %s.\n", + sBlk.s.s_major == 4 ? "" : swap ? "little endian " : + "big endian ", sBlk.s.s_major, sBlk.s.s_minor, source); +#else + printf("Found a valid %sSQUASHFS %d:%d superblock on %s.\n", + sBlk.s.s_major == 4 ? "" : swap ? "big endian " : + "little endian ", sBlk.s.s_major, sBlk.s.s_minor, source); +#endif + + printf("Creation or last append time %s", mkfs_str ? mkfs_str : + "failed to get time\n"); + printf("Filesystem size %.2f Kbytes (%.2f Mbytes)\n", + sBlk.s.bytes_used / 1024.0, sBlk.s.bytes_used / + (1024.0 * 1024.0)); + + if(sBlk.s.s_major == 4) + printf("Compression %s\n", comp->name); + + printf("Block size %d\n", sBlk.s.block_size); + printf("Filesystem is %sexportable via NFS\n", + SQUASHFS_EXPORTABLE(sBlk.s.flags) ? "" : "not "); + printf("Inodes are %scompressed\n", + SQUASHFS_UNCOMPRESSED_INODES(sBlk.s.flags) ? "un" : ""); + printf("Data is %scompressed\n", + SQUASHFS_UNCOMPRESSED_DATA(sBlk.s.flags) ? "un" : ""); + + if(sBlk.s.s_major > 1) { + if(SQUASHFS_NO_FRAGMENTS(sBlk.s.flags)) + printf("Fragments are not stored\n"); + else { + printf("Fragments are %scompressed\n", + SQUASHFS_UNCOMPRESSED_FRAGMENTS(sBlk.s.flags) ? + "un" : ""); + printf("Always_use_fragments option is %sspecified\n", + SQUASHFS_ALWAYS_FRAGMENTS(sBlk.s.flags) ? "" : + "not "); + } + } + + if(sBlk.s.s_major == 4) { + if(SQUASHFS_NO_XATTRS(sBlk.s.flags)) + printf("Xattrs are not stored\n"); + else + printf("Xattrs are %scompressed\n", + SQUASHFS_UNCOMPRESSED_XATTRS(sBlk.s.flags) ? + "un" : ""); + } + + if(sBlk.s.s_major < 4) + printf("Check data is %spresent in the filesystem\n", + SQUASHFS_CHECK_DATA(sBlk.s.flags) ? "" : + "not "); + + if(sBlk.s.s_major > 1) + printf("Duplicates are %sremoved\n", + SQUASHFS_DUPLICATES(sBlk.s.flags) ? "" : "not "); + else + printf("Duplicates are removed\n"); + + if(sBlk.s.s_major > 1) + printf("Number of fragments %d\n", sBlk.s.fragments); + + printf("Number of inodes %d\n", sBlk.s.inodes); + + if(sBlk.s.s_major == 4) + printf("Number of ids %d\n", sBlk.s.no_ids); + else { + printf("Number of uids %d\n", sBlk.no_uids); + printf("Number of gids %d\n", sBlk.no_guids); + } + + TRACE("sBlk.s.inode_table_start 0x%llx\n", sBlk.s.inode_table_start); + TRACE("sBlk.s.directory_table_start 0x%llx\n", + sBlk.s.directory_table_start); + + if(sBlk.s.s_major == 4) { + TRACE("sBlk.s.id_table_start 0x%llx\n", sBlk.s.id_table_start); + TRACE("sBlk.s.xattr_id_table_start 0x%llx\n", + sBlk.s.xattr_id_table_start); + } else { + TRACE("sBlk.uid_start 0x%llx\n", sBlk.uid_start); + TRACE("sBlk.guid_start 0x%llx\n", sBlk.guid_start); + } + + if(sBlk.s.s_major > 1) + TRACE("sBlk.s.fragment_table_start 0x%llx\n\n", + sBlk.s.fragment_table_start); +} + + +int read_super(char *source) +{ + squashfs_super_block_3 sBlk_3; + struct squashfs_super_block sBlk_4; + + /* + * Try to read a Squashfs 4 superblock + */ + read_fs_bytes(fd, SQUASHFS_START, sizeof(struct squashfs_super_block), + &sBlk_4); + swap = sBlk_4.s_magic != SQUASHFS_MAGIC; + SQUASHFS_INSWAP_SUPER_BLOCK(&sBlk_4); + + if(sBlk_4.s_magic == SQUASHFS_MAGIC && sBlk_4.s_major == 4 && + sBlk_4.s_minor == 0) { + s_ops.squashfs_opendir = squashfs_opendir_4; + s_ops.read_fragment = read_fragment_4; + s_ops.read_fragment_table = read_fragment_table_4; + s_ops.read_block_list = read_block_list_2; + s_ops.read_inode = read_inode_4; + s_ops.read_uids_guids = read_uids_guids_4; + memcpy(&sBlk, &sBlk_4, sizeof(sBlk_4)); + + /* + * Check the compression type + */ + comp = lookup_compressor_id(sBlk.s.compression); + return TRUE; + } + + /* + * Not a Squashfs 4 superblock, try to read a squashfs 3 superblock + * (compatible with 1 and 2 filesystems) + */ + read_fs_bytes(fd, SQUASHFS_START, sizeof(squashfs_super_block_3), + &sBlk_3); + + /* + * Check it is a SQUASHFS superblock + */ + swap = 0; + if(sBlk_3.s_magic != SQUASHFS_MAGIC) { + if(sBlk_3.s_magic == SQUASHFS_MAGIC_SWAP) { + squashfs_super_block_3 sblk; + ERROR("Reading a different endian SQUASHFS filesystem " + "on %s\n", source); + SQUASHFS_SWAP_SUPER_BLOCK_3(&sblk, &sBlk_3); + memcpy(&sBlk_3, &sblk, sizeof(squashfs_super_block_3)); + swap = 1; + } else { + ERROR("Can't find a SQUASHFS superblock on %s\n", + source); + goto failed_mount; + } + } + + sBlk.s.s_magic = sBlk_3.s_magic; + sBlk.s.inodes = sBlk_3.inodes; + sBlk.s.mkfs_time = sBlk_3.mkfs_time; + sBlk.s.block_size = sBlk_3.block_size; + sBlk.s.fragments = sBlk_3.fragments; + sBlk.s.block_log = sBlk_3.block_log; + sBlk.s.flags = sBlk_3.flags; + sBlk.s.s_major = sBlk_3.s_major; + sBlk.s.s_minor = sBlk_3.s_minor; + sBlk.s.root_inode = sBlk_3.root_inode; + sBlk.s.bytes_used = sBlk_3.bytes_used; + sBlk.s.inode_table_start = sBlk_3.inode_table_start; + sBlk.s.directory_table_start = sBlk_3.directory_table_start; + sBlk.s.fragment_table_start = sBlk_3.fragment_table_start; + sBlk.s.lookup_table_start = sBlk_3.lookup_table_start; + sBlk.no_uids = sBlk_3.no_uids; + sBlk.no_guids = sBlk_3.no_guids; + sBlk.uid_start = sBlk_3.uid_start; + sBlk.guid_start = sBlk_3.guid_start; + sBlk.s.xattr_id_table_start = SQUASHFS_INVALID_BLK; + + /* Check the MAJOR & MINOR versions */ + if(sBlk.s.s_major == 1 || sBlk.s.s_major == 2) { + sBlk.s.bytes_used = sBlk_3.bytes_used_2; + sBlk.uid_start = sBlk_3.uid_start_2; + sBlk.guid_start = sBlk_3.guid_start_2; + sBlk.s.inode_table_start = sBlk_3.inode_table_start_2; + sBlk.s.directory_table_start = sBlk_3.directory_table_start_2; + + if(sBlk.s.s_major == 1) { + sBlk.s.block_size = sBlk_3.block_size_1; + sBlk.s.fragment_table_start = sBlk.uid_start; + s_ops.squashfs_opendir = squashfs_opendir_1; + s_ops.read_fragment_table = read_fragment_table_1; + s_ops.read_block_list = read_block_list_1; + s_ops.read_inode = read_inode_1; + s_ops.read_uids_guids = read_uids_guids_1; + } else { + sBlk.s.fragment_table_start = + sBlk_3.fragment_table_start_2; + s_ops.squashfs_opendir = squashfs_opendir_1; + s_ops.read_fragment = read_fragment_2; + s_ops.read_fragment_table = read_fragment_table_2; + s_ops.read_block_list = read_block_list_2; + s_ops.read_inode = read_inode_2; + s_ops.read_uids_guids = read_uids_guids_1; + } + } else if(sBlk.s.s_major == 3) { + s_ops.squashfs_opendir = squashfs_opendir_3; + s_ops.read_fragment = read_fragment_3; + s_ops.read_fragment_table = read_fragment_table_3; + s_ops.read_block_list = read_block_list_2; + s_ops.read_inode = read_inode_3; + s_ops.read_uids_guids = read_uids_guids_1; + } else { + ERROR("Filesystem on %s is (%d:%d), ", source, sBlk.s.s_major, + sBlk.s.s_minor); + ERROR("which is a later filesystem version than I support!\n"); + goto failed_mount; + } + + /* + * 1.x, 2.x and 3.x filesystems use gzip compression. + */ + comp = lookup_compressor("gzip"); + return TRUE; + +failed_mount: + return FALSE; +} + + +struct pathname *process_extract_files(struct pathname *path, char *filename) +{ + FILE *fd; + char name[16384]; + + fd = fopen(filename, "r"); + if(fd == NULL) + EXIT_UNSQUASH("Could not open %s, because %s\n", filename, + strerror(errno)); + + while(fscanf(fd, "%16384[^\n]\n", name) != EOF) + path = add_path(path, name, name); + + fclose(fd); + return path; +} + + +/* + * reader thread. This thread processes read requests queued by the + * cache_get() routine. + */ +void *reader(void *arg) +{ + while(1) { + struct cache_entry *entry = queue_get(to_reader); + int res = read_fs_bytes(fd, entry->block, + SQUASHFS_COMPRESSED_SIZE_BLOCK(entry->size), + entry->data); + + if(res && SQUASHFS_COMPRESSED_BLOCK(entry->size)) + /* + * queue successfully read block to the deflate + * thread(s) for further processing + */ + queue_put(to_deflate, entry); + else + /* + * block has either been successfully read and is + * uncompressed, or an error has occurred, clear pending + * flag, set error appropriately, and wake up any + * threads waiting on this buffer + */ + cache_block_ready(entry, !res); + } +} + + +/* + * writer thread. This processes file write requests queued by the + * write_file() routine. + */ +void *writer(void *arg) +{ + int i; + + while(1) { + struct squashfs_file *file = queue_get(to_writer); + int file_fd; + long long hole = 0; + int failed = FALSE; + int error; + + if(file == NULL) { + queue_put(from_writer, NULL); + continue; + } + + TRACE("writer: regular file, blocks %d\n", file->blocks); + + file_fd = file->fd; + + for(i = 0; i < file->blocks; i++, cur_blocks ++) { + struct file_entry *block = queue_get(to_writer); + + if(block->buffer == 0) { /* sparse file */ + hole += block->size; + free(block); + continue; + } + + cache_block_wait(block->buffer); + + if(block->buffer->error) + failed = TRUE; + + if(failed) + continue; + + error = write_block(file_fd, block->buffer->data + + block->offset, block->size, hole, file->sparse); + + if(error == FALSE) { + ERROR("writer: failed to write data block %d\n", + i); + failed = TRUE; + } + + hole = 0; + cache_block_put(block->buffer); + free(block); + } + + if(hole && failed == FALSE) { + /* + * corner case for hole extending to end of file + */ + if(file->sparse == FALSE || + lseek(file_fd, hole, SEEK_CUR) == -1) { + /* + * for files which we don't want to write + * sparsely, or for broken lseeks which cannot + * seek beyond end of file, write_block will do + * the right thing + */ + hole --; + if(write_block(file_fd, "\0", 1, hole, + file->sparse) == FALSE) { + ERROR("writer: failed to write sparse " + "data block\n"); + failed = TRUE; + } + } else if(ftruncate(file_fd, file->file_size) == -1) { + ERROR("writer: failed to write sparse data " + "block\n"); + failed = TRUE; + } + } + + close(file_fd); + if(failed == FALSE) + set_attributes(file->pathname, file->mode, file->uid, + file->gid, file->time, file->xattr, force); + else { + ERROR("Failed to write %s, skipping\n", file->pathname); + unlink(file->pathname); + } + free(file->pathname); + free(file); + + } +} + + +/* + * decompress thread. This decompresses buffers queued by the read thread + */ +void *deflator(void *arg) +{ + char tmp[block_size]; + + while(1) { + struct cache_entry *entry = queue_get(to_deflate); + int error, res; + + res = compressor_uncompress(comp, tmp, entry->data, + SQUASHFS_COMPRESSED_SIZE_BLOCK(entry->size), block_size, + &error); + + if(res == -1) + ERROR("%s uncompress failed with error code %d\n", + comp->name, error); + else + memcpy(entry->data, tmp, res); + + /* + * block has been either successfully decompressed, or an error + * occurred, clear pending flag, set error appropriately and + * wake up any threads waiting on this block + */ + cache_block_ready(entry, res == -1); + } +} + + +void *progress_thread(void *arg) +{ + struct timeval timeval; + struct timespec timespec; + struct itimerval itimerval; + struct winsize winsize; + + if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { + if(isatty(STDOUT_FILENO)) + ERROR("TIOCGWINSZ ioctl failed, defaulting to 80 " + "columns\n"); + columns = 80; + } else + columns = winsize.ws_col; + signal(SIGWINCH, sigwinch_handler); + signal(SIGALRM, sigalrm_handler); + + itimerval.it_value.tv_sec = 0; + itimerval.it_value.tv_usec = 250000; + itimerval.it_interval.tv_sec = 0; + itimerval.it_interval.tv_usec = 250000; + setitimer(ITIMER_REAL, &itimerval, NULL); + + pthread_cond_init(&progress_wait, NULL); + + pthread_mutex_lock(&screen_mutex); + while(1) { + gettimeofday(&timeval, NULL); + timespec.tv_sec = timeval.tv_sec; + if(timeval.tv_usec + 250000 > 999999) + timespec.tv_sec++; + timespec.tv_nsec = ((timeval.tv_usec + 250000) % 1000000) * + 1000; + pthread_cond_timedwait(&progress_wait, &screen_mutex, + ×pec); + if(progress_enabled) + progress_bar(sym_count + dev_count + + fifo_count + cur_blocks, total_inodes - + total_files + total_blocks, columns); + } +} + + +void initialise_threads(int fragment_buffer_size, int data_buffer_size) +{ + int i; + sigset_t sigmask, old_mask; + int all_buffers_size = fragment_buffer_size + data_buffer_size; + + sigemptyset(&sigmask); + sigaddset(&sigmask, SIGINT); + sigaddset(&sigmask, SIGQUIT); + if(sigprocmask(SIG_BLOCK, &sigmask, &old_mask) == -1) + EXIT_UNSQUASH("Failed to set signal mask in intialise_threads" + "\n"); + + if(processors == -1) { +#ifndef linux + int mib[2]; + size_t len = sizeof(processors); + + mib[0] = CTL_HW; +#ifdef HW_AVAILCPU + mib[1] = HW_AVAILCPU; +#else + mib[1] = HW_NCPU; +#endif + + if(sysctl(mib, 2, &processors, &len, NULL, 0) == -1) { + ERROR("Failed to get number of available processors. " + "Defaulting to 1\n"); + processors = 1; + } +#else + processors = sysconf(_SC_NPROCESSORS_ONLN); +#endif + } + + thread = malloc((3 + processors) * sizeof(pthread_t)); + if(thread == NULL) + EXIT_UNSQUASH("Out of memory allocating thread descriptors\n"); + deflator_thread = &thread[3]; + + to_reader = queue_init(all_buffers_size); + to_deflate = queue_init(all_buffers_size); + to_writer = queue_init(1000); + from_writer = queue_init(1); + fragment_cache = cache_init(block_size, fragment_buffer_size); + data_cache = cache_init(block_size, data_buffer_size); + pthread_create(&thread[0], NULL, reader, NULL); + pthread_create(&thread[1], NULL, writer, NULL); + pthread_create(&thread[2], NULL, progress_thread, NULL); + pthread_mutex_init(&fragment_mutex, NULL); + + for(i = 0; i < processors; i++) { + if(pthread_create(&deflator_thread[i], NULL, deflator, NULL) != + 0) + EXIT_UNSQUASH("Failed to create thread\n"); + } + + printf("Parallel unsquashfs: Using %d processor%s\n", processors, + processors == 1 ? "" : "s"); + + if(sigprocmask(SIG_SETMASK, &old_mask, NULL) == -1) + EXIT_UNSQUASH("Failed to set signal mask in intialise_threads" + "\n"); +} + + +void enable_progress_bar() +{ + pthread_mutex_lock(&screen_mutex); + progress_enabled = TRUE; + pthread_mutex_unlock(&screen_mutex); +} + + +void disable_progress_bar() +{ + pthread_mutex_lock(&screen_mutex); + progress_enabled = FALSE; + pthread_mutex_unlock(&screen_mutex); +} + + +void update_progress_bar() +{ + pthread_mutex_lock(&screen_mutex); + pthread_cond_signal(&progress_wait); + pthread_mutex_unlock(&screen_mutex); +} + + +void progress_bar(long long current, long long max, int columns) +{ + char rotate_list[] = { '|', '/', '-', '\\' }; + int max_digits, used, hashes, spaces; + static int tty = -1; + + if(max == 0) + return; + + max_digits = floor(log10(max)) + 1; + used = max_digits * 2 + 11; + hashes = (current * (columns - used)) / max; + spaces = columns - used - hashes; + + if((current > max) || (columns - used < 0)) + return; + + if(tty == -1) + tty = isatty(STDOUT_FILENO); + if(!tty) { + static long long previous = -1; + + /* + * Updating much more frequently than this results in huge + * log files. + */ + if((current % 100) != 0 && current != max) + return; + /* Don't update just to rotate the spinner. */ + if(current == previous) + return; + previous = current; + } + + printf("\r["); + + while (hashes --) + putchar('='); + + putchar(rotate_list[rotate]); + + while(spaces --) + putchar(' '); + + printf("] %*lld/%*lld", max_digits, current, max_digits, max); + printf(" %3lld%%", current * 100 / max); + fflush(stdout); +} + + +#define VERSION() \ + printf("unsquashfs version 4.2 (2011/02/28)\n");\ + printf("copyright (C) 2011 Phillip Lougher "\ + "\n\n");\ + printf("This program is free software; you can redistribute it and/or"\ + "\n");\ + printf("modify it under the terms of the GNU General Public License"\ + "\n");\ + printf("as published by the Free Software Foundation; either version "\ + "2,\n");\ + printf("or (at your option) any later version.\n\n");\ + printf("This program is distributed in the hope that it will be "\ + "useful,\n");\ + printf("but WITHOUT ANY WARRANTY; without even the implied warranty of"\ + "\n");\ + printf("MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the"\ + "\n");\ + printf("GNU General Public License for more details.\n"); +int main(int argc, char *argv[]) +{ + char *dest = "squashfs-root"; + int i, stat_sys = FALSE, version = FALSE; + int n; + struct pathnames *paths = NULL; + struct pathname *path = NULL; + int fragment_buffer_size = FRAGMENT_BUFFER_DEFAULT; + int data_buffer_size = DATA_BUFFER_DEFAULT; + char *b; + + pthread_mutex_init(&screen_mutex, NULL); + root_process = geteuid() == 0; + if(root_process) + umask(0); + + for(i = 1; i < argc; i++) { + if(*argv[i] != '-') + break; + if(strcmp(argv[i], "-version") == 0 || + strcmp(argv[i], "-v") == 0) { + VERSION(); + version = TRUE; + } else if(strcmp(argv[i], "-info") == 0 || + strcmp(argv[i], "-i") == 0) + info = TRUE; + else if(strcmp(argv[i], "-ls") == 0 || + strcmp(argv[i], "-l") == 0) + lsonly = TRUE; + else if(strcmp(argv[i], "-no-progress") == 0 || + strcmp(argv[i], "-n") == 0) + progress = FALSE; + else if(strcmp(argv[i], "-no-xattrs") == 0 || + strcmp(argv[i], "-no") == 0) + no_xattrs = TRUE; + else if(strcmp(argv[i], "-xattrs") == 0 || + strcmp(argv[i], "-x") == 0) + no_xattrs = FALSE; + else if(strcmp(argv[i], "-dest") == 0 || + strcmp(argv[i], "-d") == 0) { + if(++i == argc) { + fprintf(stderr, "%s: -dest missing filename\n", + argv[0]); + exit(1); + } + dest = argv[i]; + } else if(strcmp(argv[i], "-processors") == 0 || + strcmp(argv[i], "-p") == 0) { + if((++i == argc) || + (processors = strtol(argv[i], &b, 10), + *b != '\0')) { + ERROR("%s: -processors missing or invalid " + "processor number\n", argv[0]); + exit(1); + } + if(processors < 1) { + ERROR("%s: -processors should be 1 or larger\n", + argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-data-queue") == 0 || + strcmp(argv[i], "-da") == 0) { + if((++i == argc) || + (data_buffer_size = strtol(argv[i], &b, + 10), *b != '\0')) { + ERROR("%s: -data-queue missing or invalid " + "queue size\n", argv[0]); + exit(1); + } + if(data_buffer_size < 1) { + ERROR("%s: -data-queue should be 1 Mbyte or " + "larger\n", argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-frag-queue") == 0 || + strcmp(argv[i], "-fr") == 0) { + if((++i == argc) || + (fragment_buffer_size = strtol(argv[i], + &b, 10), *b != '\0')) { + ERROR("%s: -frag-queue missing or invalid " + "queue size\n", argv[0]); + exit(1); + } + if(fragment_buffer_size < 1) { + ERROR("%s: -frag-queue should be 1 Mbyte or " + "larger\n", argv[0]); + exit(1); + } + } else if(strcmp(argv[i], "-force") == 0 || + strcmp(argv[i], "-f") == 0) + force = TRUE; + else if(strcmp(argv[i], "-stat") == 0 || + strcmp(argv[i], "-s") == 0) + stat_sys = TRUE; + else if(strcmp(argv[i], "-lls") == 0 || + strcmp(argv[i], "-ll") == 0) { + lsonly = TRUE; + short_ls = FALSE; + } else if(strcmp(argv[i], "-linfo") == 0 || + strcmp(argv[i], "-li") == 0) { + info = TRUE; + short_ls = FALSE; + } else if(strcmp(argv[i], "-ef") == 0 || + strcmp(argv[i], "-e") == 0) { + if(++i == argc) { + fprintf(stderr, "%s: -ef missing filename\n", + argv[0]); + exit(1); + } + path = process_extract_files(path, argv[i]); + } else if(strcmp(argv[i], "-regex") == 0 || + strcmp(argv[i], "-r") == 0) + use_regex = TRUE; + else + goto options; + } + + if(lsonly || info) + progress = FALSE; + +#ifdef SQUASHFS_TRACE + progress = FALSE; +#endif + + if(i == argc) { + if(!version) { +options: + ERROR("SYNTAX: %s [options] filesystem [directories or " + "files to extract]\n", argv[0]); + ERROR("\t-v[ersion]\t\tprint version, licence and " + "copyright information\n"); + ERROR("\t-d[est] \tunsquash to , " + "default \"squashfs-root\"\n"); + ERROR("\t-n[o-progress]\t\tdon't display the progress " + "bar\n"); + ERROR("\t-no[-xattrs]\t\tdon't extract xattrs in file system" + NOXOPT_STR"\n"); + ERROR("\t-x[attrs]\t\textract xattrs in file system" + XOPT_STR "\n"); + ERROR("\t-p[rocessors] \tuse " + "processors. By default will use\n"); + ERROR("\t\t\t\tnumber of processors available\n"); + ERROR("\t-i[nfo]\t\t\tprint files as they are " + "unsquashed\n"); + ERROR("\t-li[nfo]\t\tprint files as they are " + "unsquashed with file\n"); + ERROR("\t\t\t\tattributes (like ls -l output)\n"); + ERROR("\t-l[s]\t\t\tlist filesystem, but don't unsquash" + "\n"); + ERROR("\t-ll[s]\t\t\tlist filesystem with file " + "attributes (like\n"); + ERROR("\t\t\t\tls -l output), but don't unsquash\n"); + ERROR("\t-f[orce]\t\tif file already exists then " + "overwrite\n"); + ERROR("\t-s[tat]\t\t\tdisplay filesystem superblock " + "information\n"); + ERROR("\t-e[f] \tlist of directories or " + "files to extract.\n\t\t\t\tOne per line\n"); + ERROR("\t-da[ta-queue] \tSet data queue to " + " Mbytes. Default %d\n\t\t\t\tMbytes\n", + DATA_BUFFER_DEFAULT); + ERROR("\t-fr[ag-queue] \tSet fragment queue to " + " Mbytes. Default\n\t\t\t\t%d Mbytes\n", + FRAGMENT_BUFFER_DEFAULT); + ERROR("\t-r[egex]\t\ttreat extract names as POSIX " + "regular expressions\n"); + ERROR("\t\t\t\trather than use the default shell " + "wildcard\n\t\t\t\texpansion (globbing)\n"); + ERROR("\nDecompressors available:\n"); + display_compressors("", ""); + } + exit(1); + } + + for(n = i + 1; n < argc; n++) + path = add_path(path, argv[n], argv[n]); + + if((fd = open(argv[i], O_RDONLY)) == -1) { + ERROR("Could not open %s, because %s\n", argv[i], + strerror(errno)); + exit(1); + } + + if(read_super(argv[i]) == FALSE) + exit(1); + + if(stat_sys) { + squashfs_stat(argv[i]); + exit(0); + } + + if(!comp->supported) { + ERROR("Filesystem uses %s compression, this is " + "unsupported by this version\n", comp->name); + ERROR("Decompressors available:\n"); + display_compressors("", ""); + exit(1); + } + + block_size = sBlk.s.block_size; + block_log = sBlk.s.block_log; + + fragment_buffer_size <<= 20 - block_log; + data_buffer_size <<= 20 - block_log; + initialise_threads(fragment_buffer_size, data_buffer_size); + + fragment_data = malloc(block_size); + if(fragment_data == NULL) + EXIT_UNSQUASH("failed to allocate fragment_data\n"); + + file_data = malloc(block_size); + if(file_data == NULL) + EXIT_UNSQUASH("failed to allocate file_data"); + + data = malloc(block_size); + if(data == NULL) + EXIT_UNSQUASH("failed to allocate data\n"); + + created_inode = malloc(sBlk.s.inodes * sizeof(char *)); + if(created_inode == NULL) + EXIT_UNSQUASH("failed to allocate created_inode\n"); + + memset(created_inode, 0, sBlk.s.inodes * sizeof(char *)); + + if(s_ops.read_uids_guids() == FALSE) + EXIT_UNSQUASH("failed to uid/gid table\n"); + + if(s_ops.read_fragment_table() == FALSE) + EXIT_UNSQUASH("failed to read fragment table\n"); + + uncompress_inode_table(sBlk.s.inode_table_start, + sBlk.s.directory_table_start); + + uncompress_directory_table(sBlk.s.directory_table_start, + sBlk.s.fragment_table_start); + + if(no_xattrs) + sBlk.s.xattr_id_table_start = SQUASHFS_INVALID_BLK; + + if(read_xattrs_from_disk(fd, &sBlk.s) == 0) + EXIT_UNSQUASH("failed to read the xattr table\n"); + + if(path) { + paths = init_subdir(); + paths = add_subdir(paths, path); + } + + pre_scan(dest, SQUASHFS_INODE_BLK(sBlk.s.root_inode), + SQUASHFS_INODE_OFFSET(sBlk.s.root_inode), paths); + + memset(created_inode, 0, sBlk.s.inodes * sizeof(char *)); + inode_number = 1; + + printf("%d inodes (%d blocks) to write\n\n", total_inodes, + total_inodes - total_files + total_blocks); + + if(progress) + enable_progress_bar(); + + dir_scan(dest, SQUASHFS_INODE_BLK(sBlk.s.root_inode), + SQUASHFS_INODE_OFFSET(sBlk.s.root_inode), paths); + + queue_put(to_writer, NULL); + queue_get(from_writer); + + if(progress) { + disable_progress_bar(); + progress_bar(sym_count + dev_count + fifo_count + cur_blocks, + total_inodes - total_files + total_blocks, columns); + } + + if(!lsonly) { + printf("\n"); + printf("created %d files\n", file_count); + printf("created %d directories\n", dir_count); + printf("created %d symlinks\n", sym_count); + printf("created %d devices\n", dev_count); + printf("created %d fifos\n", fifo_count); + } + + return 0; +} diff --git a/squashfs-tools/unsquashfs.h b/squashfs-tools/unsquashfs.h new file mode 100644 index 0000000..90ea138 --- /dev/null +++ b/squashfs-tools/unsquashfs.h @@ -0,0 +1,297 @@ +/* + * Unsquash a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * unsquashfs.h + */ + +#define TRUE 1 +#define FALSE 0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifndef linux +#define __BYTE_ORDER BYTE_ORDER +#define __BIG_ENDIAN BIG_ENDIAN +#define __LITTLE_ENDIAN LITTLE_ENDIAN +#else +#include +#endif + +#include "squashfs_fs.h" + +#ifdef SQUASHFS_TRACE +#define TRACE(s, args...) \ + do { \ + pthread_mutex_lock(&screen_mutex); \ + if(progress_enabled) \ + printf("\n"); \ + printf("unsquashfs: "s, ## args); \ + pthread_mutex_unlock(&screen_mutex);\ + } while(0) +#else +#define TRACE(s, args...) +#endif + +#define ERROR(s, args...) \ + do { \ + pthread_mutex_lock(&screen_mutex); \ + if(progress_enabled) \ + fprintf(stderr, "\n"); \ + fprintf(stderr, s, ## args); \ + pthread_mutex_unlock(&screen_mutex);\ + } while(0) + +#define EXIT_UNSQUASH(s, args...) \ + do { \ + pthread_mutex_lock(&screen_mutex); \ + fprintf(stderr, "FATAL ERROR aborting: "s, ## args); \ + pthread_mutex_unlock(&screen_mutex);\ + exit(1); \ + } while(0) + +#define CALCULATE_HASH(start) (start & 0xffff) + +/* + * Unified superblock containing fields for all superblocks + */ +struct super_block { + struct squashfs_super_block s; + /* fields only used by squashfs 3 and earlier layouts */ + unsigned int no_uids; + unsigned int no_guids; + long long uid_start; + long long guid_start; +}; + +struct hash_table_entry { + long long start; + int bytes; + struct hash_table_entry *next; +}; + +struct inode { + int blocks; + char *block_ptr; + long long data; + int fragment; + int frag_bytes; + gid_t gid; + int inode_number; + int mode; + int offset; + long long start; + char *symlink; + time_t time; + int type; + uid_t uid; + char sparse; + unsigned int xattr; +}; + +typedef struct squashfs_operations { + struct dir *(*squashfs_opendir)(unsigned int block_start, + unsigned int offset, struct inode **i); + void (*read_fragment)(unsigned int fragment, long long *start_block, + int *size); + int (*read_fragment_table)(); + void (*read_block_list)(unsigned int *block_list, char *block_ptr, + int blocks); + struct inode *(*read_inode)(unsigned int start_block, + unsigned int offset); + int (*read_uids_guids)(); +} squashfs_operations; + +struct test { + int mask; + int value; + int position; + char mode; +}; + + +/* Cache status struct. Caches are used to keep + track of memory buffers passed between different threads */ +struct cache { + int max_buffers; + int count; + int buffer_size; + int wait_free; + int wait_pending; + pthread_mutex_t mutex; + pthread_cond_t wait_for_free; + pthread_cond_t wait_for_pending; + struct cache_entry *free_list; + struct cache_entry *hash_table[65536]; +}; + +/* struct describing a cache entry passed between threads */ +struct cache_entry { + struct cache *cache; + long long block; + int size; + int used; + int error; + int pending; + struct cache_entry *hash_next; + struct cache_entry *hash_prev; + struct cache_entry *free_next; + struct cache_entry *free_prev; + char *data; +}; + +/* struct describing queues used to pass data between threads */ +struct queue { + int size; + int readp; + int writep; + pthread_mutex_t mutex; + pthread_cond_t empty; + pthread_cond_t full; + void **data; +}; + +/* default size of fragment buffer in Mbytes */ +#define FRAGMENT_BUFFER_DEFAULT 256 +/* default size of data buffer in Mbytes */ +#define DATA_BUFFER_DEFAULT 256 + +#define DIR_ENT_SIZE 16 + +struct dir_ent { + char name[SQUASHFS_NAME_LEN + 1]; + unsigned int start_block; + unsigned int offset; + unsigned int type; +}; + +struct dir { + int dir_count; + int cur_entry; + unsigned int mode; + uid_t uid; + gid_t guid; + unsigned int mtime; + unsigned int xattr; + struct dir_ent *dirs; +}; + +struct file_entry { + int offset; + int size; + struct cache_entry *buffer; +}; + + +struct squashfs_file { + int fd; + int blocks; + long long file_size; + int mode; + uid_t uid; + gid_t gid; + time_t time; + char *pathname; + char sparse; + unsigned int xattr; +}; + +struct path_entry { + char *name; + regex_t *preg; + struct pathname *paths; +}; + +struct pathname { + int names; + struct path_entry *name; +}; + +struct pathnames { + int count; + struct pathname *path[0]; +}; +#define PATHS_ALLOC_SIZE 10 + +/* globals */ +extern struct super_block sBlk; +extern squashfs_operations s_ops; +extern int swap; +extern char *inode_table, *directory_table; +extern struct hash_table_entry *inode_table_hash[65536], + *directory_table_hash[65536]; +extern unsigned int *uid_table, *guid_table; +extern pthread_mutex_t screen_mutex; +extern int progress_enabled; +extern int inode_number; +extern int lookup_type[]; +extern int fd; + +/* unsquashfs.c */ +extern int lookup_entry(struct hash_table_entry **, long long); +extern int read_fs_bytes(int fd, long long, int, void *); +extern int read_block(int, long long, long long *, void *); + +/* unsquash-1.c */ +extern void read_block_list_1(unsigned int *, char *, int); +extern int read_fragment_table_1(); +extern struct inode *read_inode_1(unsigned int, unsigned int); +extern struct dir *squashfs_opendir_1(unsigned int, unsigned int, + struct inode **); +extern int read_uids_guids_1(); + +/* unsquash-2.c */ +extern void read_block_list_2(unsigned int *, char *, int); +extern int read_fragment_table_2(); +extern void read_fragment_2(unsigned int, long long *, int *); +extern struct inode *read_inode_2(unsigned int, unsigned int); + +/* unsquash-3.c */ +extern int read_fragment_table_3(); +extern void read_fragment_3(unsigned int, long long *, int *); +extern struct inode *read_inode_3(unsigned int, unsigned int); +extern struct dir *squashfs_opendir_3(unsigned int, unsigned int, + struct inode **); + +/* unsquash-4.c */ +extern int read_fragment_table_4(); +extern void read_fragment_4(unsigned int, long long *, int *); +extern struct inode *read_inode_4(unsigned int, unsigned int); +extern struct dir *squashfs_opendir_4(unsigned int, unsigned int, + struct inode **); +extern int read_uids_guids_4(); diff --git a/squashfs-tools/unsquashfs_xattr.c b/squashfs-tools/unsquashfs_xattr.c new file mode 100644 index 0000000..f612931 --- /dev/null +++ b/squashfs-tools/unsquashfs_xattr.c @@ -0,0 +1,72 @@ +/* + * Unsquash a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * unsquashfs_xattr.c + */ + +#include "unsquashfs.h" +#include "xattr.h" + +#include + +extern int root_process; + +void write_xattr(char *pathname, unsigned int xattr) +{ + unsigned int count; + struct xattr_list *xattr_list; + int i; + + if(xattr == SQUASHFS_INVALID_XATTR || + sBlk.s.xattr_id_table_start == SQUASHFS_INVALID_BLK) + return; + + xattr_list = get_xattr(xattr, &count); + if(xattr_list == NULL) { + ERROR("Failed to read xattrs for file %s\n", pathname); + return; + } + + for(i = 0; i < count; i++) { + int prefix = xattr_list[i].type & SQUASHFS_XATTR_PREFIX_MASK; + + if(root_process || prefix == SQUASHFS_XATTR_USER) { + int res = lsetxattr(pathname, xattr_list[i].full_name, + xattr_list[i].value, xattr_list[i].vsize, 0); + + if(res == -1) + ERROR("write_xattr: failed to write xattr %s" + " for file %s because %s\n", + xattr_list[i].full_name, pathname, + errno == ENOSPC || errno == EDQUOT ? + "no extended attribute space remaining " + "on destination filesystem" : + errno == ENOTSUP ? + "extended attributes are not supported " + "by the destination filesystem" : + "a weird error occurred"); + } else + ERROR("write_xattr: could not write xattr %s " + "for file %s because you're not " + "superuser!\n", + xattr_list[i].full_name, pathname); + } +} diff --git a/squashfs-tools/xattr.c b/squashfs-tools/xattr.c new file mode 100644 index 0000000..c1b307f --- /dev/null +++ b/squashfs-tools/xattr.c @@ -0,0 +1,749 @@ +/* + * Create a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2008, 2009, 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * xattr.c + */ + +#define TRUE 1 +#define FALSE 0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "squashfs_fs.h" +#include "squashfs_swap.h" +#include "mksquashfs.h" +#include "xattr.h" + +#ifdef SQUASHFS_TRACE +#define TRACE(s, args...) \ + do { \ + printf("mksquashfs: "s, ## args); \ + } while(0) +#else +#define TRACE(s, args...) +#endif + +#define ERROR(s, args...) \ + do { \ + fprintf(stderr, s, ## args); \ + } while(0) + +/* compressed xattr table */ +static char *xattr_table = NULL; +static unsigned int xattr_size = 0; + +/* cached uncompressed xattr data */ +static char *data_cache = NULL; +static int cache_bytes = 0, cache_size = 0; + +/* cached uncompressed xattr id table */ +static struct squashfs_xattr_id *xattr_id_table = NULL; +static int xattr_ids = 0; + +/* saved compressed xattr table */ +unsigned int sxattr_bytes = 0, stotal_xattr_bytes = 0; + +/* saved cached uncompressed xattr data */ +static char *sdata_cache = NULL; +static int scache_bytes = 0; + +/* saved cached uncompressed xattr id table */ +static int sxattr_ids = 0; + +/* xattr hash table for value duplicate detection */ +static struct xattr_list *dupl[65536]; + +/* xattr hash table for id duplicate detection */ +static struct dupl_id *dupl_id[65536]; + +/* file system globals from mksquashfs.c */ +extern int no_xattrs, noX; +extern long long bytes; +extern int fd; +extern unsigned int xattr_bytes, total_xattr_bytes; + +/* helper functions from mksquashfs.c */ +extern unsigned short get_checksum(char *, int, unsigned short); +extern void write_destination(int, long long, int, void *); +extern long long generic_write_table(int, void *, int, void *, int); +extern int mangle(char *, char *, int, int, int, int); + +/* helper functions and definitions from read_xattrs.c */ +extern int read_xattrs_from_disk(int, struct squashfs_super_block *); +extern struct xattr_list *get_xattr(int, unsigned int *); +extern struct prefix prefix_table[]; + + +static int get_prefix(struct xattr_list *xattr, char *name) +{ + int i; + + xattr->full_name = strdup(name); + + for(i = 0; prefix_table[i].type != -1; i++) { + struct prefix *p = &prefix_table[i]; + if(strncmp(xattr->full_name, p->prefix, strlen(p->prefix)) == 0) + break; + } + + if(prefix_table[i].type != -1) { + xattr->name = xattr->full_name + strlen(prefix_table[i].prefix); + xattr->size = strlen(xattr->name); + } + + return prefix_table[i].type; +} + + +static int read_xattrs_from_system(char *filename, struct xattr_list **xattrs) +{ + ssize_t size, vsize; + char *xattr_names, *p; + int i; + struct xattr_list *xattr_list = NULL; + + while(1) { + size = llistxattr(filename, NULL, 0); + if(size <= 0) { + if(size < 0 && errno != ENOTSUP) + ERROR("llistxattr for %s failed in read_attrs," + " because %s\n", filename, + strerror(errno)); + return 0; + } + + xattr_names = malloc(size); + if(xattr_names == NULL) { + ERROR("Out of memory in read_attrs\n"); + return 0; + } + + size = llistxattr(filename, xattr_names, size); + if(size < 0) { + free(xattr_names); + if(errno == ERANGE) + /* xattr list grew? Try again */ + continue; + else { + ERROR("llistxattr for %s failed in read_attrs," + " because %s\n", filename, + strerror(errno)); + return 0; + } + } + + break; + } + + for(i = 0, p = xattr_names; p < xattr_names + size; i++) { + struct xattr_list *x = realloc(xattr_list, (i + 1) * + sizeof(struct xattr_list)); + if(x == NULL) { + ERROR("Out of memory in read_attrs\n"); + goto failed; + } else + xattr_list = x; + + xattr_list[i].type = get_prefix(&xattr_list[i], p); + p += strlen(p) + 1; + if(xattr_list[i].type == -1) { + ERROR("Unrecognised xattr prefix %s\n", + xattr_list[i].full_name); + free(xattr_list[i].full_name); + i--; + continue; + } + + while(1) { + vsize = lgetxattr(filename, xattr_list[i].full_name, + NULL, 0); + if(vsize < 0) { + ERROR("lgetxattr failed for %s in read_attrs," + " because %s\n", filename, + strerror(errno)); + free(xattr_list[i].full_name); + goto failed; + } + + xattr_list[i].value = malloc(vsize); + if(xattr_list[i].value == NULL) { + ERROR("Out of memory in read_attrs\n"); + free(xattr_list[i].full_name); + goto failed; + } + + vsize = lgetxattr(filename, xattr_list[i].full_name, + xattr_list[i].value, vsize); + if(vsize < 0) { + free(xattr_list[i].value); + if(errno == ERANGE) + /* xattr grew? Try again */ + continue; + else { + ERROR("lgetxattr failed for %s in " + "read_attrs, because %s\n", + filename, strerror(errno)); + free(xattr_list[i].full_name); + goto failed; + } + } + + break; + } + xattr_list[i].vsize = vsize; + + TRACE("read_xattrs_from_system: filename %s, xattr name %s," + " vsize %d\n", filename, xattr_list[i].full_name, + xattr_list[i].vsize); + } + free(xattr_names); + *xattrs = xattr_list; + return i; + +failed: + while(--i >= 0) { + free(xattr_list[i].full_name); + free(xattr_list[i].value); + } + free(xattr_list); + free(xattr_names); + return 0; +} + + +static int get_xattr_size(struct xattr_list *xattr) +{ + int size = sizeof(struct squashfs_xattr_entry) + + sizeof(struct squashfs_xattr_val) + xattr->size; + + if(xattr->type & XATTR_VALUE_OOL) + size += XATTR_VALUE_OOL_SIZE; + else + size += xattr->vsize; + + return size; +} + + +static void *get_xattr_space(unsigned int req_size, long long *disk) +{ + int data_space; + unsigned short c_byte; + + /* + * Move and compress cached uncompressed data into xattr table. + */ + while(cache_bytes >= SQUASHFS_METADATA_SIZE) { + if((xattr_size - xattr_bytes) < + ((SQUASHFS_METADATA_SIZE << 1)) + 2) { + xattr_table = realloc(xattr_table, xattr_size + + (SQUASHFS_METADATA_SIZE << 1) + 2); + if(xattr_table == NULL) { + goto failed; + } + xattr_size += (SQUASHFS_METADATA_SIZE << 1) + 2; + } + + c_byte = mangle(xattr_table + xattr_bytes + BLOCK_OFFSET, + data_cache, SQUASHFS_METADATA_SIZE, + SQUASHFS_METADATA_SIZE, noX, 0); + TRACE("Xattr block @ 0x%x, size %d\n", xattr_bytes, c_byte); + SQUASHFS_SWAP_SHORTS(&c_byte, xattr_table + xattr_bytes, 1); + xattr_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; + memmove(data_cache, data_cache + SQUASHFS_METADATA_SIZE, + cache_bytes - SQUASHFS_METADATA_SIZE); + cache_bytes -= SQUASHFS_METADATA_SIZE; + } + + /* + * Ensure there's enough space in the uncompressed data cache + */ + data_space = cache_size - cache_bytes; + if(data_space < req_size) { + int realloc_size = req_size - data_space; + data_cache = realloc(data_cache, cache_size + + realloc_size); + if(data_cache == NULL) { + goto failed; + } + cache_size += realloc_size; + } + + if(disk) + *disk = ((long long) xattr_bytes << 16) | cache_bytes; + cache_bytes += req_size; + return data_cache + cache_bytes - req_size; + +failed: + ERROR("Out of memory in inode table reallocation!\n"); + return NULL; +} + + +static struct dupl_id *check_id_dupl(struct xattr_list *xattr_list, int xattrs) +{ + struct dupl_id *entry; + int i; + unsigned short checksum = 0; + + /* compute checksum over all xattrs */ + for(i = 0; i < xattrs; i++) { + struct xattr_list *xattr = &xattr_list[i]; + + checksum = get_checksum(xattr->full_name, + strlen(xattr->full_name), checksum); + checksum = get_checksum(xattr->value, + xattr->vsize, checksum); + } + + for(entry = dupl_id[checksum]; entry; entry = entry->next) { + if (entry->xattrs != xattrs) + continue; + + for(i = 0; i < xattrs; i++) { + struct xattr_list *xattr = &xattr_list[i]; + struct xattr_list *dup_xattr = &entry->xattr_list[i]; + + if(strcmp(xattr->full_name, dup_xattr->full_name)) + break; + + if(memcmp(xattr->value, dup_xattr->value, xattr->vsize)) + break; + } + + if(i == xattrs) + break; + } + + if(entry == NULL) { + /* no duplicate exists */ + entry = malloc(sizeof(*entry)); + if(entry == NULL) { + ERROR("malloc failed in check_ip_dupl\n"); + return NULL; + } + entry->xattrs = xattrs; + entry->xattr_list = xattr_list; + entry->xattr_id = SQUASHFS_INVALID_XATTR; + entry->next = dupl_id[checksum]; + dupl_id[checksum] = entry; + } + + return entry; +} + + +static void check_value_dupl(struct xattr_list *xattr) +{ + struct xattr_list *entry; + + if(xattr->vsize < XATTR_VALUE_OOL_SIZE) + return; + + /* Check if this is a duplicate of an existing value */ + xattr->vchecksum = get_checksum(xattr->value, xattr->vsize, 0); + for(entry = dupl[xattr->vchecksum]; entry; entry = entry->vnext) { + if(entry->vsize != xattr->vsize) + continue; + + if(memcmp(entry->value, xattr->value, xattr->vsize) == 0) + break; + } + + if(entry == NULL) { + /* + * No duplicate exists, add to hash table, and mark as + * requiring writing + */ + xattr->vnext = dupl[xattr->vchecksum]; + dupl[xattr->vchecksum] = xattr; + xattr->ool_value = SQUASHFS_INVALID_BLK; + } else { + /* + * Duplicate exists, make type XATTR_VALUE_OOL, and + * remember where the duplicate is + */ + xattr->type |= XATTR_VALUE_OOL; + xattr->ool_value = entry->ool_value; + /* on appending don't free duplicate values because the + * duplicate value already points to the non-duplicate value */ + if(xattr->value != entry->value) { + free(xattr->value); + xattr->value = entry->value; + } + } +} + + +static int get_xattr_id(int xattrs, struct xattr_list *xattr_list, + long long xattr_disk, struct dupl_id *xattr_dupl) +{ + int i, size = 0; + struct squashfs_xattr_id *xattr_id; + + xattr_id_table = realloc(xattr_id_table, (xattr_ids + 1) * + sizeof(struct squashfs_xattr_id)); + if(xattr_id_table == NULL) { + ERROR("Out of memory in xattr_id_table reallocation!\n"); + return -1; + } + + /* get total uncompressed size of xattr data, needed for stat */ + for(i = 0; i < xattrs; i++) + size += strlen(xattr_list[i].full_name) + 1 + + xattr_list[i].vsize; + + xattr_id = &xattr_id_table[xattr_ids]; + xattr_id->xattr = xattr_disk; + xattr_id->count = xattrs; + xattr_id->size = size; + + /* + * keep track of total uncompressed xattr data, needed for mksquashfs + * file system summary + */ + total_xattr_bytes += size; + + xattr_dupl->xattr_id = xattr_ids ++; + return xattr_dupl->xattr_id; +} + + +long long write_xattrs() +{ + unsigned short c_byte; + int i, avail_bytes; + char *datap = data_cache; + long long start_bytes = bytes; + struct squashfs_xattr_table header; + + if(xattr_ids == 0) + return SQUASHFS_INVALID_BLK; + + /* + * Move and compress cached uncompressed data into xattr table. + */ + while(cache_bytes) { + if((xattr_size - xattr_bytes) < + ((SQUASHFS_METADATA_SIZE << 1)) + 2) { + xattr_table = realloc(xattr_table, xattr_size + + (SQUASHFS_METADATA_SIZE << 1) + 2); + if(xattr_table == NULL) { + goto failed; + } + xattr_size += (SQUASHFS_METADATA_SIZE << 1) + 2; + } + + avail_bytes = cache_bytes > SQUASHFS_METADATA_SIZE ? + SQUASHFS_METADATA_SIZE : cache_bytes; + c_byte = mangle(xattr_table + xattr_bytes + BLOCK_OFFSET, datap, + avail_bytes, SQUASHFS_METADATA_SIZE, noX, 0); + TRACE("Xattr block @ 0x%x, size %d\n", xattr_bytes, c_byte); + SQUASHFS_SWAP_SHORTS(&c_byte, xattr_table + xattr_bytes, 1); + xattr_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; + datap += avail_bytes; + cache_bytes -= avail_bytes; + } + + /* + * Write compressed xattr table to file system + */ + write_destination(fd, bytes, xattr_bytes, xattr_table); + bytes += xattr_bytes; + + /* + * Swap if necessary the xattr id table + */ + for(i = 0; i < xattr_ids; i++) + SQUASHFS_INSWAP_XATTR_ID(&xattr_id_table[i]); + + header.xattr_ids = xattr_ids; + header.xattr_table_start = start_bytes; + SQUASHFS_INSWAP_XATTR_TABLE(&header); + + return generic_write_table(xattr_ids * sizeof(struct squashfs_xattr_id), + xattr_id_table, sizeof(header), &header, noX); + +failed: + ERROR("Out of memory in xattr_table reallocation!\n"); + return -1; +} + + +int generate_xattrs(int xattrs, struct xattr_list *xattr_list) +{ + int total_size, i; + int xattr_value_max; + void *xp; + long long xattr_disk; + struct dupl_id *xattr_dupl; + + /* + * check if the file xattrs are a complete duplicate of a pre-existing + * id + */ + xattr_dupl = check_id_dupl(xattr_list, xattrs); + if (xattr_dupl == NULL) + return SQUASHFS_INVALID_XATTR; + if(xattr_dupl->xattr_id != SQUASHFS_INVALID_XATTR) + return xattr_dupl->xattr_id; + + /* + * Scan the xattr_list deciding which type to assign to each + * xattr. The choice is fairly straightforward, and depends on the + * size of each xattr name/value and the overall size of the + * resultant xattr list stored in the xattr metadata table. + * + * Choices are whether to store data inline or out of line. + * + * The overall goal is to optimise xattr scanning and lookup, and + * to enable the file system layout to scale from a couple of + * small xattr name/values to a large number of large xattr + * names/values without affecting performance. While hopefully + * enabling the common case of a couple of small xattr name/values + * to be stored efficiently + * + * Code repeatedly scans, doing the following + * move xattr data out of line if it exceeds + * xattr_value_max. Where xattr_value_max is + * initially XATTR_INLINE_MAX. If the final uncompressed + * xattr list is larger than XATTR_TARGET_MAX then more + * aggressively move xattr data out of line by repeatedly + * setting inline threshold to 1/2, then 1/4, 1/8 of + * XATTR_INLINE_MAX until target achieved or there's + * nothing left to move out of line + */ + xattr_value_max = XATTR_INLINE_MAX; + while(1) { + for(total_size = 0, i = 0; i < xattrs; i++) { + struct xattr_list *xattr = &xattr_list[i]; + xattr->type &= XATTR_PREFIX_MASK; /* all inline */ + if (xattr->vsize > xattr_value_max) + xattr->type |= XATTR_VALUE_OOL; + + total_size += get_xattr_size(xattr); + } + + /* + * If the total size of the uncompressed xattr list is <= + * XATTR_TARGET_MAX we're done + */ + if(total_size <= XATTR_TARGET_MAX) + break; + + if(xattr_value_max == XATTR_VALUE_OOL_SIZE) + break; + + /* + * Inline target not yet at minimum and so reduce it, and + * try again + */ + xattr_value_max /= 2; + if(xattr_value_max < XATTR_VALUE_OOL_SIZE) + xattr_value_max = XATTR_VALUE_OOL_SIZE; + } + + /* + * Check xattr values for duplicates + */ + for(i = 0; i < xattrs; i++) { + check_value_dupl(&xattr_list[i]); + } + + /* + * Add each out of line value to the file system xattr table + * if it doesn't already exist as a duplicate + */ + for(i = 0; i < xattrs; i++) { + struct xattr_list *xattr = &xattr_list[i]; + + if((xattr->type & XATTR_VALUE_OOL) && + (xattr->ool_value == SQUASHFS_INVALID_BLK)) { + struct squashfs_xattr_val val; + int size = sizeof(val) + xattr->vsize; + xp = get_xattr_space(size, &xattr->ool_value); + val.vsize = xattr->vsize; + SQUASHFS_SWAP_XATTR_VAL(&val, xp); + memcpy(xp + sizeof(val), xattr->value, xattr->vsize); + } + } + + /* + * Create xattr list and add to file system xattr table + */ + get_xattr_space(0, &xattr_disk); + for(i = 0; i < xattrs; i++) { + struct xattr_list *xattr = &xattr_list[i]; + struct squashfs_xattr_entry entry; + struct squashfs_xattr_val val; + + xp = get_xattr_space(sizeof(entry) + xattr->size, NULL); + entry.type = xattr->type; + entry.size = xattr->size; + SQUASHFS_SWAP_XATTR_ENTRY(&entry, xp); + memcpy(xp + sizeof(entry), xattr->name, xattr->size); + + if(xattr->type & XATTR_VALUE_OOL) { + int size = sizeof(val) + XATTR_VALUE_OOL_SIZE; + xp = get_xattr_space(size, NULL); + val.vsize = XATTR_VALUE_OOL_SIZE; + SQUASHFS_SWAP_XATTR_VAL(&val, xp); + SQUASHFS_SWAP_LONG_LONGS(&xattr->ool_value, xp + + sizeof(val), 1); + } else { + int size = sizeof(val) + xattr->vsize; + xp = get_xattr_space(size, &xattr->ool_value); + val.vsize = xattr->vsize; + SQUASHFS_SWAP_XATTR_VAL(&val, xp); + memcpy(xp + sizeof(val), xattr->value, xattr->vsize); + } + } + + /* + * Add to xattr id lookup table + */ + return get_xattr_id(xattrs, xattr_list, xattr_disk, xattr_dupl); +} + + +int read_xattrs(void *d) +{ + struct dir_ent *dir_ent = d; + struct inode_info *inode = dir_ent->inode; + char *filename = dir_ent->pathname; + struct xattr_list *xattr_list; + int xattrs; + + if(no_xattrs || IS_PSEUDO(inode) || inode->root_entry) + return SQUASHFS_INVALID_XATTR; + + xattrs = read_xattrs_from_system(filename, &xattr_list); + if(xattrs == 0) + return SQUASHFS_INVALID_XATTR; + + return generate_xattrs(xattrs, xattr_list); +} + + +/* + * Add the existing xattr ids and xattr metadata in the file system being + * appended to, to the in-memory xattr cache. This allows duplicate checking to + * take place against the xattrs already in the file system being appended to, + * and ensures the pre-existing xattrs are written out along with any new xattrs + */ +int get_xattrs(int fd, struct squashfs_super_block *sBlk) +{ + int ids, res, i, id; + unsigned int count; + + TRACE("get_xattrs\n"); + + res = read_xattrs_from_disk(fd, sBlk); + if(res == SQUASHFS_INVALID_BLK || res == 0) + goto done; + ids = res; + + /* + * for each xattr id read and construct its list of xattr + * name:value pairs, and add them to the in-memory xattr cache + */ + for(i = 0; i < ids; i++) { + struct xattr_list *xattr_list = get_xattr(i, &count); + if(xattr_list == NULL) { + res = 0; + goto done; + } + id = generate_xattrs(count, xattr_list); + + /* + * Sanity check, the new xattr id should be the same as the + * xattr id in the original file system + */ + if(id != i) { + ERROR("BUG, different xattr_id in get_xattrs\n"); + res = 0; + goto done; + } + } + +done: + return res; +} + + +/* + * Save current state of xattrs, needed for restoring state in the event of an + * abort in appending + */ +int save_xattrs() +{ + /* save the current state of the compressed xattr data */ + sxattr_bytes = xattr_bytes; + stotal_xattr_bytes = total_xattr_bytes; + + /* + * save the current state of the cached uncompressed xattr data. + * Note we have to save the contents of the data cache because future + * operations will delete the current contents + */ + sdata_cache = malloc(cache_bytes); + if(sdata_cache == NULL) + goto failed; + + memcpy(sdata_cache, data_cache, cache_bytes); + scache_bytes = cache_bytes; + + /* save the current state of the xattr id table */ + sxattr_ids = xattr_ids; + + return TRUE; + +failed: + ERROR("Out of memory in save_xattrs\n"); + return FALSE; +} + + +/* + * Restore xattrs in the event of an abort in appending + */ +void restore_xattrs() +{ + /* restore the state of the compressed xattr data */ + xattr_bytes = sxattr_bytes; + total_xattr_bytes = stotal_xattr_bytes; + + /* restore the state of the uncomoressed xattr data */ + memcpy(data_cache, sdata_cache, scache_bytes); + cache_bytes = scache_bytes; + + /* restore the state of the xattr id table */ + xattr_ids = sxattr_ids; +} diff --git a/squashfs-tools/xattr.h b/squashfs-tools/xattr.h new file mode 100644 index 0000000..1759b23 --- /dev/null +++ b/squashfs-tools/xattr.h @@ -0,0 +1,148 @@ +/* + * Create a squashfs filesystem. This is a highly compressed read only + * filesystem. + * + * Copyright (c) 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * xattr.h + */ + +#define XATTR_VALUE_OOL SQUASHFS_XATTR_VALUE_OOL +#define XATTR_PREFIX_MASK SQUASHFS_XATTR_PREFIX_MASK + +#define XATTR_VALUE_OOL_SIZE sizeof(long long) + +/* maximum size of xattr value data that will be inlined */ +#define XATTR_INLINE_MAX 128 + +/* the target size of an inode's xattr name:value list. If it + * exceeds this, then xattr value data will be successively out of lined + * until it meets the target */ +#define XATTR_TARGET_MAX 65536 + +#define IS_XATTR(a) (a != SQUASHFS_INVALID_XATTR) + +struct xattr_list { + char *name; + char *full_name; + int size; + int vsize; + void *value; + int type; + long long ool_value; + unsigned short vchecksum; + struct xattr_list *vnext; +}; + +struct dupl_id { + struct xattr_list *xattr_list; + int xattrs; + int xattr_id; + struct dupl_id *next; +}; + +struct prefix { + char *prefix; + int type; +}; + +extern int generate_xattrs(int, struct xattr_list *); + +#ifdef XATTR_SUPPORT +extern int get_xattrs(int, struct squashfs_super_block *); +extern int read_xattrs(void *); +extern long long write_xattrs(); +extern int save_xattrs(); +extern void restore_xattrs(); +extern unsigned int xattr_bytes, total_xattr_bytes; +extern void write_xattr(char *, unsigned int); +extern int read_xattrs_from_disk(int, struct squashfs_super_block *); +extern struct xattr_list *get_xattr(int, unsigned int *); +#else +static inline int get_xattrs(int fd, struct squashfs_super_block *sBlk) +{ + if(sBlk->xattr_id_table_start != SQUASHFS_INVALID_BLK) { + fprintf(stderr, "Xattrs in filesystem! These are not " + "supported on this version of Squashfs\n"); + return 0; + } else + return SQUASHFS_INVALID_BLK; +} + + +static inline int read_xattrs(void *dir_ent) +{ + return SQUASHFS_INVALID_XATTR; +} + + +static inline long long write_xattrs() +{ + return SQUASHFS_INVALID_BLK; +} + + +static inline int save_xattrs() +{ + return 1; +} + + +static inline void restore_xattrs() +{ +} + + +static inline void write_xattr(char *pathname, unsigned int xattr) +{ +} + + +static inline int read_xattrs_from_disk(int fd, struct squashfs_super_block *sBlk) +{ + if(sBlk->xattr_id_table_start != SQUASHFS_INVALID_BLK) { + fprintf(stderr, "Xattrs in filesystem! These are not " + "supported on this version of Squashfs\n"); + return 0; + } else + return SQUASHFS_INVALID_BLK; +} + + +static inline struct xattr_list *get_xattr(int i, unsigned int *count) +{ + return NULL; +} +#endif + +#ifdef XATTR_SUPPORT +#ifdef XATTR_DEFAULT +#define NOXOPT_STR +#define XOPT_STR " (default)" +#define XATTR_DEF 0 +#else +#define NOXOPT_STR " (default)" +#define XOPT_STR +#define XATTR_DEF 1 +#endif +#else +#define NOXOPT_STR " (default)" +#define XOPT_STR " (unsupported)" +#define XATTR_DEF 1 +#endif + diff --git a/squashfs-tools/xz_wrapper.c b/squashfs-tools/xz_wrapper.c new file mode 100644 index 0000000..36c4bb8 --- /dev/null +++ b/squashfs-tools/xz_wrapper.c @@ -0,0 +1,418 @@ +/* + * Copyright (c) 2010, 2011 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * xz_wrapper.c + * + * Support for XZ (LZMA2) compression using XZ Utils liblzma + * http://tukaani.org/xz/ + */ + +#include +#include +#include +#include + +#include "squashfs_fs.h" +#include "xz_wrapper.h" +#include "compressor.h" + +static struct bcj bcj[] = { + { "x86", LZMA_FILTER_X86, 0 }, + { "powerpc", LZMA_FILTER_POWERPC, 0 }, + { "ia64", LZMA_FILTER_IA64, 0 }, + { "arm", LZMA_FILTER_ARM, 0 }, + { "armthumb", LZMA_FILTER_ARMTHUMB, 0 }, + { "sparc", LZMA_FILTER_SPARC, 0 }, + { NULL, LZMA_VLI_UNKNOWN, 0 } +}; + +static struct comp_opts comp_opts; + +static int filter_count = 1; +static int dictionary_size = 0; +static float dictionary_percent = 0; + + +static int xz_options(char *argv[], int argc) +{ + int i; + char *name; + + if(strcmp(argv[0], "-Xbcj") == 0) { + if(argc < 2) { + fprintf(stderr, "xz: -Xbcj missing filter\n"); + goto failed; + } + + name = argv[1]; + while(name[0] != '\0') { + for(i = 0; bcj[i].name; i++) { + int n = strlen(bcj[i].name); + if((strncmp(name, bcj[i].name, n) == 0) && + (name[n] == '\0' || + name[n] == ',')) { + if(bcj[i].selected == 0) { + bcj[i].selected = 1; + filter_count++; + } + name += name[n] == ',' ? n + 1 : n; + break; + } + } + if(bcj[i].name == NULL) { + fprintf(stderr, "xz: -Xbcj unrecognised " + "filter\n"); + goto failed; + } + } + + return 1; + } else if(strcmp(argv[0], "-Xdict-size") == 0) { + char *b; + float size; + + if(argc < 2) { + fprintf(stderr, "xz: -Xdict-size missing dict-size\n"); + goto failed; + } + + size = strtof(argv[1], &b); + if(*b == '%') { + if(size <= 0 || size > 100) { + fprintf(stderr, "xz: -Xdict-size percentage " + "should be 0 < dict-size <= 100\n"); + goto failed; + } + + dictionary_percent = size; + dictionary_size = 0; + } else { + if((float) ((int) size) != size) { + fprintf(stderr, "xz: -Xdict-size can't be " + "fractional unless a percentage of the" + " block size\n"); + goto failed; + } + + dictionary_percent = 0; + dictionary_size = (int) size; + + if(*b == 'k' || *b == 'K') + dictionary_size *= 1024; + else if(*b == 'm' || *b == 'M') + dictionary_size *= 1024 * 1024; + else if(*b != '\0') { + fprintf(stderr, "xz: -Xdict-size invalid " + "dict-size\n"); + goto failed; + } + } + + return 1; + } + + return -1; + +failed: + return -2; +} + + +static int xz_options_post(int block_size) +{ + /* + * if -Xdict-size has been specified use this to compute the datablock + * dictionary size + */ + if(dictionary_size || dictionary_percent) { + int n; + + if(dictionary_size) { + if(dictionary_size > block_size) { + fprintf(stderr, "xz: -Xdict-size is larger than" + " block_size\n"); + goto failed; + } + } else + dictionary_size = block_size * dictionary_percent / 100; + + if(dictionary_size < 8192) { + fprintf(stderr, "xz: -Xdict-size should be 8192 bytes " + "or larger\n"); + goto failed; + } + + /* + * dictionary_size must be storable in xz header as either + * 2^n or as 2^n+2^(n+1) + */ + n = ffs(dictionary_size) - 1; + if(dictionary_size != (1 << n) && + dictionary_size != ((1 << n) + (1 << (n + 1)))) { + fprintf(stderr, "xz: -Xdict-size is an unsupported " + "value, dict-size must be storable in xz " + "header\n"); + fprintf(stderr, "as either 2^n or as 2^n+2^(n+1). " + "Example dict-sizes are 75%%, 50%%, 37.5%%, " + "25%%,\n"); + fprintf(stderr, "or 32K, 16K, 8K etc.\n"); + goto failed; + } + + } else + /* No -Xdict-size specified, use defaults */ + dictionary_size = block_size; + + return 0; + +failed: + return -1; +} + + +static void *xz_dump_options(int block_size, int *size) +{ + int flags = 0, i; + + /* + * don't store compressor specific options in file system if the + * default options are being used - no compressor options in the + * file system means the default options are always assumed + * + * Defaults are: + * metadata dictionary size: SQUASHFS_METADATA_SIZE + * datablock dictionary size: block_size + * 1 filter + */ + if(dictionary_size == block_size && filter_count == 1) + return NULL; + + for(i = 0; bcj[i].name; i++) + flags |= bcj[i].selected << i; + + comp_opts.dictionary_size = dictionary_size; + comp_opts.flags = flags; + + SQUASHFS_INSWAP_COMP_OPTS(&comp_opts); + + *size = sizeof(comp_opts); + return &comp_opts; +} + + +static int xz_extract_options(int block_size, void *buffer, int size) +{ + struct comp_opts *comp_opts = buffer; + int flags, i, n; + + if(size == 0) { + /* set defaults */ + dictionary_size = block_size; + flags = 0; + } else { + /* check passed comp opts struct is of the correct length */ + if(size != sizeof(struct comp_opts)) + goto failed; + + SQUASHFS_INSWAP_COMP_OPTS(comp_opts); + + dictionary_size = comp_opts->dictionary_size; + flags = comp_opts->flags; + + /* + * check that the dictionary size seems correct - the dictionary + * size should 2^n or 2^n+2^(n+1) + */ + n = ffs(dictionary_size) - 1; + if(dictionary_size != (1 << n) && + dictionary_size != ((1 << n) + (1 << (n + 1)))) + goto failed; + } + + filter_count = 1; + for(i = 0; bcj[i].name; i++) { + if((flags >> i) & 1) { + bcj[i].selected = 1; + filter_count ++; + } else + bcj[i].selected = 0; + } + + return 0; + +failed: + fprintf(stderr, "xz: error reading stored compressor options from " + "filesystem!\n"); + + return -1; +} + + +static int xz_init(void **strm, int block_size, int datablock) +{ + int i, j, filters = datablock ? filter_count : 1; + struct filter *filter = malloc(filters * sizeof(struct filter)); + struct xz_stream *stream; + + if(filter == NULL) + goto failed; + + stream = *strm = malloc(sizeof(struct xz_stream)); + if(stream == NULL) + goto failed2; + + stream->filter = filter; + stream->filters = filters; + + memset(filter, 0, filters * sizeof(struct filter)); + + stream->dictionary_size = datablock ? dictionary_size : + SQUASHFS_METADATA_SIZE; + + filter[0].filter[0].id = LZMA_FILTER_LZMA2; + filter[0].filter[0].options = &stream->opt; + filter[0].filter[1].id = LZMA_VLI_UNKNOWN; + + for(i = 0, j = 1; datablock && bcj[i].name; i++) { + if(bcj[i].selected) { + filter[j].buffer = malloc(block_size); + if(filter[j].buffer == NULL) + goto failed3; + filter[j].filter[0].id = bcj[i].id; + filter[j].filter[1].id = LZMA_FILTER_LZMA2; + filter[j].filter[1].options = &stream->opt; + filter[j].filter[2].id = LZMA_VLI_UNKNOWN; + j++; + } + } + + return 0; + +failed3: + for(i = 1; i < filters; i++) + free(filter[i].buffer); + free(stream); + +failed2: + free(filter); + +failed: + return -1; +} + + +static int xz_compress(void *strm, void *dest, void *src, int size, + int block_size, int *error) +{ + int i; + lzma_ret res = 0; + struct xz_stream *stream = strm; + struct filter *selected = NULL; + + stream->filter[0].buffer = dest; + + for(i = 0; i < stream->filters; i++) { + struct filter *filter = &stream->filter[i]; + + if(lzma_lzma_preset(&stream->opt, LZMA_PRESET_DEFAULT)) + goto failed; + + stream->opt.dict_size = stream->dictionary_size; + + filter->length = 0; + res = lzma_stream_buffer_encode(filter->filter, + LZMA_CHECK_CRC32, NULL, src, size, filter->buffer, + &filter->length, block_size); + + if(res == LZMA_OK) { + if(!selected || selected->length > filter->length) + selected = filter; + } else if(res != LZMA_BUF_ERROR) + goto failed; + } + + if(!selected) + /* + * Output buffer overflow. Return out of buffer space + */ + return 0; + + if(selected->buffer != dest) + memcpy(dest, selected->buffer, selected->length); + + return (int) selected->length; + +failed: + /* + * All other errors return failure, with the compressor + * specific error code in *error + */ + *error = res; + return -1; +} + + +static int xz_uncompress(void *dest, void *src, int size, int block_size, + int *error) +{ + size_t src_pos = 0; + size_t dest_pos = 0; + uint64_t memlimit = MEMLIMIT; + + lzma_ret res = lzma_stream_buffer_decode(&memlimit, 0, NULL, + src, &src_pos, size, dest, &dest_pos, block_size); + + *error = res; + return res == LZMA_OK && size == (int) src_pos ? (int) dest_pos : -1; +} + + +void xz_usage() +{ + fprintf(stderr, "\t -Xbcj filter1,filter2,...,filterN\n"); + fprintf(stderr, "\t\tCompress using filter1,filter2,...,filterN in"); + fprintf(stderr, " turn\n\t\t(in addition to no filter), and choose"); + fprintf(stderr, " the best compression.\n"); + fprintf(stderr, "\t\tAvailable filters: x86, arm, armthumb,"); + fprintf(stderr, " powerpc, sparc, ia64\n"); + fprintf(stderr, "\t -Xdict-size \n"); + fprintf(stderr, "\t\tUse as the XZ dictionary size. The"); + fprintf(stderr, " dictionary size\n\t\tcan be specified as a"); + fprintf(stderr, " percentage of the block size, or as an\n\t\t"); + fprintf(stderr, "absolute value. The dictionary size must be less"); + fprintf(stderr, " than or equal\n\t\tto the block size and 8192 bytes"); + fprintf(stderr, " or larger. It must also be\n\t\tstorable in the xz"); + fprintf(stderr, " header as either 2^n or as 2^n+2^(n+1).\n\t\t"); + fprintf(stderr, "Example dict-sizes are 75%%, 50%%, 37.5%%, 25%%, or"); + fprintf(stderr, " 32K, 16K, 8K\n\t\tetc.\n"); +} + + +struct compressor xz_comp_ops = { + .init = xz_init, + .compress = xz_compress, + .uncompress = xz_uncompress, + .options = xz_options, + .options_post = xz_options_post, + .dump_options = xz_dump_options, + .extract_options = xz_extract_options, + .usage = xz_usage, + .id = XZ_COMPRESSION, + .name = "xz", + .supported = 1 +}; diff --git a/squashfs-tools/xz_wrapper.h b/squashfs-tools/xz_wrapper.h new file mode 100644 index 0000000..36939b5 --- /dev/null +++ b/squashfs-tools/xz_wrapper.h @@ -0,0 +1,71 @@ +#ifndef XZ_WRAPPER_H +#define XZ_WRAPPER_H +/* + * Squashfs + * + * Copyright (c) 2010 + * Phillip Lougher + * + * This program is free software; you can redistribute it and/or + * modify it under the terms of the GNU General Public License + * as published by the Free Software Foundation; either version 2, + * or (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. + * + * xz_wrapper.h + * + */ + +#ifndef linux +#define __BYTE_ORDER BYTE_ORDER +#define __BIG_ENDIAN BIG_ENDIAN +#define __LITTLE_ENDIAN LITTLE_ENDIAN +#else +#include +#endif + +#if __BYTE_ORDER == __BIG_ENDIAN +extern unsigned int inswap_le32(unsigned int); + +#define SQUASHFS_INSWAP_COMP_OPTS(s) { \ + (s)->dictionary_size = inswap_le32((s)->dictionary_size); \ + (s)->flags = inswap_le32((s)->flags); \ +} +#else +#define SQUASHFS_INSWAP_COMP_OPTS(s) +#endif + +#define MEMLIMIT (32 * 1024 * 1024) + +struct bcj { + char *name; + lzma_vli id; + int selected; +}; + +struct filter { + void *buffer; + lzma_filter filter[3]; + size_t length; +}; + +struct xz_stream { + struct filter *filter; + int filters; + int dictionary_size; + lzma_options_lzma opt; +}; + +struct comp_opts { + int dictionary_size; + int flags; +}; +#endif