Lasse Collin [Wed, 3 Sep 2008 14:06:25 +0000 (17:06 +0300)]
Minor updates to the file format specification.
Lasse Collin [Tue, 2 Sep 2008 16:33:32 +0000 (19:33 +0300)]
Command line tool fixes
Lasse Collin [Tue, 2 Sep 2008 16:31:42 +0000 (19:31 +0300)]
Auto decoder cleanup
Lasse Collin [Tue, 2 Sep 2008 16:12:12 +0000 (19:12 +0300)]
Updated auto decoder to handle LZMA_CONCATENATED when decoding
LZMA_Alone files. Decoding of concatenated LZMA_Alone files is
intentionally not supported, so it is better to put this in
auto decoder than LZMA_Alone decoder.
Lasse Collin [Tue, 2 Sep 2008 11:56:52 +0000 (14:56 +0300)]
Stream decoder cleanups
Lasse Collin [Tue, 2 Sep 2008 08:45:39 +0000 (11:45 +0300)]
Some fixes to LZ encoder.
Lasse Collin [Sun, 31 Aug 2008 08:47:01 +0000 (11:47 +0300)]
Fix wrong pointer calculation in LZMA encoder.
Lasse Collin [Thu, 28 Aug 2008 19:53:15 +0000 (22:53 +0300)]
Sort of garbage collection commit. :-| Many things are still
broken. API has changed a lot and it will still change a
little more here and there. The command line tool doesn't
have all the required changes to reflect the API changes, so
it's easy to get "internal error" or trigger assertions.
Lasse Collin [Fri, 20 Jun 2008 14:16:32 +0000 (17:16 +0300)]
Fix test_filter_flags to match the new restriction of lc+lp.
Lasse Collin [Fri, 20 Jun 2008 13:19:54 +0000 (16:19 +0300)]
Remove some redundant code from LZMA encoder.
Lasse Collin [Thu, 19 Jun 2008 13:35:08 +0000 (16:35 +0300)]
Add limit of lc + lp <= 4. Now we can allocate the
literal coder as part of the main LZMA encoder or
decoder structure.
Make the LZMA decoder to rely on the current internal API
to free the allocated memory in case an error occurs.
Lasse Collin [Wed, 18 Jun 2008 18:45:19 +0000 (21:45 +0300)]
Comments
Lasse Collin [Wed, 18 Jun 2008 16:19:02 +0000 (19:19 +0300)]
Delete old code that was supposed to be already deleted
from test_block_header.c.
Lasse Collin [Wed, 18 Jun 2008 15:02:10 +0000 (18:02 +0300)]
Update the code to mostly match the new simpler file format
specification. Simplify things by removing most of the
support for known uncompressed size in most places.
There are some miscellaneous changes here and there too.
The API of liblzma has got many changes and still some
more will be done soon. While most of the code has been
updated, some things are not fixed (the command line tool
will choke with invalid filter chain, if nothing else).
Subblock filter is somewhat broken for now. It will be
updated once the encoded format of the Subblock filter
has been decided.
Lasse Collin [Tue, 17 Jun 2008 12:03:46 +0000 (15:03 +0300)]
Update the file format specification draft. The new one is
a lot simpler than the previous versions, but it also means
that the existing code will change a lot.
Lasse Collin [Wed, 11 Jun 2008 18:42:47 +0000 (21:42 +0300)]
Fix uninitialized variable in LZMA encoder. This was
introduced in
369f72fd656f537a9a8e06f13e6d0d4c242be22f.
Lasse Collin [Wed, 11 Jun 2008 12:08:44 +0000 (15:08 +0300)]
Improve command line integer parsing a little in lzma and
lzmadec to make them accept also KiB in addition Ki etc.
Fix also memory usage information in lzmadec --help.
Lasse Collin [Tue, 10 Jun 2008 17:36:12 +0000 (20:36 +0300)]
s/decompressed/compressed/ in the command line tool's
error message.
Lasse Collin [Sun, 1 Jun 2008 09:48:17 +0000 (12:48 +0300)]
Fix a buffer overflow in the LZMA encoder. It was due to my
misunderstanding of the code. There's no tiny fix for this
problem, so I also cleaned up the code in general.
This reduces the speed of the encoder 2-5 % in the fastest
compression mode ("lzma -1"). High compression modes should
have no noticeable performance difference.
This commit breaks things (especially LZMA_SYNC_FLUSH) but I
will fix them once the new format and LZMA2 has been roughly
implemented. Plain LZMA won't support LZMA_SYNC_FLUSH at all
and won't be supported in the new .lzma format. This may
change still but this is what it looks like now.
Support for known uncompressed size (that is, LZMA or LZMA2
without EOPM) is likely to go away. This means there will
be API changes.
Lasse Collin [Fri, 30 May 2008 08:53:41 +0000 (11:53 +0300)]
Typo fixes from meyering.
Lasse Collin [Sun, 11 May 2008 11:24:42 +0000 (14:24 +0300)]
Remove support for pre-C89 libc versions that lack memcpy,
memmove, and memset.
Lasse Collin [Sun, 11 May 2008 11:17:21 +0000 (14:17 +0300)]
Improved C99 compiler detection in configure.ac. It will
pass -std=gnu99 instead of -std=c99 to GCC now, but -pedantic
should still give warnings about GNU extensions like before
except with some special keywords like asm().
Lasse Collin [Tue, 6 May 2008 12:15:07 +0000 (15:15 +0300)]
Bunch of grammar fixes from meyering.
Lasse Collin [Tue, 6 May 2008 10:41:05 +0000 (13:41 +0300)]
Typo fix
Lasse Collin [Sun, 4 May 2008 19:29:27 +0000 (22:29 +0300)]
Don't print an error message on broken pipe unless --verbose
is used.
Lasse Collin [Wed, 30 Apr 2008 19:16:17 +0000 (22:16 +0300)]
Fix a crash with --format=alone if other filters than LZMA
are specified on the command line.
Lasse Collin [Mon, 28 Apr 2008 14:08:27 +0000 (17:08 +0300)]
Updated THANKS.
Lasse Collin [Mon, 28 Apr 2008 14:06:34 +0000 (17:06 +0300)]
Fixed wrong spelling "limitter" to "limiter". This affects
liblzma's API.
Lasse Collin [Fri, 25 Apr 2008 12:39:50 +0000 (15:39 +0300)]
Prevent LZ encoder from hanging with known uncompressed
size. The "fix" breaks LZMA_SYNC_FLUSH at end of stream
with known uncompressed size, but since it currently seems
likely that support for encoding with known uncompressed
size will go away anyway, I'm not fixing this problem now.
Lasse Collin [Fri, 25 Apr 2008 10:58:56 +0000 (13:58 +0300)]
Removed src/liblzma/common/sysdefs.h symlink, which was
annoying, because "make dist" put two copies of sysdefs.h
into the tarball instead of the symlink.
Lasse Collin [Fri, 25 Apr 2008 10:41:29 +0000 (13:41 +0300)]
Added memusage.c to debug directory.
Lasse Collin [Fri, 25 Apr 2008 10:32:35 +0000 (13:32 +0300)]
Bumped version number to 4.999.3alpha. It will become 5.0.0
once we have a stable release (won't be very soon). The
version number is no longer related to version of LZMA SDK.
Made some small Automake-related changes to toplevel
Makefile.am and configure.ac.
Lasse Collin [Thu, 24 Apr 2008 17:25:39 +0000 (20:25 +0300)]
Fix a memory leak by calling free(extra->data) in
lzma_extra_free().
Lasse Collin [Thu, 24 Apr 2008 17:23:05 +0000 (20:23 +0300)]
Make unlzma and lzcat symlinks.
Lasse Collin [Thu, 24 Apr 2008 17:20:27 +0000 (20:20 +0300)]
Fixed a bug in command line option parsing.
Lasse Collin [Thu, 24 Apr 2008 17:19:20 +0000 (20:19 +0300)]
Added two assert()s.
Lasse Collin [Thu, 24 Apr 2008 16:22:53 +0000 (19:22 +0300)]
Switch to uint16_t as the type of range coder probabilities.
Lasse Collin [Thu, 24 Apr 2008 15:39:57 +0000 (18:39 +0300)]
Fix wrong return type (uint32_t -> bool).
Lasse Collin [Thu, 24 Apr 2008 15:38:00 +0000 (18:38 +0300)]
Fix data corruption in LZ encoder with LZMA_SYNC_FLUSH.
Lasse Collin [Thu, 24 Apr 2008 14:33:01 +0000 (17:33 +0300)]
Fix fastpos problem in Makefile.am when built with --enable-small.
Lasse Collin [Thu, 24 Apr 2008 14:30:51 +0000 (17:30 +0300)]
Use 64-bit integer as range encoder's cache size. This fixes a
theoretical data corruption, which should be very hard to trigger
even intentionally.
Lasse Collin [Mon, 24 Mar 2008 14:38:40 +0000 (16:38 +0200)]
Replaced the range decoder optimization that used arithmetic
right shift with as fast version that doesn't need
arithmetic right shift. Removed the related check from
configure.ac.
Lasse Collin [Sat, 22 Mar 2008 12:39:34 +0000 (14:39 +0200)]
Take advantage of arithmetic right shift in range decoder.
Lasse Collin [Sat, 22 Mar 2008 12:18:29 +0000 (14:18 +0200)]
Added autoconf check to detect if we can use arithmetic
right shift for optimizations.
Lasse Collin [Fri, 21 Mar 2008 23:26:36 +0000 (01:26 +0200)]
Update a comment to use the variable name rep_len_decoder.
(And BTW, the previous commit actually did change the
program logic slightly.)
Lasse Collin [Fri, 21 Mar 2008 22:57:33 +0000 (00:57 +0200)]
Demystified the "state" variable in LZMA code. Use the
word literal instead of char for better consistency.
There are still some names with _char instead of _literal
in lzma_optimum, these may be changed later.
Renamed length coder variables.
This commit doesn't change the program logic.
Lasse Collin [Fri, 14 Mar 2008 21:16:11 +0000 (23:16 +0200)]
Fix data corruption in LZMA encoder. Note that this bug was
specific to liblzma and was *not* present in LZMA SDK.
Lasse Collin [Fri, 14 Mar 2008 19:32:37 +0000 (21:32 +0200)]
Fix a comment API header.
Lasse Collin [Wed, 12 Mar 2008 21:14:50 +0000 (23:14 +0200)]
Make lzma_stream.next_in const. Let's see if anyone complains.
Lasse Collin [Tue, 11 Mar 2008 13:35:34 +0000 (15:35 +0200)]
Apply a minor speed optimization to LZMA decoder.
Lasse Collin [Tue, 11 Mar 2008 13:17:16 +0000 (15:17 +0200)]
Initialize the last byte of the dictionary to zero so that
lz_get_byte(lz, 0) returns zero. This was broken by
1a3b21859818e4d8e89a1da99699233c1bfd197d.
Lasse Collin [Mon, 10 Mar 2008 13:57:55 +0000 (15:57 +0200)]
Really fix the price count initialization.
Lasse Collin [Mon, 10 Mar 2008 11:47:17 +0000 (13:47 +0200)]
Updated THANKS.
Lasse Collin [Mon, 10 Mar 2008 11:46:48 +0000 (13:46 +0200)]
Initialize align_price_count and match_price_count in
lzma_encoder_init.c. While we don't call
fill_distances_prices() and fill_align_prices() in
lzma_lzma_encoder_init(), we still need to initialize
these two variables so that the fill functions get
called in lzma_encoder_getoptimum.c in the beginning
of a stream.
Lasse Collin [Mon, 10 Mar 2008 11:44:29 +0000 (13:44 +0200)]
Always initialize lz->temp_size in lz_decoder.c. temp_size did
get initialized as a side-effect after allocating a new decoder,
but not when the decoder was reused.
Lasse Collin [Mon, 10 Mar 2008 11:41:25 +0000 (13:41 +0200)]
Don't fill allocated memory with 0xFD when debugging is
enabled. It hides errors from Valgrind.
Lasse Collin [Thu, 28 Feb 2008 08:24:31 +0000 (10:24 +0200)]
Remove two redundant validity checks from the LZMA decoder.
These are already checked elsewhere, so omitting these
gives (very) tiny speed up.
Lasse Collin [Wed, 6 Feb 2008 11:25:32 +0000 (13:25 +0200)]
Tiny clean up to file-format.txt.
Lasse Collin [Sat, 2 Feb 2008 12:51:06 +0000 (14:51 +0200)]
Don't memzero() the history buffer when initializing LZ
decoder. There's no danger of information leak here, so
it isn't required. Doing memzero() takes a lot of time
with large dictionaries, which could make it easier to
construct DoS attack to consume too much CPU time.
Lasse Collin [Fri, 1 Feb 2008 06:39:26 +0000 (08:39 +0200)]
Do uncompressed size validation in raw encoder. This way
it gets done for not only raw encoder, but also Block
and LZMA_Alone encoders.
Lasse Collin [Fri, 1 Feb 2008 06:32:05 +0000 (08:32 +0200)]
Avoid unneeded function call in raw_common.c.
Lasse Collin [Sat, 26 Jan 2008 19:42:38 +0000 (21:42 +0200)]
Updated THANKS.
Lasse Collin [Sat, 26 Jan 2008 19:40:23 +0000 (21:40 +0200)]
Added note.GNU-stack to x86 assembler files. It is needed
when using non-executable stack.
Lasse Collin [Sat, 26 Jan 2008 17:12:50 +0000 (19:12 +0200)]
Added api/lzma/easy.h. I had forgot to add this to the
git repo. Thanks to Stephan Kulow.
Lasse Collin [Sat, 26 Jan 2008 09:09:17 +0000 (11:09 +0200)]
Added more test files.
Lasse Collin [Sat, 26 Jan 2008 08:47:55 +0000 (10:47 +0200)]
Added more test files.
Lasse Collin [Fri, 25 Jan 2008 22:25:34 +0000 (00:25 +0200)]
Return LZMA_HEADER_ERROR if LZMA_SYNC_FLUSH is used with any
of the so called simple filters. If there is demand, limited
support for LZMA_SYNC_FLUSH may be added in future.
After this commit, using LZMA_SYNC_FLUSH shouldn't cause
undefined behavior in any situation.
Lasse Collin [Fri, 25 Jan 2008 21:50:35 +0000 (23:50 +0200)]
Added more Multi-Block test files. Improved some
descriptions in the test files' README.
Lasse Collin [Fri, 25 Jan 2008 21:12:36 +0000 (23:12 +0200)]
Combine lzma_options_block validation needed by both Block
encoder and decoder, and put the shared things to
block_private.h. Improved the checks a little so that
they may detect too big Compressed Size at initialization
time if lzma_options_block.total_size or .total_limit is
known.
Allow encoding and decoding Blocks with combinations of
fields that are not allowed by the file format specification.
Doing this requires that the application passes such a
combination in lzma_options_lzma; liblzma doesn't do that,
but it's not impossible that someone could find them useful
in some custom file format.
Lasse Collin [Fri, 25 Jan 2008 17:21:22 +0000 (19:21 +0200)]
Added test_memlimit.c.
Lasse Collin [Fri, 25 Jan 2008 17:20:28 +0000 (19:20 +0200)]
Improved the memory limitter:
- Added lzma_memlimit_max() and lzma_memlimit_reached()
API functions.
- Added simple estimation of malloc()'s memory usage
overhead.
- Fixed integer overflow detection in lzma_memlimit_alloc().
- Made some white space cleanups and added more comments.
The description of lzma_memlimit_max() in memlimit.h is bad
and should be improved.
Lasse Collin [Fri, 25 Jan 2008 11:55:52 +0000 (13:55 +0200)]
Use more parenthesis in succeed() macro in tests/tests.h.
Lasse Collin [Thu, 24 Jan 2008 12:49:34 +0000 (14:49 +0200)]
Added more Multi-Block Stream test files.
Lasse Collin [Wed, 23 Jan 2008 22:46:05 +0000 (00:46 +0200)]
Added bunch of test files containing Multi-Block Streams.
Lasse Collin [Wed, 23 Jan 2008 21:43:00 +0000 (23:43 +0200)]
Fix decoding of empty Metadata Blocks, that don't have
even the Metadata Flags field. Earlier the code allowed
such files; now they are prohibited as the file format
specification requires.
Lasse Collin [Wed, 23 Jan 2008 21:38:18 +0000 (23:38 +0200)]
Fix a bug related to
99e12af4e2b866c011fe0106cd1e0bfdcc8fe9c6.
lzma_metadata.header_metadata_size was not properly set to
zero if the Metadata had only the Metadata Flags field.
Lasse Collin [Wed, 23 Jan 2008 21:35:49 +0000 (23:35 +0200)]
Fix decoding of Extra Records that have empty Data.
Lasse Collin [Wed, 23 Jan 2008 20:02:38 +0000 (22:02 +0200)]
Add the trailing '\0' to lzma_extra.data as the API header
already documents.
Lasse Collin [Wed, 23 Jan 2008 19:21:21 +0000 (21:21 +0200)]
Added debug/full_flush.c.
Lasse Collin [Wed, 23 Jan 2008 19:05:33 +0000 (21:05 +0200)]
Return LZMA_STREAM_END instead of LZMA_OK if
LZMA_SYNC_FLUSH or LZMA_FULL_FLUSH is used when
there's no unfinished Block open.
Lasse Collin [Wed, 23 Jan 2008 18:05:01 +0000 (20:05 +0200)]
Added bad-single-none-footer_filter_flags.lzma and
bad-single-none-too_long_vli.lzma.
Lasse Collin [Wed, 23 Jan 2008 11:42:35 +0000 (13:42 +0200)]
Fixed a typo.
Lasse Collin [Wed, 23 Jan 2008 11:40:45 +0000 (13:40 +0200)]
Fix a memory leak in the Subblock encoder.
Lasse Collin [Wed, 23 Jan 2008 11:36:07 +0000 (13:36 +0200)]
Fix Size of Header Metadata Block handling. Now
lzma_metadata.header_metadata_size == LZMA_VLI_VALUE_UNKNOWN
is not allowed at all. To indicate missing Header Metadata
Block, header_metadata_size must be set to zero. This is
what Metadata decoder does after this patch too.
Note that other missing fields in lzma_metadata are still
indicated with LZMA_VLI_VALUE_UNKNOWN. This isn't as
illogical as it sounds at first, because missing Size of
Header Metadata Block means that Header Metadata Block is
not present in the Stream. With other Metadata fields,
a missing field means only that the value is unknown.
Lasse Collin [Wed, 23 Jan 2008 11:15:55 +0000 (13:15 +0200)]
Fix a memory leak in metadata_decoder.c.
Lasse Collin [Wed, 23 Jan 2008 11:13:58 +0000 (13:13 +0200)]
Fix the fix
863028cb7ad6d8d0455fa69348f56b376d7b908f which
just moved to problem. Now it's really fixed.
Lasse Collin [Tue, 22 Jan 2008 22:21:04 +0000 (00:21 +0200)]
Take advantage of return_if_error() macro in
lzma_info_metadata_set() in info.c.
Lasse Collin [Tue, 22 Jan 2008 22:18:32 +0000 (00:18 +0200)]
Fixed a dangling pointer that caused invalid free().
Lasse Collin [Tue, 22 Jan 2008 20:49:24 +0000 (22:49 +0200)]
Added lzma_easy_* functions. These should make using
liblzma as easy as using zlib, because the easy API
don't require developers to know any fancy LZMA options.
Note that Multi-Block Stream encoding is currently broken.
The easy API should be OK, the bug(s) are elsewhere.
Lasse Collin [Tue, 22 Jan 2008 19:16:22 +0000 (21:16 +0200)]
Fix Multi-Block Stream encoder's EOPM usage.
Lasse Collin [Mon, 21 Jan 2008 22:15:11 +0000 (00:15 +0200)]
Made lzma_extra pointers const in lzma_options_stream.
Lasse Collin [Sun, 20 Jan 2008 18:15:21 +0000 (20:15 +0200)]
Updated debug/sync_flush.c.
Lasse Collin [Sun, 20 Jan 2008 18:14:26 +0000 (20:14 +0200)]
Added debug/repeat.c.
Lasse Collin [Sun, 20 Jan 2008 18:12:58 +0000 (20:12 +0200)]
Fix alignment handling bugs in Subblock encoder.
This leaves one known alignment bug unfixed: If repeat count
doesn't fit into 28-bit integer, the encoder has to split
this to multiple Subblocks with Subblock Type `Repeating Data'.
The extra Subblocks may have wrong alignment. Correct alignment
is restored after the split Repeating Data has been completely
written out.
Since the encoder doesn't even try to fix the alignment unless
the size of Data is at least 4 bytes, to trigger this bug you
need at least 4 GiB of repeating data with sequence length of
4 or more bytes. Since the worst thing done by this bug is
misaligned data (no data corruption), this bug simply isn't
worth fixing, because a proper fix isn't simple.
Lasse Collin [Sat, 19 Jan 2008 19:16:33 +0000 (21:16 +0200)]
Implemented LZMA_SYNC_FLUSH support to the Subblock encoder.
The API for handing Subfilters was changed to make it
consistent with LZMA_SYNC_FLUSH.
A few sanity checks were added for Subfilter handling. Some
small bugs were fixed. More comments were added.
Lasse Collin [Sat, 19 Jan 2008 13:19:21 +0000 (15:19 +0200)]
Revised the Delta filter implementation. The initialization
function is still shared between encoder and decoder, but the
actual coding is in separate files for encoder and decoder.
There are now separate functions for the actual delta
calculation depending on if Delta is the last filter in the
chain or not. If it is the last, the new code copies the
data from input to output buffer and does the delta
calculation at the same time. The old code first copied the
data, then did the delta in the target buffer, which required
reading through the data twice.
Support for LZMA_SYNC_FLUSH was added to the Delta encoder.
This doesn't change anything in the file format.
Lasse Collin [Fri, 18 Jan 2008 18:18:08 +0000 (20:18 +0200)]
Added the debug directory and the first debug tool
(sync_flush). These tools are not built unless the
user runs "make" in the debug directory.
Lasse Collin [Fri, 18 Jan 2008 18:13:00 +0000 (20:13 +0200)]
Added test files to test usage of flush marker in LZMA.
Lasse Collin [Fri, 18 Jan 2008 18:02:52 +0000 (20:02 +0200)]
Fix LZMA_SYNC_FLUSH handling in LZ and LZMA encoders.
That code is now almost completely in LZ coder, where
it can be shared with other LZ77-based algorithms in
future.
Lasse Collin [Fri, 18 Jan 2008 15:21:24 +0000 (17:21 +0200)]
Don't add -g to CFLAGS when --enable-debug is specified.
It's the job of the user to put that in CFLAGS.