Yann Collet [Wed, 2 May 2018 20:57:33 +0000 (13:57 -0700)]
removed test that might be optimized away
under UB rule "no overflow on int"
Yann Collet [Wed, 2 May 2018 19:56:37 +0000 (12:56 -0700)]
introduce LZ4_decoderRingBufferSize()
fuzzer : fix and robustify ring buffer tests
Yann Collet [Wed, 2 May 2018 17:33:12 +0000 (10:33 -0700)]
simplify shortcut
Yann Collet [Wed, 2 May 2018 17:08:30 +0000 (10:08 -0700)]
Merge branch 'dev' into complexShortcut
Yann Collet [Wed, 2 May 2018 17:06:07 +0000 (10:06 -0700)]
Merge pull request #525 from lz4/testDecMerge
added a test case for LZ4_decompress_fast_usingDict …
Yann Collet [Tue, 1 May 2018 03:40:34 +0000 (20:40 -0700)]
Merge pull request #521 from lz4/BD_deterministic
fix lz4hc -BD non-determinism
Cyan4973 [Tue, 1 May 2018 01:56:16 +0000 (18:56 -0700)]
renamed variable for clarity
lowLimit -> lowestMatchIndex
Cyan4973 [Tue, 1 May 2018 01:50:56 +0000 (18:50 -0700)]
added visual test dir to .gitignore
Yann Collet [Mon, 30 Apr 2018 23:08:16 +0000 (16:08 -0700)]
lz4hc changed variable
to reduce confusion
dictLowLimit => dictStart
Yann Collet [Mon, 30 Apr 2018 22:55:33 +0000 (15:55 -0700)]
clarified streaming decompression function
restrictions for ring buffer
Yann Collet [Mon, 30 Apr 2018 22:32:37 +0000 (15:32 -0700)]
Merge pull request #527 from svpv/fastDec
lz4.c: two-stage shortcut for LZ4_decompress_generic
Yann Collet [Sun, 29 Apr 2018 15:56:44 +0000 (08:56 -0700)]
Merge pull request #523 from svpv/makeV1
lib/Makefile: show commands with V=1
Cyan4973 [Sun, 29 Apr 2018 15:47:08 +0000 (08:47 -0700)]
Merge branch 'dev' of github.com:lz4/lz4 into dev
Cyan4973 [Sun, 29 Apr 2018 15:46:39 +0000 (08:46 -0700)]
updated NEWS for v1.8.2
mentioning work from @svpv
Yann Collet [Sun, 29 Apr 2018 15:45:16 +0000 (08:45 -0700)]
Merge pull request #526 from svpv/makeV1
lib/Makefile: show commands with V=1
Cyan4973 [Sun, 29 Apr 2018 14:42:24 +0000 (07:42 -0700)]
added a test case for LZ4_decompress_fast_usingDict
with a separated dictionary
since a joined dictionary is now detected as prefix64K.
Also : fixed a minor warning under msys
Yann Collet [Sun, 29 Apr 2018 14:41:35 +0000 (07:41 -0700)]
Merge pull request #515 from svpv/refactorDec
lz4.c: refactor the decoding routines
Cyan4973 [Sat, 28 Apr 2018 17:42:52 +0000 (10:42 -0700)]
ignore windows+msys artefacts
Alexey Tourbin [Sat, 28 Apr 2018 08:14:40 +0000 (11:14 +0300)]
lz4.c: two-stage shortcut for LZ4_decompress_generic
Alexey Tourbin [Sat, 28 Apr 2018 04:16:46 +0000 (07:16 +0300)]
lib/Makefile: show commands with V=1
`make V=1` will now show the commands executed to build the library.
A similar technique is used in e.g. linux/Makefile.
The bulk of this change is produced with the following vim command:
:g!/^\t@echo\>/s/^\t@/\t\$(Q)/
Yann Collet [Sat, 28 Apr 2018 00:22:06 +0000 (17:22 -0700)]
Merge pull request #522 from svpv/refactorDec
Refactor dec
Yann Collet [Fri, 27 Apr 2018 20:52:30 +0000 (13:52 -0700)]
Merge pull request #520 from felixhandte/frame-dict-nits
Minor Fixes to Dictionary Preparation in LZ4 Frame
Yann Collet [Fri, 27 Apr 2018 19:59:20 +0000 (12:59 -0700)]
Merge branch 'dev' into BD_deterministic
Yann Collet [Fri, 27 Apr 2018 19:46:49 +0000 (12:46 -0700)]
fix lz4hc -BD non-determinism
related to chain table update
Yann Collet [Fri, 27 Apr 2018 18:44:47 +0000 (11:44 -0700)]
lz4hc : minor editions for clarity
Yann Collet [Fri, 27 Apr 2018 18:46:29 +0000 (11:46 -0700)]
Merge pull request #519 from lz4/fdParser
Faster decoding speed
W. Felix Handte [Fri, 27 Apr 2018 18:10:27 +0000 (14:10 -0400)]
Avoid Possibly Redundant Table Clears When Loading HC Dict
W. Felix Handte [Fri, 27 Apr 2018 17:59:02 +0000 (13:59 -0400)]
Remove Redundant LZ4_resetStream() Call
W. Felix Handte [Fri, 27 Apr 2018 17:57:10 +0000 (13:57 -0400)]
Rename LZ4F_applyCDict() -> LZ4F_initStream()
Yann Collet [Fri, 27 Apr 2018 16:04:09 +0000 (09:04 -0700)]
ensure favorDecSpeed is properly initialized
also :
- fix a potential malloc error
- proper use of ALLOC macro inside lz4hc
- update html API doc
Yann Collet [Fri, 27 Apr 2018 15:43:40 +0000 (08:43 -0700)]
updated NEWS, in preparation for v1.8.2
Alexey Tourbin [Fri, 27 Apr 2018 12:00:11 +0000 (15:00 +0300)]
lz4.c: fixed the LZ4_decompress_fast_continue case
The change is very similar to that of the LZ4_decompress_safe_continue
case. The only reason a make this a separate change is to ensure that
the fuzzer, after it's been enhanced, can detect the flaw in
LZ4_decompress_fast_continue, and that the change indeed fixes the flaw.
Alexey Tourbin [Fri, 27 Apr 2018 04:06:37 +0000 (07:06 +0300)]
fuzzer.c: enabled ring buffer tests for decompress_fast
Ring buffer tests were performed only with LZ4_decompress_safe_continue,
leaving my buggy changes to LZ4_decompress_safe_continue undetected.
The tests are now replicated and performed in a similar manner for both
LZ4_decompress_safe_continue and LZ4_decompress_safe_continue (except
for the small buffer case where only one function can be tested,
because part of the dictionary is overwritten with the output).
I also updated function names in the messages (changed them to the
actual ones). The error was reported for LZ4_decompress_safe(),
which I found misleading.
Yann Collet [Fri, 27 Apr 2018 00:02:20 +0000 (17:02 -0700)]
fixed a number of minor cast warnings
Yann Collet [Thu, 26 Apr 2018 23:47:50 +0000 (16:47 -0700)]
Merge pull request #518 from felixhandte/fix-517-dict-size-truncation
Limit Dictionary Size During LZ4F Decompression
Yann Collet [Thu, 26 Apr 2018 23:40:33 +0000 (16:40 -0700)]
Merge pull request #516 from felixhandte/merge-dest-size
Merge _destSize Compress Variant into LZ4_compress_generic()
Yann Collet [Thu, 26 Apr 2018 22:49:32 +0000 (15:49 -0700)]
fasterDecSpeed can be triggered from cli with --favor-decSpeed
Yann Collet [Thu, 26 Apr 2018 22:18:44 +0000 (15:18 -0700)]
favorDecSpeed feature can be triggered from lz4frame
and lz4hc.
W. Felix Handte [Thu, 26 Apr 2018 19:42:16 +0000 (15:42 -0400)]
Merge _destSize Compress Variant into LZ4_compress_generic()
W. Felix Handte [Thu, 26 Apr 2018 21:25:12 +0000 (17:25 -0400)]
Add _destSize() to Fullbench
W. Felix Handte [Thu, 26 Apr 2018 20:53:40 +0000 (16:53 -0400)]
Limit Dictionary Size During LZ4F Decompression
Fixes lz4/lz4#517.
Yann Collet [Thu, 26 Apr 2018 20:01:59 +0000 (13:01 -0700)]
introduced ability to parse for decompression speed
triggered through an enum.
Now, it's still necessary to properly expose this capability
all the way up to the cli.
Alexey Tourbin [Thu, 26 Apr 2018 04:46:26 +0000 (07:46 +0300)]
lz4.c: fixed the LZ4_decompress_safe_continue case
The previous change broke decoding with a ring buffer. That's because
I didn't realize that the "double dictionary mode" was possible, i.e.
that the decoding routine can look both at the first part of the
dictionary passed as prefix and the second part passed via dictStart+dictSize.
So this change introduces the LZ4_decompress_safe_doubleDict helper,
which handles this "double dictionary" situation. (This is a bit of
a misnomer, there is only one dictionary, but I can't think of a better
name, and perhaps the designation is not all too bad.) The helper is
used only once, in LZ4_decompress_safe_continue, it should be inlined
with LZ4_FORCE_O2_GCC_PPC64LE attached to LZ4_decompress_safe_continue.
(Also, in the helper functions, I change the dictStart parameter type
to "const void*", to avoid a cast when calling helpers. In the helpers,
the upcast to "BYTE*" is still required, for compatibility with C++.)
So this fixes the case of LZ4_decompress_safe_continue, and I'm
surprised by the fact that the fuzzer is now happy and does not detect
a similar problem with LZ4_decompress_fast_continue. So before fixing
LZ4_decompress_fast_continue, the next logical step is to enhance
the fuzzer.
Cyan4973 [Wed, 25 Apr 2018 13:42:57 +0000 (06:42 -0700)]
minor edit of block format
clarifying parsing restrictions near end of block.
Yann Collet [Wed, 25 Apr 2018 13:13:08 +0000 (06:13 -0700)]
Merge pull request #514 from svpv/clarifyBlockFormat
lz4_Block_format.md: clarify on short inputs and restrictions
Alexey Tourbin [Mon, 23 Apr 2018 05:37:44 +0000 (08:37 +0300)]
lz4.c: refactor the decoding routines
I noticed that LZ4_decompress_generic is sometimes instantiated with
identical set of parameters, or (what's worse) with a subtly different
sets of parameters. For example, LZ4_decompress_fast_withPrefix64k is
instantiated as follows:
return LZ4_decompress_generic(source, dest, 0, originalSize, endOnOutputSize,
full, 0, withPrefix64k, (BYTE*)dest - 64 KB, NULL, 64 KB);
while the equivalent withPrefix64k call in LZ4_decompress_usingDict_generic
passes 0 for the last argument instead of 64 KB. It turns out that there
is no difference in this case: if you change 64 KB to 0 KB in
LZ4_decompress_fast_withPrefix64k, you get the same binary code.
Moreover, because it's been clarified that LZ4_decompress_fast doesn't
check match offsets, it is now obvious that both of these fast/withPrefix64k
instantiations are simply redundant. Exactly because LZ4_decompress_fast
doesn't check offsets, it serves well with any prefixed dictionary.
There's a difference, though, with LZ4_decompress_safe_withPrefix64k.
It also passes 64 KB as the last argument, and if you change that to 0,
as in LZ4_decompress_usingDict_generic, you get a completely different
binary code. It seems that passing 0 enables offset checking:
const int checkOffset = ((safeDecode) && (dictSize < (int)(64 KB)));
However, the resulting code seems to run a bit faster. How come
enabling extra checks can make the code run faster? Curiouser and
curiouser! This needs extra study. Currently I take the view that
the dictSize should be set to non-zero when nothing else will do,
i.e. when passing the external dictionary via dictStart. Otherwise,
lowPrefix betrays just enough information about the dictionary.
* * *
Anyway, with this change, I instantiate all the necessary cases as
functions with distinctive names, which also take fewer arguments and
are therefore less error-prone. I also make the functions non-inline.
(The compiler won't inline the functions because they are used more than
once. Hence I attach LZ4_FORCE_O2_GCC_PPC64LE to the instances while
removing from the callers.) The number of instances is now is reduced
from 18 (safe+fast+partial+4*continue+4*prefix+4*dict+2*prefix64+forceExtDict)
down to 7 (safe+fast+partial+2*prefix+2*dict). The size of the code is
not the only issue here. Separate helper function are much more
amenable to profile-guided optimization: it is enough to profile only
a few basic functions, while the other less-often used functions, such
as LZ4_decompress_*_continue, will benefit automatically.
This is the list of LZ4_decompress* functions in liblz4.so, sorted by size.
Exported functions are marked with a capital T.
$ nm -S lib/liblz4.so |grep -wi T |grep LZ4_decompress |sort -k2
0000000000016260 0000000000000005 T LZ4_decompress_fast_withPrefix64k
0000000000016dc0 0000000000000025 T LZ4_decompress_fast_usingDict
0000000000016d80 0000000000000040 T LZ4_decompress_safe_usingDict
0000000000016d10 000000000000006b T LZ4_decompress_fast_continue
0000000000016c70 000000000000009f T LZ4_decompress_safe_continue
00000000000156c0 000000000000059c T LZ4_decompress_fast
0000000000014a90 00000000000005fa T LZ4_decompress_safe
0000000000015c60 00000000000005fa T LZ4_decompress_safe_withPrefix64k
0000000000002280 00000000000005fa t LZ4_decompress_safe_withSmallPrefix
0000000000015090 000000000000062f T LZ4_decompress_safe_partial
0000000000002880 00000000000008ea t LZ4_decompress_fast_extDict
0000000000016270 0000000000000993 t LZ4_decompress_safe_forceExtDict
Yann Collet [Tue, 24 Apr 2018 23:40:13 +0000 (16:40 -0700)]
Merge pull request #513 from felixhandte/integrate-static-frame-functions
Integrate Contents of `lz4frame_static.h` into `lz4frame.h`
Alexey Tourbin [Tue, 24 Apr 2018 22:40:12 +0000 (01:40 +0300)]
lz4_Block_format.md: clarify on short inputs and restrictions
It occurred to me that the formula "The last 5 bytes are always
literals", on the list of "assumptions made by the decoder", is
remarkably ambiguous. Suppose the decoder is presented with 5 bytes.
Are they literals? It may seem that the decoder degenerates
to memcpy on short inputs. But of course the answer is no,
so the formula needs some clarification.
Parsing restrictions should be explained as well, otherwise they look
like arbitrary numbers. The 5-byte restriction has been mentioned
recently in connection with the shortcut in LZ4_decompress_generic,
so I add that. The second restriction is left to be explained
by the author.
I also took the liberty to explain that empty inputs "are either
unrepresentable or can be represented with a null byte". This wording
may actually have some merit: it leaves for the implementation,
as opposed to the spec, to decide whether the encoder can compress
empty inputs, and whether the decoder can produce an empty output
(which the implementation should further clarify).
W. Felix Handte [Tue, 24 Apr 2018 22:50:03 +0000 (18:50 -0400)]
Multiply-Include Header to Check Guard Macro Correctness
W. Felix Handte [Sat, 21 Apr 2018 00:55:38 +0000 (20:55 -0400)]
Change Over Includes in the Project
W. Felix Handte [Fri, 20 Apr 2018 22:41:41 +0000 (18:41 -0400)]
Integrate lz4frame_static.h Declarations into lz4frame.h
Yann Collet [Tue, 24 Apr 2018 20:18:40 +0000 (13:18 -0700)]
Merge pull request #512 from lz4/HC_dict
In-place unmutable dictionaries for LZ4HC
Yann Collet [Tue, 24 Apr 2018 18:25:57 +0000 (11:25 -0700)]
Merge pull request #511 from lz4/decFast
Fixed performance issue with LZ4_decompress_fast()
Yann Collet [Tue, 24 Apr 2018 17:49:41 +0000 (10:49 -0700)]
Merge pull request #488 from felixhandte/hc-dict-ctx
Use Dictionary In-Place in HC Mode
W. Felix Handte [Fri, 20 Apr 2018 01:00:19 +0000 (21:00 -0400)]
Remove Debug Log Statements
W. Felix Handte [Thu, 19 Apr 2018 22:56:01 +0000 (18:56 -0400)]
Remove the Framebench Tool
W. Felix Handte [Tue, 24 Apr 2018 15:55:53 +0000 (11:55 -0400)]
Revert Stream Size Const to Correct Value
Yann Collet [Tue, 24 Apr 2018 06:44:04 +0000 (23:44 -0700)]
Merge pull request #504 from baruchsiach/static-only-support
lib: allow to disable shared libraries
Cyan4973 [Tue, 24 Apr 2018 02:26:02 +0000 (19:26 -0700)]
re-ordered parenthesis
to avoid mixing && and &
as suggested by @terrelln
Yann Collet [Mon, 23 Apr 2018 22:55:56 +0000 (15:55 -0700)]
Merge pull request #507 from lz4/clangPerf
fixed lz4_fast clang performance
Cyan4973 [Mon, 23 Apr 2018 22:52:44 +0000 (15:52 -0700)]
fixed minor declaration issue with clang on msys
Cyan4973 [Mon, 23 Apr 2018 22:47:08 +0000 (15:47 -0700)]
disable shortcut for LZ4_decompress_fast()
improving speed
Cyan4973 [Mon, 23 Apr 2018 22:42:27 +0000 (15:42 -0700)]
fullbench compiled without assert()
to better reflect release speed
Yann Collet [Mon, 23 Apr 2018 22:28:19 +0000 (15:28 -0700)]
Merge pull request #510 from terrelln/bug-fix
Fix input size validation edge cases
Nick Terrell [Mon, 23 Apr 2018 21:21:02 +0000 (14:21 -0700)]
Fix compilation error and assert.
Nick Terrell [Mon, 23 Apr 2018 20:14:19 +0000 (13:14 -0700)]
Fix input size validation edge cases
The bug is a read up to 2 bytes past the end of the buffer.
There are three cases for this bug, one for each test case added.
* An empty input causes `token = *ip++` to read one byte too far.
* A one byte input with `(token >> ML_BITS) == RUN_MASK` causes
one extra byte to be read without validation. This could be
combined with the first bug to cause 2 extra bytes to be read.
* The case pointed out in issue #508, where `ip == iend` at the
beginning of the loop after taking the shortcut.
Benchmarks show no regressions on clang or gcc-7 on both my mac
and devserver.
Fixes #508.
Yann Collet [Mon, 23 Apr 2018 02:30:24 +0000 (19:30 -0700)]
Merge pull request #509 from svpv/clarifyFastRisks
lz4.h: clarify the risks of using LZ4_decompress_fast()
Alexey Tourbin [Sun, 22 Apr 2018 22:43:30 +0000 (01:43 +0300)]
lz4.h: clarify the risks of using LZ4_decompress_fast()
The notes about "security guarantee" and "malicious inputs" seemed
a bit non-technical to me, so I took the liberty to tone them down
and instead describe the actual risks in technical terms. Namely,
the function never writes past the end of the output buffer, so
a direct hostile takeover (resulting in arbitrary code execution
soon after the return from the function) is not possible. However,
the application can crash because of reads from unmapped pages.
I also took the liberty to describe what I believe is the only sensible
usage scenario for the function: "This function is only usable if the
originalSize of uncompressed data is known in advance," etc.
Cyan4973 [Sat, 21 Apr 2018 07:11:51 +0000 (00:11 -0700)]
fixed incorrect comment
Yann Collet [Sat, 21 Apr 2018 01:09:51 +0000 (18:09 -0700)]
fixed clang performance in lz4_fast
The simple change from
`matchIndex+MAX_DISTANCE < current`
towards
`current - matchIndex > MAX_DISTANCE`
is enough to generate a 10% performance drop under clang.
Quite massive.
(I missed as my eyes were concentrated on gcc performance at that time).
The second version is more robust, because it also survives a situation where
`matchIndex > current`
due to overflows.
The first version requires matchIndex to not overflow.
Hence were added `assert()` conditions.
The only case where this can happen is with dictCtx compression,
in the case where the dictionary context is not initialized before loading the dictionary.
So it's enough to always initialize the context while loading the dictionary.
W. Felix Handte [Sat, 21 Apr 2018 00:18:30 +0000 (20:18 -0400)]
Change vLimit Calculation
W. Felix Handte [Sat, 21 Apr 2018 00:14:12 +0000 (20:14 -0400)]
Remove Redundant Static Assert
W. Felix Handte [Fri, 20 Apr 2018 23:37:28 +0000 (19:37 -0400)]
Simpler loadDict() Reset
W. Felix Handte [Fri, 20 Apr 2018 23:37:07 +0000 (19:37 -0400)]
Tolerate Base Pointer Underflow
W. Felix Handte [Fri, 20 Apr 2018 23:35:51 +0000 (19:35 -0400)]
Don't Segfault on Malloc Failure
W. Felix Handte [Fri, 20 Apr 2018 21:56:26 +0000 (17:56 -0400)]
Sign-Extend -1 to Pointer Width
W. Felix Handte [Fri, 20 Apr 2018 21:13:40 +0000 (17:13 -0400)]
Fix Constant Value
W. Felix Handte [Fri, 20 Apr 2018 21:10:47 +0000 (17:10 -0400)]
Handle Index Underflows Safely
W. Felix Handte [Fri, 20 Apr 2018 19:30:08 +0000 (15:30 -0400)]
Consts and Asserts and Other Minor Nits
W. Felix Handte [Fri, 20 Apr 2018 19:16:41 +0000 (15:16 -0400)]
Add Some Simple Fuzzer Tests
W. Felix Handte [Fri, 20 Apr 2018 19:00:53 +0000 (15:00 -0400)]
Add Comments on New Public APIs
W. Felix Handte [Fri, 20 Apr 2018 18:52:13 +0000 (14:52 -0400)]
Add API for Attaching Dictionaries
W. Felix Handte [Fri, 20 Apr 2018 18:16:27 +0000 (14:16 -0400)]
Also Reset the Chain Table
W. Felix Handte [Fri, 20 Apr 2018 18:08:06 +0000 (14:08 -0400)]
Remove inputBuffer from Context, Work Around its Absence
W. Felix Handte [Fri, 20 Apr 2018 17:14:37 +0000 (13:14 -0400)]
Remove Commented Out Support for Match Continuation over Segment Boundary
W. Felix Handte [Thu, 19 Apr 2018 23:36:34 +0000 (19:36 -0400)]
Fix Signedness of Comparison
W. Felix Handte [Thu, 19 Apr 2018 21:51:10 +0000 (17:51 -0400)]
Don't Clear the Dictionary Context Until No Longer Useful
W. Felix Handte [Thu, 19 Apr 2018 17:02:55 +0000 (13:02 -0400)]
Copy DictCtx into Working Context on Inputs Larger than 4 KB
W. Felix Handte [Wed, 18 Apr 2018 19:52:04 +0000 (15:52 -0400)]
Force Inline on HashChain
W. Felix Handte [Wed, 18 Apr 2018 17:55:58 +0000 (13:55 -0400)]
Split DictCtx-using Code Into Separate Inlining Chain
W. Felix Handte [Tue, 17 Apr 2018 00:23:19 +0000 (20:23 -0400)]
Use Fast Reset in LZ4F Again
W. Felix Handte [Mon, 16 Apr 2018 19:25:48 +0000 (15:25 -0400)]
Use Fast Reset API in LZ4F
W. Felix Handte [Mon, 16 Apr 2018 19:09:59 +0000 (15:09 -0400)]
Add Fast Reset Paths
W. Felix Handte [Thu, 5 Apr 2018 21:41:15 +0000 (17:41 -0400)]
Remove Match Upper Bounds Check
W. Felix Handte [Fri, 6 Apr 2018 23:24:22 +0000 (19:24 -0400)]
Fix Some Cast/Conversion Warnings
W. Felix Handte [Thu, 5 Apr 2018 20:32:40 +0000 (16:32 -0400)]
Fix Offset Math
W. Felix Handte [Thu, 5 Apr 2018 20:32:26 +0000 (16:32 -0400)]
Reset Stream in LZ4_compress_HC
W. Felix Handte [Wed, 4 Apr 2018 19:59:00 +0000 (15:59 -0400)]
Don't Bother Clearing Chain Table for Working Contexts
W. Felix Handte [Wed, 4 Apr 2018 19:20:47 +0000 (15:20 -0400)]
Push Previous Compression Offsets into the Past
W. Felix Handte [Wed, 28 Mar 2018 16:26:54 +0000 (12:26 -0400)]
Shift Dict Limit Checks out of the Loop