Yann Collet [Tue, 24 Apr 2018 20:18:40 +0000 (13:18 -0700)]
Merge pull request #512 from lz4/HC_dict
In-place unmutable dictionaries for LZ4HC
Yann Collet [Tue, 24 Apr 2018 18:25:57 +0000 (11:25 -0700)]
Merge pull request #511 from lz4/decFast
Fixed performance issue with LZ4_decompress_fast()
Yann Collet [Tue, 24 Apr 2018 17:49:41 +0000 (10:49 -0700)]
Merge pull request #488 from felixhandte/hc-dict-ctx
Use Dictionary In-Place in HC Mode
W. Felix Handte [Fri, 20 Apr 2018 01:00:19 +0000 (21:00 -0400)]
Remove Debug Log Statements
W. Felix Handte [Thu, 19 Apr 2018 22:56:01 +0000 (18:56 -0400)]
Remove the Framebench Tool
W. Felix Handte [Tue, 24 Apr 2018 15:55:53 +0000 (11:55 -0400)]
Revert Stream Size Const to Correct Value
Yann Collet [Tue, 24 Apr 2018 06:44:04 +0000 (23:44 -0700)]
Merge pull request #504 from baruchsiach/static-only-support
lib: allow to disable shared libraries
Cyan4973 [Tue, 24 Apr 2018 02:26:02 +0000 (19:26 -0700)]
re-ordered parenthesis
to avoid mixing && and &
as suggested by @terrelln
Yann Collet [Mon, 23 Apr 2018 22:55:56 +0000 (15:55 -0700)]
Merge pull request #507 from lz4/clangPerf
fixed lz4_fast clang performance
Cyan4973 [Mon, 23 Apr 2018 22:52:44 +0000 (15:52 -0700)]
fixed minor declaration issue with clang on msys
Cyan4973 [Mon, 23 Apr 2018 22:47:08 +0000 (15:47 -0700)]
disable shortcut for LZ4_decompress_fast()
improving speed
Cyan4973 [Mon, 23 Apr 2018 22:42:27 +0000 (15:42 -0700)]
fullbench compiled without assert()
to better reflect release speed
Yann Collet [Mon, 23 Apr 2018 22:28:19 +0000 (15:28 -0700)]
Merge pull request #510 from terrelln/bug-fix
Fix input size validation edge cases
Nick Terrell [Mon, 23 Apr 2018 21:21:02 +0000 (14:21 -0700)]
Fix compilation error and assert.
Nick Terrell [Mon, 23 Apr 2018 20:14:19 +0000 (13:14 -0700)]
Fix input size validation edge cases
The bug is a read up to 2 bytes past the end of the buffer.
There are three cases for this bug, one for each test case added.
* An empty input causes `token = *ip++` to read one byte too far.
* A one byte input with `(token >> ML_BITS) == RUN_MASK` causes
one extra byte to be read without validation. This could be
combined with the first bug to cause 2 extra bytes to be read.
* The case pointed out in issue #508, where `ip == iend` at the
beginning of the loop after taking the shortcut.
Benchmarks show no regressions on clang or gcc-7 on both my mac
and devserver.
Fixes #508.
Yann Collet [Mon, 23 Apr 2018 02:30:24 +0000 (19:30 -0700)]
Merge pull request #509 from svpv/clarifyFastRisks
lz4.h: clarify the risks of using LZ4_decompress_fast()
Alexey Tourbin [Sun, 22 Apr 2018 22:43:30 +0000 (01:43 +0300)]
lz4.h: clarify the risks of using LZ4_decompress_fast()
The notes about "security guarantee" and "malicious inputs" seemed
a bit non-technical to me, so I took the liberty to tone them down
and instead describe the actual risks in technical terms. Namely,
the function never writes past the end of the output buffer, so
a direct hostile takeover (resulting in arbitrary code execution
soon after the return from the function) is not possible. However,
the application can crash because of reads from unmapped pages.
I also took the liberty to describe what I believe is the only sensible
usage scenario for the function: "This function is only usable if the
originalSize of uncompressed data is known in advance," etc.
Cyan4973 [Sat, 21 Apr 2018 07:11:51 +0000 (00:11 -0700)]
fixed incorrect comment
Yann Collet [Sat, 21 Apr 2018 01:09:51 +0000 (18:09 -0700)]
fixed clang performance in lz4_fast
The simple change from
`matchIndex+MAX_DISTANCE < current`
towards
`current - matchIndex > MAX_DISTANCE`
is enough to generate a 10% performance drop under clang.
Quite massive.
(I missed as my eyes were concentrated on gcc performance at that time).
The second version is more robust, because it also survives a situation where
`matchIndex > current`
due to overflows.
The first version requires matchIndex to not overflow.
Hence were added `assert()` conditions.
The only case where this can happen is with dictCtx compression,
in the case where the dictionary context is not initialized before loading the dictionary.
So it's enough to always initialize the context while loading the dictionary.
W. Felix Handte [Sat, 21 Apr 2018 00:18:30 +0000 (20:18 -0400)]
Change vLimit Calculation
W. Felix Handte [Sat, 21 Apr 2018 00:14:12 +0000 (20:14 -0400)]
Remove Redundant Static Assert
W. Felix Handte [Fri, 20 Apr 2018 23:37:28 +0000 (19:37 -0400)]
Simpler loadDict() Reset
W. Felix Handte [Fri, 20 Apr 2018 23:37:07 +0000 (19:37 -0400)]
Tolerate Base Pointer Underflow
W. Felix Handte [Fri, 20 Apr 2018 23:35:51 +0000 (19:35 -0400)]
Don't Segfault on Malloc Failure
W. Felix Handte [Fri, 20 Apr 2018 21:56:26 +0000 (17:56 -0400)]
Sign-Extend -1 to Pointer Width
W. Felix Handte [Fri, 20 Apr 2018 21:13:40 +0000 (17:13 -0400)]
Fix Constant Value
W. Felix Handte [Fri, 20 Apr 2018 21:10:47 +0000 (17:10 -0400)]
Handle Index Underflows Safely
W. Felix Handte [Fri, 20 Apr 2018 19:30:08 +0000 (15:30 -0400)]
Consts and Asserts and Other Minor Nits
W. Felix Handte [Fri, 20 Apr 2018 19:16:41 +0000 (15:16 -0400)]
Add Some Simple Fuzzer Tests
W. Felix Handte [Fri, 20 Apr 2018 19:00:53 +0000 (15:00 -0400)]
Add Comments on New Public APIs
W. Felix Handte [Fri, 20 Apr 2018 18:52:13 +0000 (14:52 -0400)]
Add API for Attaching Dictionaries
W. Felix Handte [Fri, 20 Apr 2018 18:16:27 +0000 (14:16 -0400)]
Also Reset the Chain Table
W. Felix Handte [Fri, 20 Apr 2018 18:08:06 +0000 (14:08 -0400)]
Remove inputBuffer from Context, Work Around its Absence
W. Felix Handte [Fri, 20 Apr 2018 17:14:37 +0000 (13:14 -0400)]
Remove Commented Out Support for Match Continuation over Segment Boundary
W. Felix Handte [Thu, 19 Apr 2018 23:36:34 +0000 (19:36 -0400)]
Fix Signedness of Comparison
W. Felix Handte [Thu, 19 Apr 2018 21:51:10 +0000 (17:51 -0400)]
Don't Clear the Dictionary Context Until No Longer Useful
W. Felix Handte [Thu, 19 Apr 2018 17:02:55 +0000 (13:02 -0400)]
Copy DictCtx into Working Context on Inputs Larger than 4 KB
W. Felix Handte [Wed, 18 Apr 2018 19:52:04 +0000 (15:52 -0400)]
Force Inline on HashChain
W. Felix Handte [Wed, 18 Apr 2018 17:55:58 +0000 (13:55 -0400)]
Split DictCtx-using Code Into Separate Inlining Chain
W. Felix Handte [Tue, 17 Apr 2018 00:23:19 +0000 (20:23 -0400)]
Use Fast Reset in LZ4F Again
W. Felix Handte [Mon, 16 Apr 2018 19:25:48 +0000 (15:25 -0400)]
Use Fast Reset API in LZ4F
W. Felix Handte [Mon, 16 Apr 2018 19:09:59 +0000 (15:09 -0400)]
Add Fast Reset Paths
W. Felix Handte [Thu, 5 Apr 2018 21:41:15 +0000 (17:41 -0400)]
Remove Match Upper Bounds Check
W. Felix Handte [Fri, 6 Apr 2018 23:24:22 +0000 (19:24 -0400)]
Fix Some Cast/Conversion Warnings
W. Felix Handte [Thu, 5 Apr 2018 20:32:40 +0000 (16:32 -0400)]
Fix Offset Math
W. Felix Handte [Thu, 5 Apr 2018 20:32:26 +0000 (16:32 -0400)]
Reset Stream in LZ4_compress_HC
W. Felix Handte [Wed, 4 Apr 2018 19:59:00 +0000 (15:59 -0400)]
Don't Bother Clearing Chain Table for Working Contexts
W. Felix Handte [Wed, 4 Apr 2018 19:20:47 +0000 (15:20 -0400)]
Push Previous Compression Offsets into the Past
W. Felix Handte [Wed, 28 Mar 2018 16:26:54 +0000 (12:26 -0400)]
Shift Dict Limit Checks out of the Loop
W. Felix Handte [Wed, 28 Mar 2018 16:25:05 +0000 (12:25 -0400)]
Clear Tables on Dict Load
W. Felix Handte [Mon, 26 Mar 2018 15:43:52 +0000 (11:43 -0400)]
Only Perform Dict Lookup if Attempts Remain
W. Felix Handte [Thu, 22 Mar 2018 21:00:04 +0000 (17:00 -0400)]
Avoid Resetting Chain Table
W. Felix Handte [Thu, 22 Mar 2018 20:59:50 +0000 (16:59 -0400)]
Avoid Resetting Hash Table
W. Felix Handte [Wed, 21 Mar 2018 20:54:36 +0000 (16:54 -0400)]
Perform Lookups into the Dictionary Context
W. Felix Handte [Wed, 21 Mar 2018 20:43:15 +0000 (16:43 -0400)]
Call LZ4F_applyCDict Even on NULL CDict
W. Felix Handte [Wed, 21 Mar 2018 18:49:47 +0000 (14:49 -0400)]
Set dictCtx Rather than memcpy'ing Ctx
W. Felix Handte [Wed, 21 Mar 2018 15:21:07 +0000 (11:21 -0400)]
Fully Bounds Check Hash Table Reads
W. Felix Handte [Mon, 19 Mar 2018 17:16:05 +0000 (13:16 -0400)]
Add a Dictionary Context Pointer to the HC Context
W. Felix Handte [Mon, 19 Mar 2018 21:47:52 +0000 (17:47 -0400)]
Add Debug Log Statements to HC
W. Felix Handte [Wed, 18 Apr 2018 19:38:41 +0000 (15:38 -0400)]
Fix Framebench Output Buffer Sizing
W. Felix Handte [Wed, 11 Apr 2018 16:39:31 +0000 (12:39 -0400)]
Fix Cast
W. Felix Handte [Thu, 5 Apr 2018 22:08:13 +0000 (18:08 -0400)]
Print Failure Message in Framebench
W. Felix Handte [Wed, 4 Apr 2018 18:06:23 +0000 (14:06 -0400)]
Switch to Unaligned Samples to Compress Different Blobs Each Time
W. Felix Handte [Wed, 4 Apr 2018 17:12:28 +0000 (13:12 -0400)]
Fix Framebench Statistics
W. Felix Handte [Wed, 28 Mar 2018 21:16:23 +0000 (17:16 -0400)]
Add Run Name to Frame Bench Output
W. Felix Handte [Wed, 28 Mar 2018 20:36:35 +0000 (16:36 -0400)]
Auto-Calculate Appropriate Repetition Count
W. Felix Handte [Wed, 28 Mar 2018 20:19:30 +0000 (16:19 -0400)]
Print More Detailed Results Inside bench(), Add Compression Levels
W. Felix Handte [Thu, 22 Mar 2018 20:26:44 +0000 (16:26 -0400)]
Check Compressed Buffer is Correct in Frame Bench
W. Felix Handte [Mon, 19 Mar 2018 19:00:59 +0000 (15:00 -0400)]
Add HC Calls to Framebench
W. Felix Handte [Mon, 19 Mar 2018 18:01:57 +0000 (14:01 -0400)]
Restore Framebench Tool
This reverts commit
70f14823a46719e81e808d9ed9df90f478bcfd3f.
Yann Collet [Fri, 20 Apr 2018 00:36:24 +0000 (17:36 -0700)]
Merge pull request #505 from lz4/dev
update to `dev`
Yann Collet [Thu, 19 Apr 2018 18:50:20 +0000 (11:50 -0700)]
Merge pull request #503 from lz4/l120
minor length reduction of several large lines
Yann Collet [Thu, 19 Apr 2018 17:52:48 +0000 (10:52 -0700)]
Merge pull request #502 from lhacc1/dev
Wrap likely/unlikely macroses with #ifndef
Yann Collet [Thu, 19 Apr 2018 17:50:40 +0000 (10:50 -0700)]
modified indentation for consistency
Baruch Siach [Thu, 19 Apr 2018 09:28:11 +0000 (12:28 +0300)]
lib: allow to disable shared libraries
Just like BUILD_STATIC=no disables static libraries, BUILD_SHARED=no
disabled shared libraries. This is useful to support toolchains that do
not support shared libraries.
Yann Collet [Wed, 18 Apr 2018 23:49:27 +0000 (16:49 -0700)]
minor length reduction of several large lines
Yann Collet [Wed, 18 Apr 2018 17:16:25 +0000 (10:16 -0700)]
Merge pull request #497 from lz4/lowAddr
Compatibility with low memory addresses
Dmitrii Rodionov [Wed, 18 Apr 2018 09:20:56 +0000 (12:20 +0300)]
Wrap likely/unlikely macroses with #ifndef
It prevent redefine error when project using lz4 has its own likely/unlikely
macroses.
Yann Collet [Tue, 17 Apr 2018 23:47:56 +0000 (16:47 -0700)]
fixed LZ4_compress_fast_extState_fastReset() in 32-bit mode
Yann Collet [Tue, 17 Apr 2018 23:18:37 +0000 (16:18 -0700)]
fix dictDelta setting error
wrong test
Yann Collet [Tue, 17 Apr 2018 22:29:17 +0000 (15:29 -0700)]
fix matchIndex overflow
can happen with dictCtx
Yann Collet [Tue, 17 Apr 2018 19:07:22 +0000 (12:07 -0700)]
Merge branch 'dev' into lowAddr
Yann Collet [Tue, 17 Apr 2018 19:06:44 +0000 (12:06 -0700)]
Merge pull request #501 from felixhandte/fix-dict-load-offset
Always Bump Offset by 64 KB in LZ4_loadDict()
W. Felix Handte [Tue, 17 Apr 2018 18:01:44 +0000 (14:01 -0400)]
Always Bump Offset by 64 KB in LZ4_loadDict()
This actually ensures the guarantee referred to in the comment in
LZ4_compress_fast_continue().
Yann Collet [Tue, 17 Apr 2018 06:59:42 +0000 (23:59 -0700)]
fixed dictCtx compression
Yann Collet [Tue, 17 Apr 2018 00:15:02 +0000 (17:15 -0700)]
edited a few traces for debugging
Yann Collet [Mon, 16 Apr 2018 23:54:03 +0000 (16:54 -0700)]
fixed minor format warnings
Yann Collet [Mon, 16 Apr 2018 23:14:28 +0000 (16:14 -0700)]
fixed fuzzer tests
which were modified in parallel within branc `dev`
Yann Collet [Mon, 16 Apr 2018 23:12:38 +0000 (16:12 -0700)]
Merge branch 'dev' into lowAddr
Yann Collet [Mon, 16 Apr 2018 22:11:28 +0000 (15:11 -0700)]
fixed gcc performance regression
Yann Collet [Fri, 13 Apr 2018 20:22:38 +0000 (13:22 -0700)]
Merge pull request #499 from felixhandte/lz4-attach-dict-tests
Test LZ4_attach_dictionary() and Friends
W. Felix Handte [Thu, 12 Apr 2018 23:17:53 +0000 (19:17 -0400)]
Further Test that ExtDictCtx Mode Produces the Exact Same Output
W. Felix Handte [Thu, 12 Apr 2018 22:23:01 +0000 (18:23 -0400)]
Add Tests for LZ4_attach_dictionary and Friends
Yann Collet [Fri, 13 Apr 2018 09:45:32 +0000 (02:45 -0700)]
fixed minor unused variable warning
Yann Collet [Fri, 13 Apr 2018 09:26:14 +0000 (02:26 -0700)]
added comment on variables required after _next_match
Yann Collet [Fri, 13 Apr 2018 09:10:53 +0000 (02:10 -0700)]
fixed potential ptrdiff_t overflow (32-bits mode)
Also removed pointer comparison, which should solve #485
Cyan4973 [Fri, 13 Apr 2018 08:01:54 +0000 (01:01 -0700)]
compatibility with gcc-4.4 string.h version
Someone found it would be a great idea to define there a global variable under the very generic name "index".
Cause problem with shadow warnings, so no variable can be named "index" now ...
Also : automatically update API manual
Cyan4973 [Fri, 13 Apr 2018 07:59:27 +0000 (00:59 -0700)]
added sudo rights for low-mem-address tests
test4973 [Thu, 12 Apr 2018 23:12:21 +0000 (16:12 -0700)]
fixed : counting matches which overlap extDict and prefix
test4973 [Thu, 12 Apr 2018 14:25:40 +0000 (07:25 -0700)]
modified a few traces for debug