platform/upstream/lz4.git
6 years agoMultiply-Include Header to Check Guard Macro Correctness
W. Felix Handte [Tue, 24 Apr 2018 22:50:03 +0000 (18:50 -0400)]
Multiply-Include Header to Check Guard Macro Correctness

6 years agoChange Over Includes in the Project
W. Felix Handte [Sat, 21 Apr 2018 00:55:38 +0000 (20:55 -0400)]
Change Over Includes in the Project

6 years agoIntegrate lz4frame_static.h Declarations into lz4frame.h
W. Felix Handte [Fri, 20 Apr 2018 22:41:41 +0000 (18:41 -0400)]
Integrate lz4frame_static.h Declarations into lz4frame.h

6 years agoMerge pull request #512 from lz4/HC_dict
Yann Collet [Tue, 24 Apr 2018 20:18:40 +0000 (13:18 -0700)]
Merge pull request #512 from lz4/HC_dict

In-place unmutable dictionaries for LZ4HC

6 years agoMerge pull request #511 from lz4/decFast
Yann Collet [Tue, 24 Apr 2018 18:25:57 +0000 (11:25 -0700)]
Merge pull request #511 from lz4/decFast

Fixed performance issue with LZ4_decompress_fast()

6 years agoMerge pull request #488 from felixhandte/hc-dict-ctx
Yann Collet [Tue, 24 Apr 2018 17:49:41 +0000 (10:49 -0700)]
Merge pull request #488 from felixhandte/hc-dict-ctx

Use Dictionary In-Place in HC Mode

6 years agoRemove Debug Log Statements
W. Felix Handte [Fri, 20 Apr 2018 01:00:19 +0000 (21:00 -0400)]
Remove Debug Log Statements

6 years agoRemove the Framebench Tool
W. Felix Handte [Thu, 19 Apr 2018 22:56:01 +0000 (18:56 -0400)]
Remove the Framebench Tool

6 years agoRevert Stream Size Const to Correct Value
W. Felix Handte [Tue, 24 Apr 2018 15:55:53 +0000 (11:55 -0400)]
Revert Stream Size Const to Correct Value

6 years agoMerge pull request #504 from baruchsiach/static-only-support
Yann Collet [Tue, 24 Apr 2018 06:44:04 +0000 (23:44 -0700)]
Merge pull request #504 from baruchsiach/static-only-support

lib: allow to disable shared libraries

6 years agore-ordered parenthesis
Cyan4973 [Tue, 24 Apr 2018 02:26:02 +0000 (19:26 -0700)]
re-ordered parenthesis

to avoid mixing && and &
as suggested by @terrelln

6 years agoMerge pull request #507 from lz4/clangPerf
Yann Collet [Mon, 23 Apr 2018 22:55:56 +0000 (15:55 -0700)]
Merge pull request #507 from lz4/clangPerf

fixed lz4_fast clang performance

6 years agofixed minor declaration issue with clang on msys
Cyan4973 [Mon, 23 Apr 2018 22:52:44 +0000 (15:52 -0700)]
fixed minor declaration issue with clang on msys

6 years agodisable shortcut for LZ4_decompress_fast()
Cyan4973 [Mon, 23 Apr 2018 22:47:08 +0000 (15:47 -0700)]
disable shortcut for LZ4_decompress_fast()

improving speed

6 years agofullbench compiled without assert()
Cyan4973 [Mon, 23 Apr 2018 22:42:27 +0000 (15:42 -0700)]
fullbench compiled without assert()

to better reflect release speed

6 years agoMerge pull request #510 from terrelln/bug-fix
Yann Collet [Mon, 23 Apr 2018 22:28:19 +0000 (15:28 -0700)]
Merge pull request #510 from terrelln/bug-fix

Fix input size validation edge cases

6 years agoFix compilation error and assert.
Nick Terrell [Mon, 23 Apr 2018 21:21:02 +0000 (14:21 -0700)]
Fix compilation error and assert.

6 years agoFix input size validation edge cases
Nick Terrell [Mon, 23 Apr 2018 20:14:19 +0000 (13:14 -0700)]
Fix input size validation edge cases

The bug is a read up to 2 bytes past the end of the buffer.
There are three cases for this bug, one for each test case added.

* An empty input causes `token = *ip++` to read one byte too far.
* A one byte input with `(token >> ML_BITS) == RUN_MASK` causes
  one extra byte to be read without validation. This could be
  combined with the first bug to cause 2 extra bytes to be read.
* The case pointed out in issue #508, where `ip == iend` at the
  beginning of the loop after taking the shortcut.

Benchmarks show no regressions on clang or gcc-7 on both my mac
and devserver.

Fixes #508.

6 years agoMerge pull request #509 from svpv/clarifyFastRisks
Yann Collet [Mon, 23 Apr 2018 02:30:24 +0000 (19:30 -0700)]
Merge pull request #509 from svpv/clarifyFastRisks

lz4.h: clarify the risks of using LZ4_decompress_fast()

6 years agolz4.h: clarify the risks of using LZ4_decompress_fast()
Alexey Tourbin [Sun, 22 Apr 2018 22:43:30 +0000 (01:43 +0300)]
lz4.h: clarify the risks of using LZ4_decompress_fast()

The notes about "security guarantee" and "malicious inputs" seemed
a bit non-technical to me, so I took the liberty to tone them down
and instead describe the actual risks in technical terms.  Namely,
the function never writes past the end of the output buffer, so
a direct hostile takeover (resulting in arbitrary code execution
soon after the return from the function) is not possible.  However,
the application can crash because of reads from unmapped pages.

I also took the liberty to describe what I believe is the only sensible
usage scenario for the function: "This function is only usable if the
originalSize of uncompressed data is known in advance," etc.

6 years agofixed incorrect comment
Cyan4973 [Sat, 21 Apr 2018 07:11:51 +0000 (00:11 -0700)]
fixed incorrect comment

6 years agofixed clang performance in lz4_fast
Yann Collet [Sat, 21 Apr 2018 01:09:51 +0000 (18:09 -0700)]
fixed clang performance in lz4_fast

The simple change from
`matchIndex+MAX_DISTANCE < current`
towards
`current - matchIndex > MAX_DISTANCE`

is enough to generate a 10% performance drop under clang.
Quite massive.
(I missed as my eyes were concentrated on gcc performance at that time).

The second version is more robust, because it also survives a situation where
`matchIndex > current`
due to overflows.

The first version requires matchIndex to not overflow.
Hence were added `assert()` conditions.

The only case where this can happen is with dictCtx compression,
in the case where the dictionary context is not initialized before loading the dictionary.
So it's enough to always initialize the context while loading the dictionary.

6 years agoChange vLimit Calculation
W. Felix Handte [Sat, 21 Apr 2018 00:18:30 +0000 (20:18 -0400)]
Change vLimit Calculation

6 years agoRemove Redundant Static Assert
W. Felix Handte [Sat, 21 Apr 2018 00:14:12 +0000 (20:14 -0400)]
Remove Redundant Static Assert

6 years agoSimpler loadDict() Reset
W. Felix Handte [Fri, 20 Apr 2018 23:37:28 +0000 (19:37 -0400)]
Simpler loadDict() Reset

6 years agoTolerate Base Pointer Underflow
W. Felix Handte [Fri, 20 Apr 2018 23:37:07 +0000 (19:37 -0400)]
Tolerate Base Pointer Underflow

6 years agoDon't Segfault on Malloc Failure
W. Felix Handte [Fri, 20 Apr 2018 23:35:51 +0000 (19:35 -0400)]
Don't Segfault on Malloc Failure

6 years agoSign-Extend -1 to Pointer Width
W. Felix Handte [Fri, 20 Apr 2018 21:56:26 +0000 (17:56 -0400)]
Sign-Extend -1 to Pointer Width

6 years agoFix Constant Value
W. Felix Handte [Fri, 20 Apr 2018 21:13:40 +0000 (17:13 -0400)]
Fix Constant Value

6 years agoHandle Index Underflows Safely
W. Felix Handte [Fri, 20 Apr 2018 21:10:47 +0000 (17:10 -0400)]
Handle Index Underflows Safely

6 years agoConsts and Asserts and Other Minor Nits
W. Felix Handte [Fri, 20 Apr 2018 19:30:08 +0000 (15:30 -0400)]
Consts and Asserts and Other Minor Nits

6 years agoAdd Some Simple Fuzzer Tests
W. Felix Handte [Fri, 20 Apr 2018 19:16:41 +0000 (15:16 -0400)]
Add Some Simple Fuzzer Tests

6 years agoAdd Comments on New Public APIs
W. Felix Handte [Fri, 20 Apr 2018 19:00:53 +0000 (15:00 -0400)]
Add Comments on New Public APIs

6 years agoAdd API for Attaching Dictionaries
W. Felix Handte [Fri, 20 Apr 2018 18:52:13 +0000 (14:52 -0400)]
Add API for Attaching Dictionaries

6 years agoAlso Reset the Chain Table
W. Felix Handte [Fri, 20 Apr 2018 18:16:27 +0000 (14:16 -0400)]
Also Reset the Chain Table

6 years agoRemove inputBuffer from Context, Work Around its Absence
W. Felix Handte [Fri, 20 Apr 2018 18:08:06 +0000 (14:08 -0400)]
Remove inputBuffer from Context, Work Around its Absence

6 years agoRemove Commented Out Support for Match Continuation over Segment Boundary
W. Felix Handte [Fri, 20 Apr 2018 17:14:37 +0000 (13:14 -0400)]
Remove Commented Out Support for Match Continuation over Segment Boundary

6 years agoFix Signedness of Comparison
W. Felix Handte [Thu, 19 Apr 2018 23:36:34 +0000 (19:36 -0400)]
Fix Signedness of Comparison

6 years agoDon't Clear the Dictionary Context Until No Longer Useful
W. Felix Handte [Thu, 19 Apr 2018 21:51:10 +0000 (17:51 -0400)]
Don't Clear the Dictionary Context Until No Longer Useful

6 years agoCopy DictCtx into Working Context on Inputs Larger than 4 KB
W. Felix Handte [Thu, 19 Apr 2018 17:02:55 +0000 (13:02 -0400)]
Copy DictCtx into Working Context on Inputs Larger than 4 KB

6 years agoForce Inline on HashChain
W. Felix Handte [Wed, 18 Apr 2018 19:52:04 +0000 (15:52 -0400)]
Force Inline on HashChain

6 years agoSplit DictCtx-using Code Into Separate Inlining Chain
W. Felix Handte [Wed, 18 Apr 2018 17:55:58 +0000 (13:55 -0400)]
Split DictCtx-using Code Into Separate Inlining Chain

6 years agoUse Fast Reset in LZ4F Again
W. Felix Handte [Tue, 17 Apr 2018 00:23:19 +0000 (20:23 -0400)]
Use Fast Reset in LZ4F Again

6 years agoUse Fast Reset API in LZ4F
W. Felix Handte [Mon, 16 Apr 2018 19:25:48 +0000 (15:25 -0400)]
Use Fast Reset API in LZ4F

6 years agoAdd Fast Reset Paths
W. Felix Handte [Mon, 16 Apr 2018 19:09:59 +0000 (15:09 -0400)]
Add Fast Reset Paths

6 years agoRemove Match Upper Bounds Check
W. Felix Handte [Thu, 5 Apr 2018 21:41:15 +0000 (17:41 -0400)]
Remove Match Upper Bounds Check

6 years agoFix Some Cast/Conversion Warnings
W. Felix Handte [Fri, 6 Apr 2018 23:24:22 +0000 (19:24 -0400)]
Fix Some Cast/Conversion Warnings

6 years agoFix Offset Math
W. Felix Handte [Thu, 5 Apr 2018 20:32:40 +0000 (16:32 -0400)]
Fix Offset Math

6 years agoReset Stream in LZ4_compress_HC
W. Felix Handte [Thu, 5 Apr 2018 20:32:26 +0000 (16:32 -0400)]
Reset Stream in LZ4_compress_HC

6 years agoDon't Bother Clearing Chain Table for Working Contexts
W. Felix Handte [Wed, 4 Apr 2018 19:59:00 +0000 (15:59 -0400)]
Don't Bother Clearing Chain Table for Working Contexts

6 years agoPush Previous Compression Offsets into the Past
W. Felix Handte [Wed, 4 Apr 2018 19:20:47 +0000 (15:20 -0400)]
Push Previous Compression Offsets into the Past

6 years agoShift Dict Limit Checks out of the Loop
W. Felix Handte [Wed, 28 Mar 2018 16:26:54 +0000 (12:26 -0400)]
Shift Dict Limit Checks out of the Loop

6 years agoClear Tables on Dict Load
W. Felix Handte [Wed, 28 Mar 2018 16:25:05 +0000 (12:25 -0400)]
Clear Tables on Dict Load

6 years agoOnly Perform Dict Lookup if Attempts Remain
W. Felix Handte [Mon, 26 Mar 2018 15:43:52 +0000 (11:43 -0400)]
Only Perform Dict Lookup if Attempts Remain

6 years agoAvoid Resetting Chain Table
W. Felix Handte [Thu, 22 Mar 2018 21:00:04 +0000 (17:00 -0400)]
Avoid Resetting Chain Table

6 years agoAvoid Resetting Hash Table
W. Felix Handte [Thu, 22 Mar 2018 20:59:50 +0000 (16:59 -0400)]
Avoid Resetting Hash Table

6 years agoPerform Lookups into the Dictionary Context
W. Felix Handte [Wed, 21 Mar 2018 20:54:36 +0000 (16:54 -0400)]
Perform Lookups into the Dictionary Context

6 years agoCall LZ4F_applyCDict Even on NULL CDict
W. Felix Handte [Wed, 21 Mar 2018 20:43:15 +0000 (16:43 -0400)]
Call LZ4F_applyCDict Even on NULL CDict

6 years agoSet dictCtx Rather than memcpy'ing Ctx
W. Felix Handte [Wed, 21 Mar 2018 18:49:47 +0000 (14:49 -0400)]
Set dictCtx Rather than memcpy'ing Ctx

6 years agoFully Bounds Check Hash Table Reads
W. Felix Handte [Wed, 21 Mar 2018 15:21:07 +0000 (11:21 -0400)]
Fully Bounds Check Hash Table Reads

6 years agoAdd a Dictionary Context Pointer to the HC Context
W. Felix Handte [Mon, 19 Mar 2018 17:16:05 +0000 (13:16 -0400)]
Add a Dictionary Context Pointer to the HC Context

6 years agoAdd Debug Log Statements to HC
W. Felix Handte [Mon, 19 Mar 2018 21:47:52 +0000 (17:47 -0400)]
Add Debug Log Statements to HC

6 years agoFix Framebench Output Buffer Sizing
W. Felix Handte [Wed, 18 Apr 2018 19:38:41 +0000 (15:38 -0400)]
Fix Framebench Output Buffer Sizing

6 years agoFix Cast
W. Felix Handte [Wed, 11 Apr 2018 16:39:31 +0000 (12:39 -0400)]
Fix Cast

6 years agoPrint Failure Message in Framebench
W. Felix Handte [Thu, 5 Apr 2018 22:08:13 +0000 (18:08 -0400)]
Print Failure Message in Framebench

6 years agoSwitch to Unaligned Samples to Compress Different Blobs Each Time
W. Felix Handte [Wed, 4 Apr 2018 18:06:23 +0000 (14:06 -0400)]
Switch to Unaligned Samples to Compress Different Blobs Each Time

6 years agoFix Framebench Statistics
W. Felix Handte [Wed, 4 Apr 2018 17:12:28 +0000 (13:12 -0400)]
Fix Framebench Statistics

6 years agoAdd Run Name to Frame Bench Output
W. Felix Handte [Wed, 28 Mar 2018 21:16:23 +0000 (17:16 -0400)]
Add Run Name to Frame Bench Output

6 years agoAuto-Calculate Appropriate Repetition Count
W. Felix Handte [Wed, 28 Mar 2018 20:36:35 +0000 (16:36 -0400)]
Auto-Calculate Appropriate Repetition Count

6 years agoPrint More Detailed Results Inside bench(), Add Compression Levels
W. Felix Handte [Wed, 28 Mar 2018 20:19:30 +0000 (16:19 -0400)]
Print More Detailed Results Inside bench(), Add Compression Levels

6 years agoCheck Compressed Buffer is Correct in Frame Bench
W. Felix Handte [Thu, 22 Mar 2018 20:26:44 +0000 (16:26 -0400)]
Check Compressed Buffer is Correct in Frame Bench

6 years agoAdd HC Calls to Framebench
W. Felix Handte [Mon, 19 Mar 2018 19:00:59 +0000 (15:00 -0400)]
Add HC Calls to Framebench

6 years agoRestore Framebench Tool
W. Felix Handte [Mon, 19 Mar 2018 18:01:57 +0000 (14:01 -0400)]
Restore Framebench Tool

This reverts commit 70f14823a46719e81e808d9ed9df90f478bcfd3f.

6 years agoMerge pull request #505 from lz4/dev
Yann Collet [Fri, 20 Apr 2018 00:36:24 +0000 (17:36 -0700)]
Merge pull request #505 from lz4/dev

update to `dev`

6 years agoMerge pull request #503 from lz4/l120
Yann Collet [Thu, 19 Apr 2018 18:50:20 +0000 (11:50 -0700)]
Merge pull request #503 from lz4/l120

minor length reduction of several large lines

6 years agoMerge pull request #502 from lhacc1/dev
Yann Collet [Thu, 19 Apr 2018 17:52:48 +0000 (10:52 -0700)]
Merge pull request #502 from lhacc1/dev

Wrap likely/unlikely macroses with #ifndef

6 years agomodified indentation for consistency
Yann Collet [Thu, 19 Apr 2018 17:50:40 +0000 (10:50 -0700)]
modified indentation for consistency

6 years agolib: allow to disable shared libraries
Baruch Siach [Thu, 19 Apr 2018 09:28:11 +0000 (12:28 +0300)]
lib: allow to disable shared libraries

Just like BUILD_STATIC=no disables static libraries, BUILD_SHARED=no
disabled shared libraries. This is useful to support toolchains that do
not support shared libraries.

6 years agominor length reduction of several large lines
Yann Collet [Wed, 18 Apr 2018 23:49:27 +0000 (16:49 -0700)]
minor length reduction of several large lines

6 years agoMerge pull request #497 from lz4/lowAddr
Yann Collet [Wed, 18 Apr 2018 17:16:25 +0000 (10:16 -0700)]
Merge pull request #497 from lz4/lowAddr

Compatibility with low memory addresses

6 years agoWrap likely/unlikely macroses with #ifndef
Dmitrii Rodionov [Wed, 18 Apr 2018 09:20:56 +0000 (12:20 +0300)]
Wrap likely/unlikely macroses with #ifndef

It prevent redefine error when project using lz4 has its own likely/unlikely
macroses.

6 years agofixed LZ4_compress_fast_extState_fastReset() in 32-bit mode
Yann Collet [Tue, 17 Apr 2018 23:47:56 +0000 (16:47 -0700)]
fixed LZ4_compress_fast_extState_fastReset() in 32-bit mode

6 years agofix dictDelta setting error
Yann Collet [Tue, 17 Apr 2018 23:18:37 +0000 (16:18 -0700)]
fix dictDelta setting error

wrong test

6 years agofix matchIndex overflow
Yann Collet [Tue, 17 Apr 2018 22:29:17 +0000 (15:29 -0700)]
fix matchIndex overflow

can happen with dictCtx

6 years agoMerge branch 'dev' into lowAddr
Yann Collet [Tue, 17 Apr 2018 19:07:22 +0000 (12:07 -0700)]
Merge branch 'dev' into lowAddr

6 years agoMerge pull request #501 from felixhandte/fix-dict-load-offset
Yann Collet [Tue, 17 Apr 2018 19:06:44 +0000 (12:06 -0700)]
Merge pull request #501 from felixhandte/fix-dict-load-offset

Always Bump Offset by 64 KB in LZ4_loadDict()

6 years agoAlways Bump Offset by 64 KB in LZ4_loadDict()
W. Felix Handte [Tue, 17 Apr 2018 18:01:44 +0000 (14:01 -0400)]
Always Bump Offset by 64 KB in LZ4_loadDict()

This actually ensures the guarantee referred to in the comment in
LZ4_compress_fast_continue().

6 years agofixed dictCtx compression
Yann Collet [Tue, 17 Apr 2018 06:59:42 +0000 (23:59 -0700)]
fixed dictCtx compression

6 years agoedited a few traces for debugging
Yann Collet [Tue, 17 Apr 2018 00:15:02 +0000 (17:15 -0700)]
edited a few traces for debugging

6 years agofixed minor format warnings
Yann Collet [Mon, 16 Apr 2018 23:54:03 +0000 (16:54 -0700)]
fixed minor format warnings

6 years agofixed fuzzer tests
Yann Collet [Mon, 16 Apr 2018 23:14:28 +0000 (16:14 -0700)]
fixed fuzzer tests

which were modified in parallel within branc `dev`

6 years agoMerge branch 'dev' into lowAddr
Yann Collet [Mon, 16 Apr 2018 23:12:38 +0000 (16:12 -0700)]
Merge branch 'dev' into lowAddr

6 years agofixed gcc performance regression
Yann Collet [Mon, 16 Apr 2018 22:11:28 +0000 (15:11 -0700)]
fixed gcc performance regression

6 years agoMerge pull request #499 from felixhandte/lz4-attach-dict-tests
Yann Collet [Fri, 13 Apr 2018 20:22:38 +0000 (13:22 -0700)]
Merge pull request #499 from felixhandte/lz4-attach-dict-tests

Test LZ4_attach_dictionary() and Friends

6 years agoFurther Test that ExtDictCtx Mode Produces the Exact Same Output
W. Felix Handte [Thu, 12 Apr 2018 23:17:53 +0000 (19:17 -0400)]
Further Test that ExtDictCtx Mode Produces the Exact Same Output

6 years agoAdd Tests for LZ4_attach_dictionary and Friends
W. Felix Handte [Thu, 12 Apr 2018 22:23:01 +0000 (18:23 -0400)]
Add Tests for LZ4_attach_dictionary and Friends

6 years agofixed minor unused variable warning
Yann Collet [Fri, 13 Apr 2018 09:45:32 +0000 (02:45 -0700)]
fixed minor unused variable warning

6 years agoadded comment on variables required after _next_match
Yann Collet [Fri, 13 Apr 2018 09:26:14 +0000 (02:26 -0700)]
added comment on variables required after _next_match

6 years agofixed potential ptrdiff_t overflow (32-bits mode)
Yann Collet [Fri, 13 Apr 2018 09:10:53 +0000 (02:10 -0700)]
fixed potential ptrdiff_t overflow (32-bits mode)

Also removed pointer comparison, which should solve #485

6 years agocompatibility with gcc-4.4 string.h version
Cyan4973 [Fri, 13 Apr 2018 08:01:54 +0000 (01:01 -0700)]
compatibility with gcc-4.4 string.h version

Someone found it would be a great idea to define there a global variable under the very generic name "index".
Cause problem with shadow warnings, so no variable can be named "index" now ...

Also : automatically update API manual