platform/upstream/lz4.git
6 years agoadded sudo rights for low-mem-address tests
Cyan4973 [Fri, 13 Apr 2018 07:59:27 +0000 (00:59 -0700)]
added sudo rights for low-mem-address tests

6 years agofixed : counting matches which overlap extDict and prefix
test4973 [Thu, 12 Apr 2018 23:12:21 +0000 (16:12 -0700)]
fixed : counting matches which overlap extDict and prefix

6 years agomodified a few traces for debug
test4973 [Thu, 12 Apr 2018 14:25:40 +0000 (07:25 -0700)]
modified a few traces for debug

6 years agofixed LZ4_compress_fast_extState_fastReset()
test4973 [Wed, 11 Apr 2018 23:49:40 +0000 (16:49 -0700)]
fixed LZ4_compress_fast_extState_fastReset()

6 years agoMerge branch 'dev' into lowAddr
test4973 [Wed, 11 Apr 2018 23:45:19 +0000 (16:45 -0700)]
Merge branch 'dev' into lowAddr

6 years agoMerge pull request #492 from felixhandte/avoid-prepare-in-continue
Yann Collet [Wed, 11 Apr 2018 23:15:42 +0000 (16:15 -0700)]
Merge pull request #492 from felixhandte/avoid-prepare-in-continue

Several Changes Concerning Table Preparation in LZ4 Fast

6 years agoFix Silly Warning (const-ness in declaration has no effect on value types!)
W. Felix Handte [Wed, 11 Apr 2018 22:42:09 +0000 (18:42 -0400)]
Fix Silly Warning (const-ness in declaration has no effect on value types!)

6 years agoMinor Fixes
W. Felix Handte [Wed, 11 Apr 2018 20:55:12 +0000 (16:55 -0400)]
Minor Fixes

6 years agoAdd a LZ4_STATIC_LINKING_ONLY Macro to Guard Experimental APIs
W. Felix Handte [Wed, 11 Apr 2018 20:31:52 +0000 (16:31 -0400)]
Add a LZ4_STATIC_LINKING_ONLY Macro to Guard Experimental APIs

6 years agoExpose dictCtx Functionality in LZ4
W. Felix Handte [Wed, 11 Apr 2018 20:04:24 +0000 (16:04 -0400)]
Expose dictCtx Functionality in LZ4

6 years agoRename _extState_noReset -> _extState_fastReset and Edit Comments
W. Felix Handte [Wed, 11 Apr 2018 19:13:01 +0000 (15:13 -0400)]
Rename _extState_noReset -> _extState_fastReset and Edit Comments

6 years agoRemove Extraneous Assignment (clearedTable == 0)
W. Felix Handte [Wed, 11 Apr 2018 19:12:34 +0000 (15:12 -0400)]
Remove Extraneous Assignment (clearedTable == 0)

6 years agoExpose a Faster Stream Reset Function
W. Felix Handte [Tue, 10 Apr 2018 17:12:30 +0000 (13:12 -0400)]
Expose a Faster Stream Reset Function

6 years agofix minor conversion warning
test4973 [Tue, 10 Apr 2018 03:38:00 +0000 (20:38 -0700)]
fix minor conversion warning

cast from void not implicit for C++

6 years agofixed minor conversion warning
test4973 [Tue, 10 Apr 2018 00:08:17 +0000 (17:08 -0700)]
fixed minor conversion warning

ptr diff -> U32

6 years agoMerge branch 'dev' into lowAddr
test4973 [Mon, 9 Apr 2018 23:23:39 +0000 (16:23 -0700)]
Merge branch 'dev' into lowAddr

6 years agoAvoid Calling LZ4_prepareTable() in LZ4_compress_fast_continue()
W. Felix Handte [Fri, 6 Apr 2018 20:52:29 +0000 (16:52 -0400)]
Avoid Calling LZ4_prepareTable() in LZ4_compress_fast_continue()

6 years agoMerge pull request #494 from felixhandte/kill-goto
Yann Collet [Sat, 7 Apr 2018 00:35:45 +0000 (17:35 -0700)]
Merge pull request #494 from felixhandte/kill-goto

Return to Allowing Early Returns in LZ4_compress_generic()

6 years agoReturn to Allowing Early Returns in LZ4_compress_generic()
W. Felix Handte [Fri, 6 Apr 2018 22:52:55 +0000 (18:52 -0400)]
Return to Allowing Early Returns in LZ4_compress_generic()

Or: `goto` Considered Harmful

Or: https://xkcd.com/292/

6 years agoMerge pull request #493 from lz4/statusLine
Yann Collet [Fri, 6 Apr 2018 22:33:28 +0000 (15:33 -0700)]
Merge pull request #493 from lz4/statusLine

fixed DISPLAYUPDATE()

6 years agofixed DISPLAYUPDATE()
Yann Collet [Fri, 6 Apr 2018 21:16:23 +0000 (14:16 -0700)]
fixed DISPLAYUPDATE()

wrong comparison, which was always overflowing (hence was always true)
except when it was not (i386, reported by pmc)
in which case it would never show any information.

6 years agonoticed a bug when re-using hash table
test4973 [Fri, 6 Apr 2018 02:05:49 +0000 (19:05 -0700)]
noticed a bug when re-using hash table

./fuzzer -vv -s4217 -t7518

6 years agoadded low-memory address test to travis
test4973 [Fri, 6 Apr 2018 01:39:22 +0000 (18:39 -0700)]
added low-memory address test to travis

requires modification linux configuration (sudo)

6 years agofixed byPtr mode
test4973 [Fri, 6 Apr 2018 01:29:42 +0000 (18:29 -0700)]
fixed byPtr mode

switch to byU32 when src address is < 64K

note : byPtr is still useful in 32-bits, as it's about ~10% faster

6 years agofixed byPtr match search
test4973 [Fri, 6 Apr 2018 00:52:54 +0000 (17:52 -0700)]
fixed byPtr match search

6 years agofixed immediate match search
test4973 [Fri, 6 Apr 2018 00:16:33 +0000 (17:16 -0700)]
fixed immediate match search

6 years agochanged LZ4_compress_generic() logic
test4973 [Thu, 5 Apr 2018 23:38:43 +0000 (16:38 -0700)]
changed LZ4_compress_generic() logic

to use indexes (U32) instead of Ptr.

byPtr is still present.

6 years agofixed lz4 compression starting at small address
test4973 [Thu, 5 Apr 2018 19:40:33 +0000 (12:40 -0700)]
fixed lz4 compression starting at small address

when using byU32 and byU16 modes

6 years agoMerge branch 'dev' into lowAddr
test4973 [Wed, 4 Apr 2018 18:38:55 +0000 (11:38 -0700)]
Merge branch 'dev' into lowAddr

6 years agoMerge pull request #490 from kenjichanhkg/dev
Yann Collet [Mon, 2 Apr 2018 03:33:42 +0000 (20:33 -0700)]
Merge pull request #490 from kenjichanhkg/dev

added vs2017 projects

6 years agoadded vs2017 projects
Kenji Chan [Mon, 2 Apr 2018 02:52:45 +0000 (10:52 +0800)]
added vs2017 projects

6 years agoMerge pull request #486 from felixhandte/fix-test-makefile-clean-up
Yann Collet [Wed, 21 Mar 2018 21:53:02 +0000 (14:53 -0700)]
Merge pull request #486 from felixhandte/fix-test-makefile-clean-up

Add Dependency to Fix Parallel `make test` Runs

6 years agoMerge pull request #487 from felixhandte/better-obsoletion-comment
Yann Collet [Wed, 21 Mar 2018 21:52:53 +0000 (14:52 -0700)]
Merge pull request #487 from felixhandte/better-obsoletion-comment

Better Describe Functionality of Obsolete Streaming Functions

6 years agoAlso Fix a Comment
W. Felix Handte [Wed, 21 Mar 2018 15:48:35 +0000 (11:48 -0400)]
Also Fix a Comment

6 years agoBetter Describe Functionality of Obsolete Streaming Functions
W. Felix Handte [Wed, 21 Mar 2018 15:39:41 +0000 (11:39 -0400)]
Better Describe Functionality of Obsolete Streaming Functions

6 years agoAdd Dependency to Fix Parallel `make test` Runs
W. Felix Handte [Wed, 21 Mar 2018 15:28:51 +0000 (11:28 -0400)]
Add Dependency to Fix Parallel `make test` Runs

When run with `-jN`, the `rm tmp*` can run in the middle of the `test-lz4-dict`
job, which will then fail, finding its files to have been axed. This adds a
dependency between the two.

6 years agoadded c90 test to c_standards
Yann Collet [Wed, 21 Mar 2018 14:19:48 +0000 (07:19 -0700)]
added c90 test to c_standards

to catch `//` comments

6 years agoadded low address fuzzer tests
test4973 [Wed, 21 Mar 2018 14:14:13 +0000 (07:14 -0700)]
added low address fuzzer tests

6 years agofix comment style
Yann Collet [Wed, 21 Mar 2018 14:07:24 +0000 (07:07 -0700)]
fix comment style

6 years agobench: introduced hidden command -S
Yann Collet [Tue, 20 Mar 2018 00:19:25 +0000 (17:19 -0700)]
bench: introduced hidden command -S

to benchmark multiple files with separate results

6 years agoMerge branch 'dev' of github.com:Cyan4973/lz4 into dev
Yann Collet [Mon, 19 Mar 2018 23:18:25 +0000 (16:18 -0700)]
Merge branch 'dev' of github.com:Cyan4973/lz4 into dev

6 years agominor man fix on clevels
Yann Collet [Mon, 19 Mar 2018 23:18:10 +0000 (16:18 -0700)]
minor man fix on clevels

6 years agoMerge pull request #484 from lz4/fasterDict
Yann Collet [Mon, 19 Mar 2018 17:06:33 +0000 (10:06 -0700)]
Merge pull request #484 from lz4/fasterDict

Faster dictionary compression

6 years agoMerge pull request #406 from felixhandte/ref-dict-table
Yann Collet [Mon, 19 Mar 2018 02:07:55 +0000 (19:07 -0700)]
Merge pull request #406 from felixhandte/ref-dict-table

Use the Dictionary Hash Table in Place

6 years agoRemove Framebench Tool
W. Felix Handte [Wed, 31 Jan 2018 23:11:37 +0000 (18:11 -0500)]
Remove Framebench Tool

6 years agoMove LZ4_compress_fast_extState_noReset Declaration to Unstable Section
W. Felix Handte [Wed, 14 Mar 2018 19:58:38 +0000 (15:58 -0400)]
Move LZ4_compress_fast_extState_noReset Declaration to Unstable Section

6 years agoRestore the Other Old Streaming Functions in a Degraded Fashion
W. Felix Handte [Wed, 14 Mar 2018 19:51:59 +0000 (15:51 -0400)]
Restore the Other Old Streaming Functions in a Degraded Fashion

6 years agoSwitch ALLOC() to ALLOC_AND_ZERO() to Paper Over Existing Uninitialized Read
W. Felix Handte [Tue, 13 Mar 2018 21:47:34 +0000 (17:47 -0400)]
Switch ALLOC() to ALLOC_AND_ZERO() to Paper Over Existing Uninitialized Read

6 years agoSplit lz4CtxLevel into Two Fields
W. Felix Handte [Tue, 13 Mar 2018 21:45:09 +0000 (17:45 -0400)]
Split lz4CtxLevel into Two Fields

6 years agoAnother Allocation Fail Check
W. Felix Handte [Tue, 13 Mar 2018 21:35:44 +0000 (17:35 -0400)]
Another Allocation Fail Check

6 years agoRestore LZ4_sizeofStreamState, We Didn't Actually Need to Delete It
W. Felix Handte [Tue, 13 Mar 2018 19:42:03 +0000 (15:42 -0400)]
Restore LZ4_sizeofStreamState, We Didn't Actually Need to Delete It

6 years agoRestore checkTag Cleaning
W. Felix Handte [Tue, 13 Mar 2018 19:18:08 +0000 (15:18 -0400)]
Restore checkTag Cleaning

6 years agoRename Enums and Add Comment
W. Felix Handte [Tue, 13 Mar 2018 19:07:19 +0000 (15:07 -0400)]
Rename Enums and Add Comment

6 years agoWhitespace Fixes
W. Felix Handte [Mon, 12 Mar 2018 22:46:54 +0000 (18:46 -0400)]
Whitespace Fixes

6 years agoAdd NULL Checks
W. Felix Handte [Mon, 12 Mar 2018 22:32:24 +0000 (18:32 -0400)]
Add NULL Checks

6 years agoSimpler Ternary Statements
W. Felix Handte [Fri, 9 Mar 2018 17:05:31 +0000 (12:05 -0500)]
Simpler Ternary Statements

6 years agoRenames and Comment Fixes
W. Felix Handte [Mon, 12 Mar 2018 22:13:24 +0000 (18:13 -0400)]
Renames and Comment Fixes

6 years agoHoist LZ4F Dictionary Setup into Helper LZ4F_applyCDict()
W. Felix Handte [Mon, 12 Mar 2018 20:11:55 +0000 (16:11 -0400)]
Hoist LZ4F Dictionary Setup into Helper LZ4F_applyCDict()

6 years agoMinor Style Fixes
W. Felix Handte [Mon, 12 Mar 2018 20:11:44 +0000 (16:11 -0400)]
Minor Style Fixes

6 years agoPreserve currentOffset==0 When Possible
W. Felix Handte [Thu, 8 Mar 2018 19:09:06 +0000 (14:09 -0500)]
Preserve currentOffset==0 When Possible

6 years agoSpecialize _extState() for Clean Ctx Rather Than Calling _safeExtState()
W. Felix Handte [Fri, 9 Mar 2018 17:14:42 +0000 (12:14 -0500)]
Specialize _extState() for Clean Ctx Rather Than Calling _safeExtState()

6 years agoRemove Switch In Favor of Ternary Statement
W. Felix Handte [Thu, 8 Mar 2018 17:30:34 +0000 (12:30 -0500)]
Remove Switch In Favor of Ternary Statement

6 years agoFurther Avoid a dictionary==NULL Check
W. Felix Handte [Thu, 8 Mar 2018 17:29:45 +0000 (12:29 -0500)]
Further Avoid a dictionary==NULL Check

6 years agoOptimize Dict Check Condition
W. Felix Handte [Tue, 6 Mar 2018 20:53:22 +0000 (15:53 -0500)]
Optimize Dict Check Condition

6 years agoMove to 4KB Cut-Off
W. Felix Handte [Tue, 6 Mar 2018 16:52:02 +0000 (11:52 -0500)]
Move to 4KB Cut-Off

6 years agoReset Table on Inputs Larger than 2KB
W. Felix Handte [Wed, 14 Feb 2018 01:06:24 +0000 (17:06 -0800)]
Reset Table on Inputs Larger than 2KB

6 years agoAvoid DictSmall Checks By Strategically Bumping CurrentOffset
W. Felix Handte [Mon, 5 Mar 2018 16:59:22 +0000 (11:59 -0500)]
Avoid DictSmall Checks By Strategically Bumping CurrentOffset

6 years agoRestore DictIssue Check
W. Felix Handte [Sat, 17 Feb 2018 01:33:51 +0000 (17:33 -0800)]
Restore DictIssue Check

6 years agoAvoid dictionary == NULL Check
W. Felix Handte [Tue, 30 Jan 2018 20:22:29 +0000 (15:22 -0500)]
Avoid dictionary == NULL Check

6 years agoReplace calloc() Calls With malloc() Where Possible
W. Felix Handte [Fri, 26 Jan 2018 22:29:50 +0000 (17:29 -0500)]
Replace calloc() Calls With malloc() Where Possible

6 years agoCopy the Dict Table Into the Context for Large Compressions
W. Felix Handte [Fri, 26 Jan 2018 17:06:43 +0000 (12:06 -0500)]
Copy the Dict Table Into the Context for Large Compressions

6 years agoMake LZ4F_compressFrame_usingCDict Take a Compression Context
W. Felix Handte [Fri, 26 Jan 2018 16:29:00 +0000 (11:29 -0500)]
Make LZ4F_compressFrame_usingCDict Take a Compression Context

6 years agoSwitch Current Offset to 1 Only When in External Dictionary Context Mode
W. Felix Handte [Tue, 13 Feb 2018 21:46:36 +0000 (13:46 -0800)]
Switch Current Offset to 1 Only When in External Dictionary Context Mode

6 years agoHoist Table Reset One Level Up
W. Felix Handte [Mon, 29 Jan 2018 22:09:52 +0000 (17:09 -0500)]
Hoist Table Reset One Level Up

6 years agoSet Dictionary Context Pointer Rather than Copying the Context In
W. Felix Handte [Mon, 12 Feb 2018 17:19:13 +0000 (12:19 -0500)]
Set Dictionary Context Pointer Rather than Copying the Context In

6 years agoLookup Matches in Separate Dictionary Context
W. Felix Handte [Mon, 12 Feb 2018 17:18:24 +0000 (12:18 -0500)]
Lookup Matches in Separate Dictionary Context

6 years agoInitialize Current Offset to 1
W. Felix Handte [Mon, 12 Feb 2018 17:10:15 +0000 (12:10 -0500)]
Initialize Current Offset to 1

6 years agoOnly Re-Alloc / Reset When Needed When Switching Between Regular and High Compression...
W. Felix Handte [Mon, 12 Feb 2018 19:44:53 +0000 (14:44 -0500)]
Only Re-Alloc / Reset When Needed When Switching Between Regular and High Compression Modes

6 years agoAvoid Resetting the Context When Possible
W. Felix Handte [Mon, 12 Feb 2018 17:09:38 +0000 (12:09 -0500)]
Avoid Resetting the Context When Possible

6 years agoConst-ify Table Arg to LZ4_getPosition(OnHash)
W. Felix Handte [Fri, 2 Feb 2018 16:45:59 +0000 (11:45 -0500)]
Const-ify Table Arg to LZ4_getPosition(OnHash)

6 years agoAdd LZ4_compress_fast_safeExtState Function
W. Felix Handte [Fri, 2 Feb 2018 16:41:11 +0000 (11:41 -0500)]
Add LZ4_compress_fast_safeExtState Function

6 years agoAdd Bounds Check to locateBuffDiff
W. Felix Handte [Fri, 2 Feb 2018 16:25:29 +0000 (11:25 -0500)]
Add Bounds Check to locateBuffDiff

6 years agoAdd a Table Type Field to LZ4_stream_t
W. Felix Handte [Fri, 2 Feb 2018 16:11:35 +0000 (11:11 -0500)]
Add a Table Type Field to LZ4_stream_t

6 years agoRemove Obsolete Stream Functions to Free Space in LZ4_stream_t
W. Felix Handte [Tue, 17 Oct 2017 19:23:02 +0000 (15:23 -0400)]
Remove Obsolete Stream Functions to Free Space in LZ4_stream_t

6 years agoAllow Empty Dictionaries
W. Felix Handte [Wed, 11 Oct 2017 20:13:33 +0000 (16:13 -0400)]
Allow Empty Dictionaries

6 years agoAdd a Benchmarking Tool For Compression with Context Re-Use
W. Felix Handte [Mon, 29 Jan 2018 18:20:16 +0000 (13:20 -0500)]
Add a Benchmarking Tool For Compression with Context Re-Use

6 years agoMerge pull request #483 from lz4/dev
Yann Collet [Fri, 9 Mar 2018 21:10:40 +0000 (13:10 -0800)]
Merge pull request #483 from lz4/dev

update to dev

6 years agofix #482: change CFLAGS to CXXFLAGS
Yann Collet [Fri, 9 Mar 2018 19:54:32 +0000 (11:54 -0800)]
fix #482: change CFLAGS to CXXFLAGS

as they are associated with $(CXX)

6 years agofix #481: ensure liblz4.a dependency for `make all`
Yann Collet [Fri, 9 Mar 2018 17:57:29 +0000 (09:57 -0800)]
fix #481: ensure liblz4.a dependency for `make all`

`make all` will trigger several sub-directory makefiles.
several of them need `liblz4.a`.
When built with `-j#`, there are several concurrent liblz4.a built

Make liblz4.a a dependency, which is built once,
before moving to sub-directory Makefiles

6 years agoupdated LZ4F_compressBound() documentation
Yann Collet [Wed, 28 Feb 2018 07:23:27 +0000 (23:23 -0800)]
updated LZ4F_compressBound() documentation

to clarify it includes potentially buffered data.

6 years agoMerge pull request #479 from lz4/check
Yann Collet [Tue, 27 Feb 2018 00:40:32 +0000 (16:40 -0800)]
Merge pull request #479 from lz4/check

added target make check

6 years agoadded target make check
Yann Collet [Mon, 26 Feb 2018 22:09:46 +0000 (14:09 -0800)]
added target make check

according to GNU Makefile conventions,
the Makefile should feature a make check target
to self-test the generated program:
https://www.gnu.org/prep/standards/html_node/Standard-Targets.html .

this is much less thorough and less taxing than `make test`,
and can be run on any target in a reasonable timeframe (several seconds).

6 years agoMerge pull request #478 from lz4/mergeOpt
Yann Collet [Mon, 26 Feb 2018 22:06:31 +0000 (14:06 -0800)]
Merge pull request #478 from lz4/mergeOpt

merge lz4opt.h into lz4hc.c

6 years agobumped version number to v1.8.2
Yann Collet [Mon, 26 Feb 2018 21:50:04 +0000 (13:50 -0800)]
bumped version number to v1.8.2

updated NEWS was current progresses

6 years agoupdate code comment on LZ4 streaming interface
Yann Collet [Mon, 26 Feb 2018 21:31:18 +0000 (13:31 -0800)]
update code comment on LZ4 streaming interface

notably regarding LZ4_saveDict() speed advantage,
answering #477.

6 years agoMerge pull request #476 from lz4/mflimit
Yann Collet [Mon, 26 Feb 2018 20:29:54 +0000 (12:29 -0800)]
Merge pull request #476 from lz4/mflimit

edge case fix : compress up to end-mflimit (12 bytes)

6 years agomerge lz4opt.h into lz4hc.c
Yann Collet [Sun, 25 Feb 2018 08:32:09 +0000 (00:32 -0800)]
merge lz4opt.h into lz4hc.c

Having a dedicated file for optimal parser
made sense during its creation,
it allowed Przemyslaw to work more freely on lz4opt, with less dependency on lz4hc,
moreover, the optimal parser was more complex, with its own search functions.

Since the optimal was rewritten last year, it's now a lot lighter.
It makes more sense now to integrate it directly inside lz4hc.c,
making it easier to edit (editors are a bit "lost" inside a `*.h` dependent on its #include position),
it also reduces the number of files in the project,
which fits pretty well with lz4 objectives.
(adding lz4hc requires "just" lz4hc.h and lz4hc.c).

6 years agoedge case : compress up to end-mflimit (12 bytes)
Yann Collet [Sat, 24 Feb 2018 19:47:53 +0000 (11:47 -0800)]
edge case : compress up to end-mflimit (12 bytes)

The LZ4 block format specification
states that the last match must start
at a minimum distance of 12 bytes from the end of the block.

However, out of an abundance of caution,
the reference implementation would actually stop searching matches
at 13 bytes from the end of the block.

This patch fixes this small detail.
The new version is now able to properly compress a limit case
such as `aaaaaaaabaaa\n`
as reported by Gao Xiang (@hsiangkao).

Obviously, it doesn't change a lot of things.
This is just one additional match candidate per block, with a maximum match length of 7 (since last 5 bytes must remain literals).

With default policy, blocks are 4 MB long, so it doesn't happen too often
Compressing silesia.tar at default level 1 saves 5 bytes (100930101 -> 100930096).
At max level 12, it saves a grand 16 bytes (77389871 -> 77389855).

The impact is a bit more visible when blocks are smaller, hence more numerous.
For example, compressing silesia with blocks of 64 KB (using -12 -B4D) saves 543 bytes (77304583 -> 77304040).
So the smaller the packet size, the more visible the impact.

And it happens we have a ton of scenarios with little blocks using LZ4 compression ...

And a useless "hooray" sidenote :
the patch improves the LZ4 compression record of silesia (using -12 -B7D --no-frame-crc) by 16 bytes (77270672 -> 77270656)
and the record on enwik9 by 44 bytes (371680396 -> 371680352) (previously claimed by [smallz4](http://create.stephan-brumme.com/smallz4/) ).

6 years agoMerge pull request #475 from lz4/betterBench
Yann Collet [Wed, 21 Feb 2018 13:48:58 +0000 (05:48 -0800)]
Merge pull request #475 from lz4/betterBench

Better bench measurements for small inputs

6 years agoMerge pull request #471 from lz4/fasterHC
Yann Collet [Wed, 21 Feb 2018 05:04:07 +0000 (21:04 -0800)]
Merge pull request #471 from lz4/fasterHC

Faster HC