Yann Collet [Mon, 19 Mar 2018 23:18:25 +0000 (16:18 -0700)]
Merge branch 'dev' of github.com:Cyan4973/lz4 into dev
Yann Collet [Mon, 19 Mar 2018 23:18:10 +0000 (16:18 -0700)]
minor man fix on clevels
Yann Collet [Mon, 19 Mar 2018 17:06:33 +0000 (10:06 -0700)]
Merge pull request #484 from lz4/fasterDict
Faster dictionary compression
Yann Collet [Mon, 19 Mar 2018 02:07:55 +0000 (19:07 -0700)]
Merge pull request #406 from felixhandte/ref-dict-table
Use the Dictionary Hash Table in Place
W. Felix Handte [Wed, 31 Jan 2018 23:11:37 +0000 (18:11 -0500)]
Remove Framebench Tool
W. Felix Handte [Wed, 14 Mar 2018 19:58:38 +0000 (15:58 -0400)]
Move LZ4_compress_fast_extState_noReset Declaration to Unstable Section
W. Felix Handte [Wed, 14 Mar 2018 19:51:59 +0000 (15:51 -0400)]
Restore the Other Old Streaming Functions in a Degraded Fashion
W. Felix Handte [Tue, 13 Mar 2018 21:47:34 +0000 (17:47 -0400)]
Switch ALLOC() to ALLOC_AND_ZERO() to Paper Over Existing Uninitialized Read
W. Felix Handte [Tue, 13 Mar 2018 21:45:09 +0000 (17:45 -0400)]
Split lz4CtxLevel into Two Fields
W. Felix Handte [Tue, 13 Mar 2018 21:35:44 +0000 (17:35 -0400)]
Another Allocation Fail Check
W. Felix Handte [Tue, 13 Mar 2018 19:42:03 +0000 (15:42 -0400)]
Restore LZ4_sizeofStreamState, We Didn't Actually Need to Delete It
W. Felix Handte [Tue, 13 Mar 2018 19:18:08 +0000 (15:18 -0400)]
Restore checkTag Cleaning
W. Felix Handte [Tue, 13 Mar 2018 19:07:19 +0000 (15:07 -0400)]
Rename Enums and Add Comment
W. Felix Handte [Mon, 12 Mar 2018 22:46:54 +0000 (18:46 -0400)]
Whitespace Fixes
W. Felix Handte [Mon, 12 Mar 2018 22:32:24 +0000 (18:32 -0400)]
Add NULL Checks
W. Felix Handte [Fri, 9 Mar 2018 17:05:31 +0000 (12:05 -0500)]
Simpler Ternary Statements
W. Felix Handte [Mon, 12 Mar 2018 22:13:24 +0000 (18:13 -0400)]
Renames and Comment Fixes
W. Felix Handte [Mon, 12 Mar 2018 20:11:55 +0000 (16:11 -0400)]
Hoist LZ4F Dictionary Setup into Helper LZ4F_applyCDict()
W. Felix Handte [Mon, 12 Mar 2018 20:11:44 +0000 (16:11 -0400)]
Minor Style Fixes
W. Felix Handte [Thu, 8 Mar 2018 19:09:06 +0000 (14:09 -0500)]
Preserve currentOffset==0 When Possible
W. Felix Handte [Fri, 9 Mar 2018 17:14:42 +0000 (12:14 -0500)]
Specialize _extState() for Clean Ctx Rather Than Calling _safeExtState()
W. Felix Handte [Thu, 8 Mar 2018 17:30:34 +0000 (12:30 -0500)]
Remove Switch In Favor of Ternary Statement
W. Felix Handte [Thu, 8 Mar 2018 17:29:45 +0000 (12:29 -0500)]
Further Avoid a dictionary==NULL Check
W. Felix Handte [Tue, 6 Mar 2018 20:53:22 +0000 (15:53 -0500)]
Optimize Dict Check Condition
W. Felix Handte [Tue, 6 Mar 2018 16:52:02 +0000 (11:52 -0500)]
Move to 4KB Cut-Off
W. Felix Handte [Wed, 14 Feb 2018 01:06:24 +0000 (17:06 -0800)]
Reset Table on Inputs Larger than 2KB
W. Felix Handte [Mon, 5 Mar 2018 16:59:22 +0000 (11:59 -0500)]
Avoid DictSmall Checks By Strategically Bumping CurrentOffset
W. Felix Handte [Sat, 17 Feb 2018 01:33:51 +0000 (17:33 -0800)]
Restore DictIssue Check
W. Felix Handte [Tue, 30 Jan 2018 20:22:29 +0000 (15:22 -0500)]
Avoid dictionary == NULL Check
W. Felix Handte [Fri, 26 Jan 2018 22:29:50 +0000 (17:29 -0500)]
Replace calloc() Calls With malloc() Where Possible
W. Felix Handte [Fri, 26 Jan 2018 17:06:43 +0000 (12:06 -0500)]
Copy the Dict Table Into the Context for Large Compressions
W. Felix Handte [Fri, 26 Jan 2018 16:29:00 +0000 (11:29 -0500)]
Make LZ4F_compressFrame_usingCDict Take a Compression Context
W. Felix Handte [Tue, 13 Feb 2018 21:46:36 +0000 (13:46 -0800)]
Switch Current Offset to 1 Only When in External Dictionary Context Mode
W. Felix Handte [Mon, 29 Jan 2018 22:09:52 +0000 (17:09 -0500)]
Hoist Table Reset One Level Up
W. Felix Handte [Mon, 12 Feb 2018 17:19:13 +0000 (12:19 -0500)]
Set Dictionary Context Pointer Rather than Copying the Context In
W. Felix Handte [Mon, 12 Feb 2018 17:18:24 +0000 (12:18 -0500)]
Lookup Matches in Separate Dictionary Context
W. Felix Handte [Mon, 12 Feb 2018 17:10:15 +0000 (12:10 -0500)]
Initialize Current Offset to 1
W. Felix Handte [Mon, 12 Feb 2018 19:44:53 +0000 (14:44 -0500)]
Only Re-Alloc / Reset When Needed When Switching Between Regular and High Compression Modes
W. Felix Handte [Mon, 12 Feb 2018 17:09:38 +0000 (12:09 -0500)]
Avoid Resetting the Context When Possible
W. Felix Handte [Fri, 2 Feb 2018 16:45:59 +0000 (11:45 -0500)]
Const-ify Table Arg to LZ4_getPosition(OnHash)
W. Felix Handte [Fri, 2 Feb 2018 16:41:11 +0000 (11:41 -0500)]
Add LZ4_compress_fast_safeExtState Function
W. Felix Handte [Fri, 2 Feb 2018 16:25:29 +0000 (11:25 -0500)]
Add Bounds Check to locateBuffDiff
W. Felix Handte [Fri, 2 Feb 2018 16:11:35 +0000 (11:11 -0500)]
Add a Table Type Field to LZ4_stream_t
W. Felix Handte [Tue, 17 Oct 2017 19:23:02 +0000 (15:23 -0400)]
Remove Obsolete Stream Functions to Free Space in LZ4_stream_t
W. Felix Handte [Wed, 11 Oct 2017 20:13:33 +0000 (16:13 -0400)]
Allow Empty Dictionaries
W. Felix Handte [Mon, 29 Jan 2018 18:20:16 +0000 (13:20 -0500)]
Add a Benchmarking Tool For Compression with Context Re-Use
Yann Collet [Fri, 9 Mar 2018 21:10:40 +0000 (13:10 -0800)]
Merge pull request #483 from lz4/dev
update to dev
Yann Collet [Fri, 9 Mar 2018 19:54:32 +0000 (11:54 -0800)]
fix #482: change CFLAGS to CXXFLAGS
as they are associated with $(CXX)
Yann Collet [Fri, 9 Mar 2018 17:57:29 +0000 (09:57 -0800)]
fix #481: ensure liblz4.a dependency for `make all`
`make all` will trigger several sub-directory makefiles.
several of them need `liblz4.a`.
When built with `-j#`, there are several concurrent liblz4.a built
Make liblz4.a a dependency, which is built once,
before moving to sub-directory Makefiles
Yann Collet [Wed, 28 Feb 2018 07:23:27 +0000 (23:23 -0800)]
updated LZ4F_compressBound() documentation
to clarify it includes potentially buffered data.
Yann Collet [Tue, 27 Feb 2018 00:40:32 +0000 (16:40 -0800)]
Merge pull request #479 from lz4/check
added target make check
Yann Collet [Mon, 26 Feb 2018 22:09:46 +0000 (14:09 -0800)]
added target make check
according to GNU Makefile conventions,
the Makefile should feature a make check target
to self-test the generated program:
https://www.gnu.org/prep/standards/html_node/Standard-Targets.html .
this is much less thorough and less taxing than `make test`,
and can be run on any target in a reasonable timeframe (several seconds).
Yann Collet [Mon, 26 Feb 2018 22:06:31 +0000 (14:06 -0800)]
Merge pull request #478 from lz4/mergeOpt
merge lz4opt.h into lz4hc.c
Yann Collet [Mon, 26 Feb 2018 21:50:04 +0000 (13:50 -0800)]
bumped version number to v1.8.2
updated NEWS was current progresses
Yann Collet [Mon, 26 Feb 2018 21:31:18 +0000 (13:31 -0800)]
update code comment on LZ4 streaming interface
notably regarding LZ4_saveDict() speed advantage,
answering #477.
Yann Collet [Mon, 26 Feb 2018 20:29:54 +0000 (12:29 -0800)]
Merge pull request #476 from lz4/mflimit
edge case fix : compress up to end-mflimit (12 bytes)
Yann Collet [Sun, 25 Feb 2018 08:32:09 +0000 (00:32 -0800)]
merge lz4opt.h into lz4hc.c
Having a dedicated file for optimal parser
made sense during its creation,
it allowed Przemyslaw to work more freely on lz4opt, with less dependency on lz4hc,
moreover, the optimal parser was more complex, with its own search functions.
Since the optimal was rewritten last year, it's now a lot lighter.
It makes more sense now to integrate it directly inside lz4hc.c,
making it easier to edit (editors are a bit "lost" inside a `*.h` dependent on its #include position),
it also reduces the number of files in the project,
which fits pretty well with lz4 objectives.
(adding lz4hc requires "just" lz4hc.h and lz4hc.c).
Yann Collet [Sat, 24 Feb 2018 19:47:53 +0000 (11:47 -0800)]
edge case : compress up to end-mflimit (12 bytes)
The LZ4 block format specification
states that the last match must start
at a minimum distance of 12 bytes from the end of the block.
However, out of an abundance of caution,
the reference implementation would actually stop searching matches
at 13 bytes from the end of the block.
This patch fixes this small detail.
The new version is now able to properly compress a limit case
such as `
aaaaaaaabaaa\n`
as reported by Gao Xiang (@hsiangkao).
Obviously, it doesn't change a lot of things.
This is just one additional match candidate per block, with a maximum match length of 7 (since last 5 bytes must remain literals).
With default policy, blocks are 4 MB long, so it doesn't happen too often
Compressing silesia.tar at default level 1 saves 5 bytes (
100930101 ->
100930096).
At max level 12, it saves a grand 16 bytes (
77389871 ->
77389855).
The impact is a bit more visible when blocks are smaller, hence more numerous.
For example, compressing silesia with blocks of 64 KB (using -12 -B4D) saves 543 bytes (
77304583 ->
77304040).
So the smaller the packet size, the more visible the impact.
And it happens we have a ton of scenarios with little blocks using LZ4 compression ...
And a useless "hooray" sidenote :
the patch improves the LZ4 compression record of silesia (using -12 -B7D --no-frame-crc) by 16 bytes (
77270672 ->
77270656)
and the record on enwik9 by 44 bytes (
371680396 ->
371680352) (previously claimed by [smallz4](http://create.stephan-brumme.com/smallz4/) ).
Yann Collet [Wed, 21 Feb 2018 13:48:58 +0000 (05:48 -0800)]
Merge pull request #475 from lz4/betterBench
Better bench measurements for small inputs
Yann Collet [Wed, 21 Feb 2018 05:04:07 +0000 (21:04 -0800)]
Merge pull request #471 from lz4/fasterHC
Faster HC
Yann Collet [Tue, 20 Feb 2018 23:26:59 +0000 (15:26 -0800)]
use TIMELOOP_NANOSEC,
as suggested by @terrelln
Yann Collet [Tue, 20 Feb 2018 23:25:45 +0000 (15:25 -0800)]
added one assert()
suggested by @terrelln
Yann Collet [Tue, 20 Feb 2018 22:13:13 +0000 (14:13 -0800)]
fixed minor conversion warning
Yann Collet [Tue, 20 Feb 2018 21:05:22 +0000 (13:05 -0800)]
ensure bench speed measurement is more accurate for small inputs
Previous method would produce too many time() invocations,
becoming a significant fraction of workload measured.
The new strategy is to use time() only once per batch,
and dynamically resize batch size so that each round lasts approximately 1 second.
This only matters for small inputs.
Measurement for large files (such as silesia.tar) are much less impacted
(though decoding speed is so fast that even medium-size files will notice an improvement).
Yann Collet [Tue, 20 Feb 2018 19:37:19 +0000 (11:37 -0800)]
update bench.c to use less time invocations
translating into more accurate speed measurements for small sources
Yann Collet [Sun, 18 Feb 2018 19:00:33 +0000 (11:00 -0800)]
update API doc regarding double-buffer strategy
answering question #473
Yann Collet [Wed, 14 Feb 2018 21:00:50 +0000 (13:00 -0800)]
Merge pull request #472 from hobomind/dev
fix: missed semicolon at programs/lz4io.c:954
hobomind [Wed, 14 Feb 2018 15:47:56 +0000 (18:47 +0300)]
fix: missed semicolon at programs/lz4io.c:954
Yann Collet [Tue, 13 Feb 2018 00:56:45 +0000 (16:56 -0800)]
Merge pull request #470 from lz4/fasterDec
Faster decoding speed
Yann Collet [Mon, 12 Feb 2018 05:03:39 +0000 (21:03 -0800)]
slight hc speed benefit (~+1%)
by optimizing countback
Yann Collet [Mon, 12 Feb 2018 06:20:09 +0000 (22:20 -0800)]
removed LZ4_copy8
better use memcpy() directly
Yann Collet [Sun, 11 Feb 2018 10:45:36 +0000 (02:45 -0800)]
slightly improved hc compression speed (+~1-2%)
by removing bad candidates faster.
Yann Collet [Sun, 11 Feb 2018 09:42:12 +0000 (01:42 -0800)]
slightly improved decompression speed (~+1-2%)
by making shortcut slightly more common
Yann Collet [Thu, 8 Feb 2018 16:45:25 +0000 (08:45 -0800)]
Merge pull request #469 from mathstuf/intel-windows-packing-selection
intel: do not use __attribute__((packed)) on Windows
Ben Boeckel [Thu, 8 Feb 2018 14:15:07 +0000 (09:15 -0500)]
intel: do not use __attribute__((packed)) on Windows
On Windows, the Intel compiler is closer to MSVC rather than GCC and
does not support the GCC attribute syntax.
Fixes #468
Yann Collet [Wed, 7 Feb 2018 10:21:25 +0000 (02:21 -0800)]
fixed code comment as detected in #466
Also clarified a few API code comments
and updated associated html documentation
Yann Collet [Fri, 2 Feb 2018 00:14:54 +0000 (16:14 -0800)]
Merge pull request #461 from terrelln/docs
Clarify the requirements of the LZ4 streaming API
Nick Terrell [Fri, 2 Feb 2018 00:08:59 +0000 (16:08 -0800)]
Clarify the requirements of the LZ4 streaming API
Yann Collet [Thu, 1 Feb 2018 18:55:02 +0000 (10:55 -0800)]
Merge pull request #458 from lz4/ff161
Minor change to LZ4 Frame format specification
Yann Collet [Thu, 1 Feb 2018 16:53:23 +0000 (08:53 -0800)]
Merge pull request #460 from lz4/frameCompress
refactored frameCompress.c example
Yann Collet [Thu, 1 Feb 2018 10:48:20 +0000 (02:48 -0800)]
modified decompression part of frameCompress.c
using same logic as prior modifications for compression part.
Yann Collet [Thu, 1 Feb 2018 09:36:38 +0000 (01:36 -0800)]
refactored ressource allocation to avoid goto
Yann Collet [Thu, 1 Feb 2018 07:17:52 +0000 (23:17 -0800)]
examples/Makefile : changed dependency order
static library *.a must come after source files *.c on linux
Yann Collet [Thu, 1 Feb 2018 01:18:57 +0000 (17:18 -0800)]
Merge branch 'dev' into frameCompress
Yann Collet [Thu, 1 Feb 2018 01:16:48 +0000 (17:16 -0800)]
travisci : ensure "clean" betweeb 2 tests
Yann Collet [Thu, 1 Feb 2018 01:15:02 +0000 (17:15 -0800)]
modified gpptest recipe
Yann Collet [Thu, 1 Feb 2018 00:42:13 +0000 (16:42 -0800)]
Merge pull request #459 from Tyilo/clang-stdc++14
Ensure LZ4_DEPRECATED("...") is before LZ4LIB_API
Yann Collet [Thu, 1 Feb 2018 00:39:37 +0000 (16:39 -0800)]
refactored frameCompress.c example code
compression function returns a struct.
Also : nested structure ressources->computation
to make it easier to manage multiple exit points.
Yann Collet [Thu, 1 Feb 2018 00:13:02 +0000 (16:13 -0800)]
fixed read size, as noticed by @terrelln
Yann Collet [Thu, 1 Feb 2018 00:11:45 +0000 (16:11 -0800)]
ensure proper dependencies are built for /examples
also : use liblz4.a static lib to share compilation time
Yann Collet [Wed, 31 Jan 2018 22:33:16 +0000 (14:33 -0800)]
refactored frameCompress example
to better reflect LZ4F API usage.
Asger Hautop Drewsen [Wed, 31 Jan 2018 19:23:20 +0000 (20:23 +0100)]
Always prefer c++14 attributes if available
Yann Collet [Wed, 31 Jan 2018 17:54:30 +0000 (09:54 -0800)]
fix typos as suggested by @psteinb
Asger Hautop Drewsen [Wed, 31 Jan 2018 12:33:07 +0000 (13:33 +0100)]
Ensure LZ4_DEPRECATED("...") is before LZ4LIB_API
When using clang++ with std c++14 or c++17 you would get the error "an attribute list cannot appear here" when including "lz4.h" as the visibility attribute is before the c++ attribute.
This ensures that the [[deprecated]] c++ attribute is before everything
else in the function declarations.
Yann Collet [Wed, 31 Jan 2018 01:53:00 +0000 (17:53 -0800)]
proposed a minor change to LZ4 Frame format specification
add new terms "LZ4 Frame Header" and "LZ4 Frame Footer"
Yann Collet [Mon, 22 Jan 2018 21:25:49 +0000 (13:25 -0800)]
Merge pull request #455 from terrelln/hc-default
[lz4hc] level == 0 means default, not level 1
Nick Terrell [Mon, 22 Jan 2018 20:50:06 +0000 (12:50 -0800)]
[lz4hc] level == 0 means default, not level 1
Yann Collet [Wed, 17 Jan 2018 20:45:09 +0000 (12:45 -0800)]
Merge pull request #453 from lz4/clockfix
fix #450 : use more restrictive conditions for clock_gettime()
Yann Collet [Wed, 17 Jan 2018 19:29:27 +0000 (11:29 -0800)]
Merge pull request #454 from sunpoet/dev
Change file format back to ASCII (from UTF-8)
Po-Chuan Hsieh [Wed, 17 Jan 2018 19:13:05 +0000 (03:13 +0800)]
Change file format back to ASCII (from UTF-8)
- Replace U+00A0 by space
- Fix build failure of archivers/py-borgbackup in FreeBSD
Reference: https://bugs.FreeBSD.org/225235