Yann Collet [Tue, 18 Aug 2020 17:24:36 +0000 (10:24 -0700)]
Merge pull request #897 from lz4/lz4wlib
added target lz4-wlib
Yann Collet [Fri, 14 Aug 2020 22:48:21 +0000 (15:48 -0700)]
Merge pull request #899 from lz4/endMark
Clarifies and fix EndMark
Yann Collet [Fri, 14 Aug 2020 22:03:03 +0000 (15:03 -0700)]
clarify endMark definition
Yann Collet [Thu, 13 Aug 2020 00:27:33 +0000 (17:27 -0700)]
Clarifies and fix EndMark
EndMark, the 4-bytes value indicating the end of frame,
must be `0x00000000`.
Previously, it was just mentioned as a `0-size` block.
But such definition could encompass uncompressed blocks of size 0,
with a header of value `0x80000000`.
But the intention was to also support uncompressed empty blocks.
They could be used as a keep-alive signal.
Note that compressed empty blocks are already supported,
it's just that they have a size 1 instead of 0 (for the `0` token).
Unfortunately, the decoder implementation was also wrong,
and would also interpret a `0x80000000` block header as an endMark.
This issue evaded detection so far simply because
this situation never happens, as LZ4Frame always issues
a clean 0x00000000 value as a endMark.
It also does not flush empty blocks.
This is fixed in this PR.
The decoder can now deal with empty uncompressed blocks,
and do not confuse them with EndMark.
The specification is also clarified.
Finally, FrameTest is updated to randomly insert empty blocks during fuzzing.
Yann Collet [Tue, 11 Aug 2020 21:03:20 +0000 (14:03 -0700)]
added target lz4-wlib
variant of lz4 linking to liblz4 dynamic library
requires the dynamic library to expose static-only symbols (experimental API)
Example for #888
Yann Collet [Tue, 11 Aug 2020 01:17:09 +0000 (18:17 -0700)]
Merge branch 'fix832' into dev
Yann Collet [Tue, 11 Aug 2020 00:11:49 +0000 (17:11 -0700)]
fixed test of gnu c version
Yann Collet [Mon, 10 Aug 2020 21:31:48 +0000 (14:31 -0700)]
Merge pull request #896 from lz4/fix832
fix #832
Yann Collet [Mon, 10 Aug 2020 19:52:32 +0000 (12:52 -0700)]
Merge pull request #895 from lz4/hugefast
Fix #876
Yann Collet [Mon, 10 Aug 2020 18:51:57 +0000 (11:51 -0700)]
fix #832
does no longer rely on default 0-interpretation when __GNUC__ is not defined
Yann Collet [Mon, 10 Aug 2020 18:28:46 +0000 (11:28 -0700)]
Merge pull request #892 from felixhandte/lz4-clean-up-fast-reset
Remove dirty Flag in LZ4_stream_t
Yann Collet [Mon, 10 Aug 2020 18:03:27 +0000 (11:03 -0700)]
fix #876
by introducing a max limit acceleration value
Yann Collet [Mon, 10 Aug 2020 17:54:35 +0000 (10:54 -0700)]
add test that breaks --fast
with huge values, as expected from #876
Also : added target `check`
W. Felix Handte [Mon, 10 Aug 2020 17:46:31 +0000 (13:46 -0400)]
Fix Enum Casts
Fixes `-Wsign-compare` issues.
Yann Collet [Sat, 8 Aug 2020 20:21:07 +0000 (13:21 -0700)]
Merge pull request #856 from indragiek/indragiek/lz4f-visibility
Add LZ4FLIB_VISIBILITY for controlling library symbol visibility
W. Felix Handte [Thu, 6 Aug 2020 20:06:40 +0000 (16:06 -0400)]
Remove dirty Field From LZ4_stream_t
Indragie Karunaratne [Wed, 5 Aug 2020 21:34:07 +0000 (15:34 -0600)]
LZ4LIB_VISIBILITY -> LZ4FLIB_VISIBILITY in comment
W. Felix Handte [Wed, 5 Aug 2020 16:46:32 +0000 (12:46 -0400)]
Remove Extraneous Reset in LZ4_attach_dictionary()
Nothing internally sets dirty anymore. The only way to get that is if you use
an uninitialized context, in which case your warranty is void anyways.
Yann Collet [Tue, 4 Aug 2020 01:34:16 +0000 (18:34 -0700)]
Merge pull request #890 from terrelln/lz4-memcpy
Call LZ4_memcpy() instead of memcpy()
Nick Terrell [Mon, 3 Aug 2020 18:18:35 +0000 (11:18 -0700)]
Call LZ4_memcpy() instead of memcpy()
`LZ4_memcpy()` uses `__builtin_memcpy()` to ensure that clang/gcc
can inline the `memcpy()` calls in freestanding mode.
This is necessary for decompressing the Linux Kernel with LZ4.
Without an analogous patch decompression ran at 77 MB/s, and with
the patch it ran at 884 MB/s.
Yann Collet [Mon, 3 Aug 2020 17:47:24 +0000 (10:47 -0700)]
Merge pull request #885 from neheb/mes
meson: remove build requirement for distutils
Yann Collet [Wed, 29 Jul 2020 00:44:25 +0000 (17:44 -0700)]
Merge pull request #860 from adeason/old-style-definitions
Avoid old-style function definitions
Rosen Penev [Tue, 21 Jul 2020 21:04:17 +0000 (14:04 -0700)]
meson: remove build requirement for distutils
Tested on a default installation of Debian 10.
Same as https://github.com/facebook/zstd/pull/2197
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Yann Collet [Thu, 16 Jul 2020 16:44:40 +0000 (09:44 -0700)]
Merge pull request #863 from Devernua/reducing_stack_usage_in_t_alignment
Reducing stack usage in _t_alignment checks
Yann Collet [Tue, 14 Jul 2020 01:09:02 +0000 (18:09 -0700)]
Merge pull request #866 from sandyharvie/dev
Fix issue #865
Yann Collet [Tue, 14 Jul 2020 01:07:05 +0000 (18:07 -0700)]
Merge pull request #871 from bimbashrestha/bsd
[build] FreeBSD update version 12.0 -> 12.1
Yann Collet [Wed, 8 Jul 2020 16:12:25 +0000 (09:12 -0700)]
Merge pull request #884 from vectorizedio/clang-ubsan
avoid computing 0 offsets from null pointers
Alexander Gallego [Wed, 8 Jul 2020 15:21:45 +0000 (08:21 -0700)]
avoid computing 0 offsets from null pointers
Similar work in the kernel:
https://patchwork.kernel.org/patch/
11351499/
UBsan (+clang-10) complains about doing pointer arithmetic (adding 0)
to a nullpointer.
This patch is tested with clang-10+ubsan
Yann Collet [Sat, 13 Jun 2020 21:31:15 +0000 (14:31 -0700)]
Update issue templates
Bimba Shrestha [Fri, 5 Jun 2020 16:57:44 +0000 (11:57 -0500)]
bsd upsate to 12-1
Yann Collet [Sat, 30 May 2020 07:07:48 +0000 (00:07 -0700)]
Merge pull request #869 from bket/fix_tests_openbsd
Fix tests on OpenBSD
Björn Ketelaars [Sat, 30 May 2020 05:36:22 +0000 (07:36 +0200)]
Fix tests on OpenBSD
Add condition for OpenBSD, which uses gmake
Christopher Harvie [Thu, 14 May 2020 03:01:43 +0000 (23:01 -0400)]
add tests
Christopher Harvie [Thu, 14 May 2020 01:48:12 +0000 (21:48 -0400)]
fix LZ4F_compressBound when automatic flushing is enabled
Aleksandr Kukuev [Mon, 11 May 2020 20:32:02 +0000 (23:32 +0300)]
Reducing stack usage in _t_alignment checks
Andrew Deason [Wed, 6 May 2020 19:36:56 +0000 (14:36 -0500)]
Avoid old-style function definitions
Define 0-argument functions like foo(void) instead of foo(), in order
to avoid a warning with -Wold-style-definition. This makes it easier
to embed lz4.c in projects that compile with -Werror
-Wold-style-definition.
Indragie Karunaratne [Sun, 26 Apr 2020 21:49:34 +0000 (14:49 -0700)]
Add LZ4FLIB_VISIBILITY for controlling library symbol visibility
Yann Collet [Tue, 3 Mar 2020 22:17:52 +0000 (14:17 -0800)]
Merge pull request #846 from bimbashrestha/issues-tmp
Adding issue templates
Bimba Shrestha [Tue, 3 Mar 2020 21:29:57 +0000 (13:29 -0800)]
Adding issue templates
Yann Collet [Wed, 26 Feb 2020 18:06:54 +0000 (10:06 -0800)]
Merge pull request #838 from Yanpas/stack_frame_2
fix: allocate LZ4HC_optimal_t opt on heap each time (#837)
Yann Collet [Sat, 8 Feb 2020 02:49:24 +0000 (18:49 -0800)]
Merge pull request #843 from filipecalasans/multiples-files-with-legacy
Implement -m option with legacy format on cli
Filipe Calasans [Sat, 8 Feb 2020 01:06:40 +0000 (17:06 -0800)]
Add test scenario test-lz4-multiple-legacy
Filipe Calasans [Fri, 7 Feb 2020 22:52:25 +0000 (14:52 -0800)]
Implement -m option with legacy format on cli
Yann Collet [Fri, 7 Feb 2020 18:17:33 +0000 (10:17 -0800)]
Merge pull request #842 from filipecalasans/fix-list
Fix lz4cli --list option
Filipe Calasans [Fri, 7 Feb 2020 05:39:54 +0000 (21:39 -0800)]
Fix lz4cli --list option
Yan Pashkovsky [Mon, 3 Feb 2020 13:50:02 +0000 (16:50 +0300)]
fix malloc handling
Yan Pashkovsky [Fri, 31 Jan 2020 16:44:56 +0000 (19:44 +0300)]
fix: allocate LZ4HC_optimal_t opt on heap each time (#837)
Yann Collet [Mon, 27 Jan 2020 21:51:06 +0000 (13:51 -0800)]
updated NEWS for v1.9.3
Yann Collet [Mon, 27 Jan 2020 21:45:02 +0000 (13:45 -0800)]
Merge pull request #826 from wolfpld/dev
Force inline small functions used by LZ4_compress_generic.
Bartosz Taudul [Thu, 16 Jan 2020 23:37:47 +0000 (00:37 +0100)]
Force inline small functions used by LZ4_compress_generic.
Yann Collet [Thu, 16 Jan 2020 17:19:02 +0000 (09:19 -0800)]
Merge pull request #825 from NancyLi1013/vcpkg-instructions
Add vcpkg installation instructions
NancyLi1013 [Thu, 16 Jan 2020 08:49:23 +0000 (00:49 -0800)]
Add vcpkg installation instructions
Yann Collet [Fri, 6 Dec 2019 04:32:02 +0000 (20:32 -0800)]
Merge pull request #810 from korli/haiku
bring support for Haiku up to date.
Yann Collet [Wed, 4 Dec 2019 00:37:17 +0000 (16:37 -0800)]
Merge pull request #803 from dkondrad/gh-797
VS2017: Add lz4 project
Yann Collet [Wed, 4 Dec 2019 00:34:11 +0000 (16:34 -0800)]
Merge pull request #817 from lz4/windfast
faster decoding speed with Visual
Yann Collet [Tue, 3 Dec 2019 22:49:22 +0000 (14:49 -0800)]
fixed lz4hc assert error
when src ptr is in very low memory area (< 64K),
the virtual reference to data in dictionary
might end up in a very high memory address.
Since it's not a "real" memory address,
just a virtual one, to calculate distance,
it doesn't matter : only distance matters.
The assert was to restrictive.
Fixed.
Yann Collet [Tue, 3 Dec 2019 00:38:33 +0000 (16:38 -0800)]
faster decoding speed with Visual
by enabling the fast decoder path.
Visual requires a different set of macro constants to detect x86 / x64.
On my laptop, decoding speed on x64 went up from 3.12 to 3.45 GB/s.
32-bit is less impressive, though still favorable,
with speed increasing from 2.55 to 2.60 GB/s.
So both cases are now enabled.
Suggested by Bartosz Taudul (@wolfpld).
Yann Collet [Sat, 30 Nov 2019 16:16:58 +0000 (08:16 -0800)]
Merge pull request #815 from andrewthad/patch-1
Fix typos in streaming_api_basics.md
Andrew Martin [Sat, 30 Nov 2019 11:58:54 +0000 (06:58 -0500)]
Fix typos in streaming_api_basics.md
Jerome Duval [Sun, 8 Sep 2019 11:36:26 +0000 (13:36 +0200)]
bring support for Haiku up to date.
Yann Collet [Wed, 6 Nov 2019 17:36:56 +0000 (09:36 -0800)]
Merge pull request #808 from rkoradi/benchmarkWithDictionary
Make benchmark compatible with dictionary compression
Reto Koradi [Mon, 4 Nov 2019 19:51:41 +0000 (11:51 -0800)]
Make benchmark compatible with dictionary compression
Support the -D command line option for running benchmarks. The
benchmark code was slightly restructured to factor out the calls
that need to be different for each benchmark scenario. Since there
are now 4 scenarios (all combinations of fast/HC and with/without
dictionary), the logic was getting somewhat convoluted otherwise.
This was done by extending the compressionParameters struct that
previously contained just a single function pointer. It now
contains 4 function pointers for init/reset/compress/cleanup,
with the related state. The functions get a pointer to the
structure as their first argument (inspired by C++), so that they
can access the state values in the struct.
David A Kondrad [Fri, 25 Oct 2019 19:04:27 +0000 (15:04 -0400)]
project: VS2017: add lz4 project
Add missing lz4 project to the VS2017 solution.
The project was made from scratch rather than importing and updated to
match the VS2010 options.
Any new options not present in the old VS2010 project we pulled from
other console binaries in the solution.
Signed-off-by: David A Kondrad <dkondrad.mscs+patches@gmail.com>
David A Kondrad [Fri, 25 Oct 2019 18:51:36 +0000 (14:51 -0400)]
project: visual: fix VS2010/17 gitignore
The `lz4` project was excluded from being checked in for VS2017.
Add some fixups to explicitly allow this project directory.
Also add an exclusion for VS2017 binaries and VS `ipch` directories.
Signed-off-by: David A Kondrad <dkondrad.mscs+patches@gmail.com>
Yann Collet [Mon, 7 Oct 2019 16:51:15 +0000 (09:51 -0700)]
Merge pull request #798 from bimbashrestha/adding_cirrus_test
Adding cirrus test for FreeBSD
Bimba Shrestha [Mon, 7 Oct 2019 15:12:42 +0000 (08:12 -0700)]
Adding unamestr var
Bimba Shrestha [Fri, 4 Oct 2019 17:12:22 +0000 (10:12 -0700)]
Adding condition for FreeBSD and using gmake
Bimba Shrestha [Fri, 4 Oct 2019 16:29:58 +0000 (09:29 -0700)]
Using instead of gmake (to address the travis failure)
Bimba Shrestha [Fri, 4 Oct 2019 15:40:46 +0000 (08:40 -0700)]
Using gmake instead of make
Bimba Shrestha [Fri, 4 Oct 2019 15:36:08 +0000 (08:36 -0700)]
Adding cirrus config file for freebsd-12-0
Yann Collet [Thu, 26 Sep 2019 20:01:55 +0000 (13:01 -0700)]
Merge pull request #796 from jcaesar/dev
meson: move one layer deeper to allow easy construction of a wrap file
Julius Michaelis [Thu, 26 Sep 2019 08:23:53 +0000 (17:23 +0900)]
meson: move one layer deeper to allow easy construction of a wrap file
Yann Collet [Mon, 23 Sep 2019 19:50:05 +0000 (12:50 -0700)]
Merge pull request #794 from bimbashrestha/compress_frame_fuzzer_heap_overflow
Using size instead of LZ4_compressBound(size) <- causes heap overflow
Bimba Shrestha [Mon, 23 Sep 2019 18:54:56 +0000 (11:54 -0700)]
Using size instead of LZ4_compressBound(size) <- causes heap overflow
Yann Collet [Sat, 21 Sep 2019 16:55:59 +0000 (09:55 -0700)]
Merge pull request #793 from nigeltao/dev
Have read_variable_length use fixed size types
Nigel Tao [Sun, 15 Sep 2019 23:20:02 +0000 (09:20 +1000)]
Have read_variable_length use fixed size types
Otherwise, the output from decoding LZ4-compressed input could be
platform dependent.
Also add a compile-time check to confirm the existing code's assumptions
that, if <stdint.h> isn't used, then sizeof(int) == 4.
Updates #792
Yann Collet [Wed, 18 Sep 2019 17:21:43 +0000 (10:21 -0700)]
Merge pull request #790 from bimbashrestha/seperating_seed_generation_and_use_in_fuzzers
Separating the seed generation and use in FUZZ_dataProducer api
Bimba Shrestha [Sat, 14 Sep 2019 01:08:58 +0000 (18:08 -0700)]
Retreiving 32 bits from the end for fuzzer
Bimba Shrestha [Fri, 13 Sep 2019 23:04:48 +0000 (16:04 -0700)]
Addressing naming nits and moving size modification up in all fuzzers
Bimba Shrestha [Fri, 13 Sep 2019 21:07:52 +0000 (14:07 -0700)]
Seperating the seed generation and use
Yann Collet [Tue, 10 Sep 2019 19:56:53 +0000 (12:56 -0700)]
Merge pull request #770 from neheb/dev
util.h: Remove deprecated utime for non-Windows
Rosen Penev [Wed, 31 Jul 2019 05:13:51 +0000 (22:13 -0700)]
util.h: Remove deprecated utime for non-Windows
utime was deprecated in POSIX 2008.
Yann Collet [Mon, 9 Sep 2019 19:03:26 +0000 (12:03 -0700)]
Merge pull request #785 from bimbashrestha/transfer_remaining_fuzzers_to_consume_from_end_of_input
Making fuzzers use dataProducer api instead of random seed for decisions
Bimba Shrestha [Fri, 30 Aug 2019 17:27:42 +0000 (10:27 -0700)]
Making fuzzers use dataProducer api instead of random seed for decisions
Yann Collet [Wed, 21 Aug 2019 11:44:24 +0000 (13:44 +0200)]
fixed #778
fixed assert() when divisor == 0
Yann Collet [Tue, 20 Aug 2019 23:48:44 +0000 (16:48 -0700)]
Merge pull request #779 from bimbashrestha/dev
Adding fuzz data producer for uint32 and using in decompress_fuzzer
Yann Collet [Tue, 20 Aug 2019 17:35:58 +0000 (10:35 -0700)]
Merge pull request #780 from lz4/dev
v1.9.2
bimbashrestha [Sat, 17 Aug 2019 00:14:47 +0000 (17:14 -0700)]
Adding comments, fixing nit, and hiding the struct in data producer api
bimbashrestha [Fri, 16 Aug 2019 23:43:28 +0000 (16:43 -0700)]
Seperating fuzz data producer api impl and header, using data producer on the easy fuzzers
bimbashrestha [Fri, 16 Aug 2019 21:19:06 +0000 (14:19 -0700)]
Created a data producer API and used in decompress_fuzzer
bimbashrestha [Fri, 16 Aug 2019 17:50:46 +0000 (10:50 -0700)]
Adding fuzz data producer for uint32 and using in decompress_fuzzer
Summary: Consuming bytes from the end of data instead of from the front to prevent "all-in-one" decisions.
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Yann Collet [Thu, 15 Aug 2019 11:59:59 +0000 (13:59 +0200)]
fixed test error
could trigger %0 on exceptional circumstances
due to wrong buffer size parameter.
Yann Collet [Sat, 10 Aug 2019 00:08:03 +0000 (02:08 +0200)]
Merge pull request #777 from terrelln/off-by-one
[LZ4_compress_destSize] Fix off-by-one error
Nick Terrell [Fri, 9 Aug 2019 17:32:26 +0000 (10:32 -0700)]
[LZ4_compress_destSize] Fix off-by-one error
PR#756 fixed the data corruption bug, but didn't clear `ip`. PR#760
fixed that off-by-one error, but missed the case where `ip == filledIp`,
which is harder for the fuzzers to find (it took 20 days not 1 day).
Verified this fixed the issue reported by OSS-Fuzz.
Credit to OSS-Fuzz.
Yann Collet [Wed, 7 Aug 2019 23:48:53 +0000 (01:48 +0200)]
Merge pull request #773 from felixhandte/attach-empty-dict-behavior-conformance
Make Attaching an Empty Dict Behave the Same as Using it Directly
W. Felix Handte [Tue, 6 Aug 2019 23:08:41 +0000 (19:08 -0400)]
Only Bump Offset When Attaching Non-Null Dictionary
We do want to bump, even if the dictionary is empty, but we **don't** want to
bump if the dictionary is null.
W. Felix Handte [Tue, 6 Aug 2019 22:54:03 +0000 (18:54 -0400)]
Add Attach Dict Debug Log
W. Felix Handte [Tue, 6 Aug 2019 19:24:51 +0000 (15:24 -0400)]
Make Attaching an Empty Dict Behave the Same as Using it Directly
When using an empty dictionary, we bail out of loading or attaching it in
ways that leave the working context in potentially slightly different states.
In particular, in some paths, we will cause the currentOffset to be non-zero,
while in others we would allow it to remain 0.
This difference in behavior is perfectly harmless, but in some situations, it
can produce slight differences in the compressed output. For sanity's sake,
we currently try to maintain a strict correspondence between the behavior of
the dict attachment and the dict loading paths. This patch restores them to
behaving identically.
This shouldn't have any negative side-effects, as far as I can tell. When
writing the dict attachment code, I tried to preserve zeroed currentOffsets
when possible, since they benchmarked as very slightly faster. However, the
case of attaching an empty dictionary is probably rare enought that it's
acceptable to minisculely degrade performance in that corner case.
Yann Collet [Tue, 6 Aug 2019 17:17:16 +0000 (19:17 +0200)]
Merge pull request #772 from lz4/offset0
silence msan warning when offset==0
Yann Collet [Tue, 6 Aug 2019 12:46:31 +0000 (14:46 +0200)]
silence msan warning when offset==0