Yann Collet [Sun, 8 Nov 2020 21:21:58 +0000 (13:21 -0800)]
fix #874
coverity reported a warning regarding a memcpy() overwrite.
This is a false positive (the memory area is large enough),
but it's true that it's not trivial to determine (encompassing struct),
and it's proper anyway to only memcpy() just the right amount of data.
Yann Collet [Sun, 8 Nov 2020 20:54:59 +0000 (12:54 -0800)]
update obsolete section documentation
and update manuals.
fix #712
Yann Collet [Sun, 8 Nov 2020 05:05:27 +0000 (21:05 -0800)]
Merge pull request #942 from lz4/fix926
fix #926
Yann Collet [Sun, 8 Nov 2020 03:38:07 +0000 (19:38 -0800)]
fix minor UBs
- check alignment before casting a pointer
- saveDict : don't memmove() on NULL dst
Yann Collet [Sun, 8 Nov 2020 02:24:02 +0000 (18:24 -0800)]
added similar tests for LZ4_saveDict()
fast (non-HC) variant
Yann Collet [Sun, 8 Nov 2020 02:12:26 +0000 (18:12 -0800)]
fix #926
fix incorrect behavior of LZ4_saveDictHC()
when invoked right after initialization.
Yann Collet [Sun, 8 Nov 2020 01:18:04 +0000 (17:18 -0800)]
Merge pull request #941 from lz4/revertinline
Revert "Replace "static" to "LZ4_FORCE_INLINE" for small functions"
Yann Collet [Sun, 8 Nov 2020 00:12:41 +0000 (16:12 -0800)]
Merge pull request #940 from lz4/fix935
fix #935
Yann Collet [Sat, 7 Nov 2020 19:02:30 +0000 (11:02 -0800)]
Revert "Replace "static" to "LZ4_FORCE_INLINE" for small functions"
This reverts commit
0e3933edd435c54cc2e21e38f5d4ba7bf644a24e.
Yann Collet [Sat, 7 Nov 2020 18:06:52 +0000 (10:06 -0800)]
fix #935
minor: identical declaration and prototypes of `LZ4HC_compress_optimal()`
also :
very minor optimization of `LZ4_memcpy_using_offset()`
Yann Collet [Sat, 7 Nov 2020 17:37:27 +0000 (09:37 -0800)]
Merge pull request #939 from lz4/fix927
LZ4F_decompress requires a valid dctx state
Yann Collet [Sat, 7 Nov 2020 04:46:35 +0000 (20:46 -0800)]
LZ4F_decompress requires a valid dctx state
This is now explicitly documented and asserted.
fix #927
Yann Collet [Sat, 7 Nov 2020 04:27:42 +0000 (20:27 -0800)]
Merge pull request #936 from lz4/alignTest
More alignment tests
Yann Collet [Sat, 7 Nov 2020 02:26:13 +0000 (18:26 -0800)]
static state size
for better inter-version compatibility
Yann Collet [Sat, 7 Nov 2020 00:43:14 +0000 (16:43 -0800)]
re-enable alignment test on all targets
Yann Collet [Sat, 7 Nov 2020 00:15:51 +0000 (16:15 -0800)]
unified internal state declaration
align on `void*` instead : there is no `long long` inside the structure
Yann Collet [Fri, 6 Nov 2020 22:48:43 +0000 (14:48 -0800)]
document LZ4_ALIGN_TEST
Yann Collet [Fri, 6 Nov 2020 22:46:48 +0000 (14:46 -0800)]
unified alignment test
across lz4.c and lz4hc.c
Yann Collet [Fri, 6 Nov 2020 18:34:05 +0000 (10:34 -0800)]
Merge pull request #938 from terrelln/fuzzer-fix
[ossfuzz] Fix parallel builds
Nick Terrell [Thu, 5 Nov 2020 18:27:43 +0000 (10:27 -0800)]
[ossfuzz] Fix parallel builds
With `make -j` multiple builds of `standaloneengine.o` happened in
parallel. Fix this by detecting `standaloneengine.o` and moving it to a
depedency.
Yann Collet [Sat, 31 Oct 2020 08:55:04 +0000 (01:55 -0700)]
fix minor explicit cast
Yann Collet [Sat, 31 Oct 2020 08:44:21 +0000 (01:44 -0700)]
Merge pull request #930 from remittor-pr/fix_msvc
Fix: The "inline" specifier do not use for LZ4_wildCopy8 and LZ4_wildCopy32
remittor [Wed, 7 Oct 2020 06:52:40 +0000 (09:52 +0300)]
Replace "static" to "LZ4_FORCE_INLINE" for small functions
The "static" specifier does not guarantee that the function will be inlined.
remittor [Wed, 7 Oct 2020 06:51:08 +0000 (09:51 +0300)]
Replace define LZ4_FORCE_O2_INLINE_GCC_PPC64LE to LZ4_FORCE_INLINE
There is no reason to separate these two definitions!
Yann Collet [Tue, 6 Oct 2020 17:49:42 +0000 (10:49 -0700)]
Merge pull request #928 from remittor-pr/human_redable
Made function LZ4HC_encodeSequence a human readable
remittor [Tue, 6 Oct 2020 14:16:43 +0000 (17:16 +0300)]
Fix: The "inline" specifier do not use for LZ4_wildCopy8 and LZ4_wildCopy32
This problem was reproduced on MSVC 2015 (32-bit). Both functions were called using the operator "call".
remittor [Sat, 3 Oct 2020 15:28:24 +0000 (18:28 +0300)]
[lz4hc] Made function LZ4HC_encodeSequence a human readable
Yann Collet [Sat, 3 Oct 2020 01:02:41 +0000 (18:02 -0700)]
reduce new test stack usage
use heap instead
Yann Collet [Fri, 2 Oct 2020 23:37:34 +0000 (16:37 -0700)]
Appveyor: added compilation and runtime fuzzer tests
to all Windows compiler targets
Yann Collet [Fri, 2 Oct 2020 22:51:14 +0000 (15:51 -0700)]
preserver alignment test on Visual Studio x64
this it works fine in this environment
(only x86 is suspicious)
Yann Collet [Fri, 2 Oct 2020 22:43:46 +0000 (15:43 -0700)]
fixed x32 test on Travis
Yann Collet [Fri, 2 Oct 2020 21:10:00 +0000 (14:10 -0700)]
fix cppcheck unused variable warning
Yann Collet [Fri, 2 Oct 2020 20:06:51 +0000 (13:06 -0700)]
added LZ4_streamHC_t init test
which includes an alignment test
Yann Collet [Fri, 2 Oct 2020 15:30:25 +0000 (08:30 -0700)]
Merge pull request #925 from lz4/test_null
add LZ4F_decompress() tests with (NULL,0) input and output
Yann Collet [Fri, 2 Oct 2020 06:42:42 +0000 (23:42 -0700)]
define sentinelTest
to help scan-build detect the condition
Yann Collet [Fri, 2 Oct 2020 06:27:02 +0000 (23:27 -0700)]
make it possible to select SCANBUILD binary on command line
Yann Collet [Fri, 2 Oct 2020 02:07:56 +0000 (19:07 -0700)]
add LZ4F_decompress() tests with (NULL,0) input and output
fix one (rare & complex) issue discovered by this test
Yann Collet [Thu, 1 Oct 2020 19:53:27 +0000 (12:53 -0700)]
Merge pull request #924 from lz4/safixes
fix minor static analyzer warnings
Yann Collet [Thu, 1 Oct 2020 17:48:22 +0000 (10:48 -0700)]
make scan-build accept assert()
Yann Collet [Thu, 1 Oct 2020 05:51:17 +0000 (22:51 -0700)]
fix bad init scenario
Yann Collet [Thu, 1 Oct 2020 00:39:42 +0000 (17:39 -0700)]
added memcpy() related SA warning fixes
memcpy() on NULL is UB, even if length is 0.
Yann Collet [Wed, 30 Sep 2020 07:23:47 +0000 (00:23 -0700)]
Merge branch 'safixes' of github.com:lz4/lz4 into safixes
Yann Collet [Wed, 30 Sep 2020 04:53:42 +0000 (21:53 -0700)]
fix conversion warning
Yann Collet [Wed, 30 Sep 2020 00:21:59 +0000 (17:21 -0700)]
Merge branch 'dev' into safixes
Yann Collet [Tue, 29 Sep 2020 05:37:20 +0000 (22:37 -0700)]
fix minor static analyzer warnings
detected by scan-build, cppcheck and advanved compilation flags
fix #786
Yann Collet [Tue, 29 Sep 2020 05:37:20 +0000 (22:37 -0700)]
fix minor static analyzer warnings
detected by scan-build and cppcheck
fix #786
Yann Collet [Tue, 29 Sep 2020 21:27:39 +0000 (14:27 -0700)]
bump version number
to v1.9.3
Yann Collet [Mon, 28 Sep 2020 21:04:56 +0000 (14:04 -0700)]
Merge pull request #923 from lz4/fix784
fix efficiency of LZ4_compress_HC_destSize()
Yann Collet [Mon, 28 Sep 2020 18:39:00 +0000 (11:39 -0700)]
improved last literals run on LZ4_compress_destSize
applying new more accurate formula from LZ4_compress_HC_destSize()
also : fix some minor display issue in tests/frametest
Yann Collet [Mon, 28 Sep 2020 06:59:56 +0000 (23:59 -0700)]
ensure last match not too close to end
must respect MFLIMIT distance from oend
Yann Collet [Mon, 28 Sep 2020 05:47:05 +0000 (22:47 -0700)]
fix incorrect counting
after truncation of last sequence
Yann Collet [Mon, 28 Sep 2020 04:09:06 +0000 (21:09 -0700)]
Merge pull request #921 from lz4/doubleNull
fix compressing into NULL
Yann Collet [Mon, 28 Sep 2020 04:08:26 +0000 (21:08 -0700)]
Merge pull request #922 from XVilka/tinycc-support
Fix compilation with TinyCC
Yann Collet [Mon, 28 Sep 2020 04:04:40 +0000 (21:04 -0700)]
fix efficiency of LZ4_compress_HC_destSize()
LZ4_compress_HC_destSize() had a tendency
to discard its last match when this match overflowed specified dstBuffer limit.
The impact is generally moderate,
but occasionally huge,
typically when this last match is very large
(such as compressing a bunch of zeroes).
Issue #784 fixed for both Chain and Opt implementations.
Added a unit test suggested by @remittor checking this topic.
Anton Kochkov [Sun, 27 Sep 2020 09:07:51 +0000 (17:07 +0800)]
Fix compilation with TinyCC
Yann Collet [Sat, 26 Sep 2020 18:31:57 +0000 (11:31 -0700)]
fix compressing into NULL
fails properly
bug discovered by oss-fuzz
Yann Collet [Wed, 23 Sep 2020 15:23:59 +0000 (08:23 -0700)]
Merge pull request #919 from attilaolah/cli
Add CMake option LZ4_BUILD_CLI
Attila Oláh [Wed, 23 Sep 2020 12:49:25 +0000 (12:49 +0000)]
Add CMake option LZ4_BUILD_CLI.
Defaults to ON so nothing changes, but it can be set to OFF so that the
CLI is not built at all, only the library.
Yann Collet [Fri, 18 Sep 2020 03:59:01 +0000 (20:59 -0700)]
comment bug on older versions of ZSTD_compress_destSize()
following investigation in #859
Yann Collet [Fri, 18 Sep 2020 03:23:06 +0000 (20:23 -0700)]
Merge pull request #918 from lz4/usan_null
Properly fix #847
Yann Collet [Thu, 17 Sep 2020 23:49:21 +0000 (16:49 -0700)]
fixed ubsan tests with dynamic library
Yann Collet [Thu, 17 Sep 2020 22:34:03 +0000 (15:34 -0700)]
ubsan: make pointer overflow recoverable
the way `base` is used must allow 2-complement address overflow.
`base` is effectively a virtual pointer, never dereferenced directly.
What matters is `base + index`.
This is the address that will be dereference and must be valid (it is properly validated).
Yann Collet [Thu, 17 Sep 2020 21:43:02 +0000 (14:43 -0700)]
fixed lz4frame with blocks of size 1
properly track history
Yann Collet [Thu, 17 Sep 2020 17:38:59 +0000 (10:38 -0700)]
Merge branch 'usan_null' of github.com:Cyan4973/lz4 into usan_null
Yann Collet [Thu, 17 Sep 2020 17:32:05 +0000 (10:32 -0700)]
Merge branch 'usan_null' of github.com:lz4/lz4 into usan_null
Yann Collet [Thu, 17 Sep 2020 17:29:31 +0000 (10:29 -0700)]
fix ubsan test
which now fails with a clear error as intended
(not just print a status and move on).
should be reproduced on travisCI
Yann Collet [Thu, 17 Sep 2020 06:46:39 +0000 (23:46 -0700)]
added the actual code change
Yann Collet [Thu, 17 Sep 2020 06:24:41 +0000 (23:24 -0700)]
Merge pull request #913 from lz4/usan_null
fix #847
Yann Collet [Tue, 15 Sep 2020 21:54:53 +0000 (14:54 -0700)]
fix #847
support NULL input
without triggering undefined sanitizer
Yann Collet [Tue, 15 Sep 2020 20:45:06 +0000 (13:45 -0700)]
added test triggering NULL arithmetic with usan
described in #847
Yann Collet [Thu, 27 Aug 2020 18:00:28 +0000 (11:00 -0700)]
Merge pull request #910 from lz4/extraInput
Fix issue #783
Yann Collet [Thu, 27 Aug 2020 16:20:01 +0000 (09:20 -0700)]
fixed minor pre-existing printf formatting
different signedness
Yann Collet [Thu, 27 Aug 2020 07:38:58 +0000 (00:38 -0700)]
fixed strict c90 support
Yann Collet [Thu, 27 Aug 2020 07:17:57 +0000 (00:17 -0700)]
fix #783
LZ4_decompress_safe_partial()
now also supports a scenario where
nb_bytes_to_generate is <= block_decompressed_size
And
nb_bytes_to_read is >= block_compressed_size.
Previously, the only supported scenario was
nb_bytes_to_read == block_compress_size.
Pay attention that,
if nb_bytes_to_read is > block_compressed_size,
then, necessarily, it requires that
nb_bytes_to_generate is <= block_decompress_size.
If both are larger, it will generate corrupted data.
Yann Collet [Thu, 27 Aug 2020 06:20:28 +0000 (23:20 -0700)]
Merge branch 'dev' into extraInput
Yann Collet [Wed, 26 Aug 2020 15:13:52 +0000 (08:13 -0700)]
Merge pull request #909 from lz4/fasterCount
Improved software byte count
Yann Collet [Wed, 26 Aug 2020 05:18:54 +0000 (22:18 -0700)]
Merge branch 'dev' into fasterCount
Yann Collet [Wed, 26 Aug 2020 05:17:29 +0000 (22:17 -0700)]
added documentation about LZ4_FORCE_SW_BITCOUNT
Also : added memory-frugal software byte count for big endian 64-bit cpus.
Disabled by default.
Yann Collet [Wed, 26 Aug 2020 04:15:31 +0000 (21:15 -0700)]
Merge pull request #908 from lz4/stdin_size
Added ability to determine content-size
Yann Collet [Tue, 25 Aug 2020 21:53:08 +0000 (14:53 -0700)]
fix test under non-interactive environment
Yann Collet [Tue, 25 Aug 2020 21:42:15 +0000 (14:42 -0700)]
added more tests around --content-size
notably in association with `stdin`
Yann Collet [Tue, 25 Aug 2020 21:25:51 +0000 (14:25 -0700)]
use variables for program invocation
Yann Collet [Tue, 25 Aug 2020 21:19:40 +0000 (14:19 -0700)]
Merge pull request #905 from nh2/use-fstat
Use fstat() to determine file size
Niklas Hambüchen [Tue, 25 Aug 2020 01:37:41 +0000 (03:37 +0200)]
fixup: Portable fileno()
Niklas Hambüchen [Tue, 25 Aug 2020 01:05:01 +0000 (03:05 +0200)]
fixup: C90 compliance, clean up stat() types/calls
Yann Collet [Mon, 24 Aug 2020 22:13:18 +0000 (15:13 -0700)]
Merge pull request #898 from aqrit/aqrit-prefixlen
rejigger bit counting intrinsics
Yann Collet [Mon, 24 Aug 2020 22:07:18 +0000 (15:07 -0700)]
Merge pull request #902 from lz4/realhw
added target aarch64, ppc64lw and s390x to TravisCI
Yann Collet [Mon, 24 Aug 2020 19:20:10 +0000 (12:20 -0700)]
Merge pull request #907 from terrelln/travis
[CI][ossfuzz] Fix string equal
Yann Collet [Mon, 24 Aug 2020 18:28:47 +0000 (11:28 -0700)]
Merge pull request #903 from lz4/nofastforce
removed LZ4_compress_fast_force()
Nick Terrell [Mon, 24 Aug 2020 18:22:57 +0000 (11:22 -0700)]
[CI][ossfuzz] Fix string equal
Niklas Hambüchen [Mon, 24 Aug 2020 04:23:51 +0000 (06:23 +0200)]
Use fstat() to determine file size.
This allows us to get the file size even when the input file is passed
via stdin. This fixes `--content-size` not working in situations like
$ lz4 -v --content-size < /tmp/test > /tmp/test.lz4
Warning : cannot determine input content size
With this change, it works.
Also helps with #904.
Yann Collet [Sat, 22 Aug 2020 02:23:49 +0000 (19:23 -0700)]
removed LZ4_compress_fast_force()
which serves no more purpose.
The comment implies that the simple presence of this unused function was affecting performance,
and that's the reason why it was not removed earlier.
This is likely another side effect of instruction alignment.
It's obviously unreliable to rely on it in this way,
meaning that the impact will be different, positive of negative,
with any minor code change, and any minor compiler version change, even parameter change.
Yann Collet [Sat, 22 Aug 2020 01:12:13 +0000 (18:12 -0700)]
added target aarch64, ppc64lw and s390x to TravisCI
more portability tests
Yann Collet [Fri, 21 Aug 2020 21:17:43 +0000 (14:17 -0700)]
Merge pull request #886 from servusDei2018/dev
Optimized by replacing `endl` with `'\n'`
Yann Collet [Thu, 20 Aug 2020 03:08:01 +0000 (20:08 -0700)]
Merge pull request #900 from terrelln/cmake-build
[build] Move CMake and Visual build systems to build/
Yann Collet [Thu, 20 Aug 2020 03:07:50 +0000 (20:07 -0700)]
Merge pull request #901 from terrelln/travis
[CI][Fuzz] Fix Travis-CI fuzzer tests
Nick Terrell [Thu, 20 Aug 2020 01:19:11 +0000 (18:19 -0700)]
[CI][Fuzz] Fix Travis-CI fuzzer tests
Fixes #781
Nick Terrell [Thu, 20 Aug 2020 01:06:49 +0000 (18:06 -0700)]
[build] Move CMake and Visual build systems to build/
Fixes #852.
Yann Collet [Tue, 18 Aug 2020 22:34:51 +0000 (15:34 -0700)]
Merge branch 'dev' into extraInput
Yann Collet [Tue, 18 Aug 2020 17:24:36 +0000 (10:24 -0700)]
Merge pull request #897 from lz4/lz4wlib
added target lz4-wlib