platform/upstream/lz4.git
3 years agofix #874
Yann Collet [Sun, 8 Nov 2020 21:21:58 +0000 (13:21 -0800)]
fix #874

coverity reported a warning regarding a memcpy() overwrite.
This is a false positive (the memory area is large enough),
but it's true that it's not trivial to determine (encompassing struct),
and it's proper anyway to only memcpy() just the right amount of data.

3 years agoupdate obsolete section documentation
Yann Collet [Sun, 8 Nov 2020 20:54:59 +0000 (12:54 -0800)]
update obsolete section documentation

and update manuals.
fix #712

3 years agoMerge pull request #942 from lz4/fix926
Yann Collet [Sun, 8 Nov 2020 05:05:27 +0000 (21:05 -0800)]
Merge pull request #942 from lz4/fix926

fix #926

3 years agofix minor UBs
Yann Collet [Sun, 8 Nov 2020 03:38:07 +0000 (19:38 -0800)]
fix minor UBs

- check alignment before casting a pointer
- saveDict : don't memmove() on NULL dst

3 years agoadded similar tests for LZ4_saveDict()
Yann Collet [Sun, 8 Nov 2020 02:24:02 +0000 (18:24 -0800)]
added similar tests for LZ4_saveDict()

fast (non-HC) variant

3 years agofix #926
Yann Collet [Sun, 8 Nov 2020 02:12:26 +0000 (18:12 -0800)]
fix #926

fix incorrect behavior of LZ4_saveDictHC()
when invoked right after initialization.

3 years agoMerge pull request #941 from lz4/revertinline
Yann Collet [Sun, 8 Nov 2020 01:18:04 +0000 (17:18 -0800)]
Merge pull request #941 from lz4/revertinline

Revert "Replace "static" to "LZ4_FORCE_INLINE" for small functions"

3 years agoMerge pull request #940 from lz4/fix935
Yann Collet [Sun, 8 Nov 2020 00:12:41 +0000 (16:12 -0800)]
Merge pull request #940 from lz4/fix935

fix #935

3 years agoRevert "Replace "static" to "LZ4_FORCE_INLINE" for small functions"
Yann Collet [Sat, 7 Nov 2020 19:02:30 +0000 (11:02 -0800)]
Revert "Replace "static" to "LZ4_FORCE_INLINE" for small functions"

This reverts commit 0e3933edd435c54cc2e21e38f5d4ba7bf644a24e.

3 years agofix #935
Yann Collet [Sat, 7 Nov 2020 18:06:52 +0000 (10:06 -0800)]
fix #935

minor: identical declaration and prototypes of `LZ4HC_compress_optimal()`

also :
very minor optimization of `LZ4_memcpy_using_offset()`

3 years agoMerge pull request #939 from lz4/fix927
Yann Collet [Sat, 7 Nov 2020 17:37:27 +0000 (09:37 -0800)]
Merge pull request #939 from lz4/fix927

LZ4F_decompress requires a valid dctx state

3 years agoLZ4F_decompress requires a valid dctx state
Yann Collet [Sat, 7 Nov 2020 04:46:35 +0000 (20:46 -0800)]
LZ4F_decompress requires a valid dctx state

This is now explicitly documented and asserted.
fix #927

3 years agoMerge pull request #936 from lz4/alignTest
Yann Collet [Sat, 7 Nov 2020 04:27:42 +0000 (20:27 -0800)]
Merge pull request #936 from lz4/alignTest

More alignment tests

3 years agostatic state size
Yann Collet [Sat, 7 Nov 2020 02:26:13 +0000 (18:26 -0800)]
static state size

for better inter-version compatibility

3 years agore-enable alignment test on all targets
Yann Collet [Sat, 7 Nov 2020 00:43:14 +0000 (16:43 -0800)]
re-enable alignment test on all targets

3 years agounified internal state declaration
Yann Collet [Sat, 7 Nov 2020 00:15:51 +0000 (16:15 -0800)]
unified internal state declaration

align on `void*` instead : there is no `long long` inside the structure

3 years agodocument LZ4_ALIGN_TEST
Yann Collet [Fri, 6 Nov 2020 22:48:43 +0000 (14:48 -0800)]
document LZ4_ALIGN_TEST

3 years agounified alignment test
Yann Collet [Fri, 6 Nov 2020 22:46:48 +0000 (14:46 -0800)]
unified alignment test

across lz4.c and lz4hc.c

3 years agoMerge pull request #938 from terrelln/fuzzer-fix
Yann Collet [Fri, 6 Nov 2020 18:34:05 +0000 (10:34 -0800)]
Merge pull request #938 from terrelln/fuzzer-fix

[ossfuzz] Fix parallel builds

3 years ago[ossfuzz] Fix parallel builds
Nick Terrell [Thu, 5 Nov 2020 18:27:43 +0000 (10:27 -0800)]
[ossfuzz] Fix parallel builds

With `make -j` multiple builds of `standaloneengine.o` happened in
parallel. Fix this by detecting `standaloneengine.o` and moving it to a
depedency.

3 years agofix minor explicit cast
Yann Collet [Sat, 31 Oct 2020 08:55:04 +0000 (01:55 -0700)]
fix minor explicit cast

3 years agoMerge pull request #930 from remittor-pr/fix_msvc
Yann Collet [Sat, 31 Oct 2020 08:44:21 +0000 (01:44 -0700)]
Merge pull request #930 from remittor-pr/fix_msvc

Fix: The "inline" specifier do not use for LZ4_wildCopy8 and LZ4_wildCopy32

3 years agoReplace "static" to "LZ4_FORCE_INLINE" for small functions
remittor [Wed, 7 Oct 2020 06:52:40 +0000 (09:52 +0300)]
Replace "static" to "LZ4_FORCE_INLINE" for small functions

The "static" specifier does not guarantee that the function will be inlined.

3 years agoReplace define LZ4_FORCE_O2_INLINE_GCC_PPC64LE to LZ4_FORCE_INLINE
remittor [Wed, 7 Oct 2020 06:51:08 +0000 (09:51 +0300)]
Replace define LZ4_FORCE_O2_INLINE_GCC_PPC64LE to LZ4_FORCE_INLINE

There is no reason to separate these two definitions!

3 years agoMerge pull request #928 from remittor-pr/human_redable
Yann Collet [Tue, 6 Oct 2020 17:49:42 +0000 (10:49 -0700)]
Merge pull request #928 from remittor-pr/human_redable

Made function LZ4HC_encodeSequence a human readable

3 years agoFix: The "inline" specifier do not use for LZ4_wildCopy8 and LZ4_wildCopy32
remittor [Tue, 6 Oct 2020 14:16:43 +0000 (17:16 +0300)]
Fix: The "inline" specifier do not use for LZ4_wildCopy8 and LZ4_wildCopy32

This problem was reproduced on MSVC 2015 (32-bit). Both functions were called using the operator "call".

3 years ago[lz4hc] Made function LZ4HC_encodeSequence a human readable
remittor [Sat, 3 Oct 2020 15:28:24 +0000 (18:28 +0300)]
[lz4hc] Made function LZ4HC_encodeSequence a human readable

3 years agoreduce new test stack usage
Yann Collet [Sat, 3 Oct 2020 01:02:41 +0000 (18:02 -0700)]
reduce new test stack usage

use heap instead

3 years agoAppveyor: added compilation and runtime fuzzer tests
Yann Collet [Fri, 2 Oct 2020 23:37:34 +0000 (16:37 -0700)]
Appveyor: added compilation and runtime fuzzer tests

to all Windows compiler targets

3 years agopreserver alignment test on Visual Studio x64
Yann Collet [Fri, 2 Oct 2020 22:51:14 +0000 (15:51 -0700)]
preserver alignment test on Visual Studio x64

this it works fine in this environment
(only x86 is suspicious)

3 years agofixed x32 test on Travis
Yann Collet [Fri, 2 Oct 2020 22:43:46 +0000 (15:43 -0700)]
fixed x32 test on Travis

3 years agofix cppcheck unused variable warning
Yann Collet [Fri, 2 Oct 2020 21:10:00 +0000 (14:10 -0700)]
fix cppcheck unused variable warning

3 years agoadded LZ4_streamHC_t init test
Yann Collet [Fri, 2 Oct 2020 20:06:51 +0000 (13:06 -0700)]
added LZ4_streamHC_t init test

which includes an alignment test

3 years agoMerge pull request #925 from lz4/test_null
Yann Collet [Fri, 2 Oct 2020 15:30:25 +0000 (08:30 -0700)]
Merge pull request #925 from lz4/test_null

add LZ4F_decompress() tests with (NULL,0) input and output

3 years agodefine sentinelTest
Yann Collet [Fri, 2 Oct 2020 06:42:42 +0000 (23:42 -0700)]
define sentinelTest

to help scan-build detect the condition

3 years agomake it possible to select SCANBUILD binary on command line
Yann Collet [Fri, 2 Oct 2020 06:27:02 +0000 (23:27 -0700)]
make it possible to select SCANBUILD binary on command line

3 years agoadd LZ4F_decompress() tests with (NULL,0) input and output
Yann Collet [Fri, 2 Oct 2020 02:07:56 +0000 (19:07 -0700)]
add LZ4F_decompress() tests with (NULL,0) input and output

fix one (rare & complex) issue discovered by this test

3 years agoMerge pull request #924 from lz4/safixes
Yann Collet [Thu, 1 Oct 2020 19:53:27 +0000 (12:53 -0700)]
Merge pull request #924 from lz4/safixes

fix minor static analyzer warnings

3 years agomake scan-build accept assert()
Yann Collet [Thu, 1 Oct 2020 17:48:22 +0000 (10:48 -0700)]
make scan-build accept assert()

3 years agofix bad init scenario
Yann Collet [Thu, 1 Oct 2020 05:51:17 +0000 (22:51 -0700)]
fix bad init scenario

3 years agoadded memcpy() related SA warning fixes
Yann Collet [Thu, 1 Oct 2020 00:39:42 +0000 (17:39 -0700)]
added memcpy() related SA warning fixes

memcpy() on NULL is UB, even if length is 0.

3 years agoMerge branch 'safixes' of github.com:lz4/lz4 into safixes
Yann Collet [Wed, 30 Sep 2020 07:23:47 +0000 (00:23 -0700)]
Merge branch 'safixes' of github.com:lz4/lz4 into safixes

3 years agofix conversion warning
Yann Collet [Wed, 30 Sep 2020 04:53:42 +0000 (21:53 -0700)]
fix conversion warning

3 years agoMerge branch 'dev' into safixes
Yann Collet [Wed, 30 Sep 2020 00:21:59 +0000 (17:21 -0700)]
Merge branch 'dev' into safixes

3 years agofix minor static analyzer warnings
Yann Collet [Tue, 29 Sep 2020 05:37:20 +0000 (22:37 -0700)]
fix minor static analyzer warnings

detected by scan-build, cppcheck and advanved compilation flags
fix #786

3 years agofix minor static analyzer warnings
Yann Collet [Tue, 29 Sep 2020 05:37:20 +0000 (22:37 -0700)]
fix minor static analyzer warnings

detected by scan-build and cppcheck
fix #786

3 years agobump version number
Yann Collet [Tue, 29 Sep 2020 21:27:39 +0000 (14:27 -0700)]
bump version number

to v1.9.3

3 years agoMerge pull request #923 from lz4/fix784
Yann Collet [Mon, 28 Sep 2020 21:04:56 +0000 (14:04 -0700)]
Merge pull request #923 from lz4/fix784

fix efficiency of LZ4_compress_HC_destSize()

3 years agoimproved last literals run on LZ4_compress_destSize
Yann Collet [Mon, 28 Sep 2020 18:39:00 +0000 (11:39 -0700)]
improved last literals run on LZ4_compress_destSize

applying new more accurate formula from LZ4_compress_HC_destSize()

also : fix some minor display issue in tests/frametest

3 years agoensure last match not too close to end
Yann Collet [Mon, 28 Sep 2020 06:59:56 +0000 (23:59 -0700)]
ensure last match not too close to end

must respect MFLIMIT distance from oend

3 years agofix incorrect counting
Yann Collet [Mon, 28 Sep 2020 05:47:05 +0000 (22:47 -0700)]
fix incorrect counting

after truncation of last sequence

3 years agoMerge pull request #921 from lz4/doubleNull
Yann Collet [Mon, 28 Sep 2020 04:09:06 +0000 (21:09 -0700)]
Merge pull request #921 from lz4/doubleNull

fix compressing into NULL

3 years agoMerge pull request #922 from XVilka/tinycc-support
Yann Collet [Mon, 28 Sep 2020 04:08:26 +0000 (21:08 -0700)]
Merge pull request #922 from XVilka/tinycc-support

Fix compilation with TinyCC

3 years agofix efficiency of LZ4_compress_HC_destSize()
Yann Collet [Mon, 28 Sep 2020 04:04:40 +0000 (21:04 -0700)]
fix efficiency of LZ4_compress_HC_destSize()

LZ4_compress_HC_destSize() had a tendency
to discard its last match when this match overflowed specified dstBuffer limit.
The impact is generally moderate,
but occasionally huge,
typically when this last match is very large
(such as compressing a bunch of zeroes).

Issue #784 fixed for both Chain and Opt implementations.

Added a unit test suggested by @remittor checking this topic.

3 years agoFix compilation with TinyCC
Anton Kochkov [Sun, 27 Sep 2020 09:07:51 +0000 (17:07 +0800)]
Fix compilation with TinyCC

3 years agofix compressing into NULL
Yann Collet [Sat, 26 Sep 2020 18:31:57 +0000 (11:31 -0700)]
fix compressing into NULL

fails properly
bug discovered by oss-fuzz

3 years agoMerge pull request #919 from attilaolah/cli
Yann Collet [Wed, 23 Sep 2020 15:23:59 +0000 (08:23 -0700)]
Merge pull request #919 from attilaolah/cli

Add CMake option LZ4_BUILD_CLI

3 years agoAdd CMake option LZ4_BUILD_CLI.
Attila Oláh [Wed, 23 Sep 2020 12:49:25 +0000 (12:49 +0000)]
Add CMake option LZ4_BUILD_CLI.

Defaults to ON so nothing changes, but it can be set to OFF so that the
CLI is not built at all, only the library.

3 years agocomment bug on older versions of ZSTD_compress_destSize()
Yann Collet [Fri, 18 Sep 2020 03:59:01 +0000 (20:59 -0700)]
comment bug on older versions of ZSTD_compress_destSize()

following investigation in #859

3 years agoMerge pull request #918 from lz4/usan_null
Yann Collet [Fri, 18 Sep 2020 03:23:06 +0000 (20:23 -0700)]
Merge pull request #918 from lz4/usan_null

Properly fix #847

3 years agofixed ubsan tests with dynamic library
Yann Collet [Thu, 17 Sep 2020 23:49:21 +0000 (16:49 -0700)]
fixed ubsan tests with dynamic library

3 years agoubsan: make pointer overflow recoverable
Yann Collet [Thu, 17 Sep 2020 22:34:03 +0000 (15:34 -0700)]
ubsan: make pointer overflow recoverable

the way `base` is used must allow 2-complement address overflow.
`base` is effectively a virtual pointer, never dereferenced directly.
What matters is `base + index`.
This is the address that will be dereference and must be valid (it is properly validated).

3 years agofixed lz4frame with blocks of size 1
Yann Collet [Thu, 17 Sep 2020 21:43:02 +0000 (14:43 -0700)]
fixed lz4frame with blocks of size 1

properly track history

3 years agoMerge branch 'usan_null' of github.com:Cyan4973/lz4 into usan_null
Yann Collet [Thu, 17 Sep 2020 17:38:59 +0000 (10:38 -0700)]
Merge branch 'usan_null' of github.com:Cyan4973/lz4 into usan_null

3 years agoMerge branch 'usan_null' of github.com:lz4/lz4 into usan_null
Yann Collet [Thu, 17 Sep 2020 17:32:05 +0000 (10:32 -0700)]
Merge branch 'usan_null' of github.com:lz4/lz4 into usan_null

3 years agofix ubsan test
Yann Collet [Thu, 17 Sep 2020 17:29:31 +0000 (10:29 -0700)]
fix ubsan test

which now fails with a clear error as intended
(not just print a status and move on).
should be reproduced on travisCI

3 years agoadded the actual code change
Yann Collet [Thu, 17 Sep 2020 06:46:39 +0000 (23:46 -0700)]
added the actual code change

3 years agoMerge pull request #913 from lz4/usan_null
Yann Collet [Thu, 17 Sep 2020 06:24:41 +0000 (23:24 -0700)]
Merge pull request #913 from lz4/usan_null

fix #847

3 years agofix #847
Yann Collet [Tue, 15 Sep 2020 21:54:53 +0000 (14:54 -0700)]
fix #847

support NULL input
without triggering undefined sanitizer

3 years agoadded test triggering NULL arithmetic with usan
Yann Collet [Tue, 15 Sep 2020 20:45:06 +0000 (13:45 -0700)]
added test triggering NULL arithmetic with usan

described in #847

4 years agoMerge pull request #910 from lz4/extraInput
Yann Collet [Thu, 27 Aug 2020 18:00:28 +0000 (11:00 -0700)]
Merge pull request #910 from lz4/extraInput

Fix issue #783

4 years agofixed minor pre-existing printf formatting
Yann Collet [Thu, 27 Aug 2020 16:20:01 +0000 (09:20 -0700)]
fixed minor pre-existing printf formatting

different signedness

4 years agofixed strict c90 support
Yann Collet [Thu, 27 Aug 2020 07:38:58 +0000 (00:38 -0700)]
fixed strict c90 support

4 years agofix #783
Yann Collet [Thu, 27 Aug 2020 07:17:57 +0000 (00:17 -0700)]
fix #783

LZ4_decompress_safe_partial()
now also supports a scenario where
nb_bytes_to_generate is <= block_decompressed_size
And
nb_bytes_to_read is >= block_compressed_size.

Previously, the only supported scenario was
nb_bytes_to_read == block_compress_size.

Pay attention that,
if nb_bytes_to_read is > block_compressed_size,
then, necessarily, it requires that
nb_bytes_to_generate is <= block_decompress_size.
If both are larger, it will generate corrupted data.

4 years agoMerge branch 'dev' into extraInput
Yann Collet [Thu, 27 Aug 2020 06:20:28 +0000 (23:20 -0700)]
Merge branch 'dev' into extraInput

4 years agoMerge pull request #909 from lz4/fasterCount
Yann Collet [Wed, 26 Aug 2020 15:13:52 +0000 (08:13 -0700)]
Merge pull request #909 from lz4/fasterCount

Improved software byte count

4 years agoMerge branch 'dev' into fasterCount
Yann Collet [Wed, 26 Aug 2020 05:18:54 +0000 (22:18 -0700)]
Merge branch 'dev' into fasterCount

4 years agoadded documentation about LZ4_FORCE_SW_BITCOUNT
Yann Collet [Wed, 26 Aug 2020 05:17:29 +0000 (22:17 -0700)]
added documentation about LZ4_FORCE_SW_BITCOUNT

Also : added memory-frugal software byte count for big endian 64-bit cpus.
Disabled by default.

4 years agoMerge pull request #908 from lz4/stdin_size
Yann Collet [Wed, 26 Aug 2020 04:15:31 +0000 (21:15 -0700)]
Merge pull request #908 from lz4/stdin_size

Added ability to determine content-size

4 years agofix test under non-interactive environment
Yann Collet [Tue, 25 Aug 2020 21:53:08 +0000 (14:53 -0700)]
fix test under non-interactive environment

4 years agoadded more tests around --content-size
Yann Collet [Tue, 25 Aug 2020 21:42:15 +0000 (14:42 -0700)]
added more tests around --content-size

notably in association with `stdin`

4 years agouse variables for program invocation
Yann Collet [Tue, 25 Aug 2020 21:25:51 +0000 (14:25 -0700)]
use variables for program invocation

4 years agoMerge pull request #905 from nh2/use-fstat
Yann Collet [Tue, 25 Aug 2020 21:19:40 +0000 (14:19 -0700)]
Merge pull request #905 from nh2/use-fstat

Use fstat() to determine file size

4 years agofixup: Portable fileno()
Niklas Hambüchen [Tue, 25 Aug 2020 01:37:41 +0000 (03:37 +0200)]
fixup: Portable fileno()

4 years agofixup: C90 compliance, clean up stat() types/calls
Niklas Hambüchen [Tue, 25 Aug 2020 01:05:01 +0000 (03:05 +0200)]
fixup: C90 compliance, clean up stat() types/calls

4 years agoMerge pull request #898 from aqrit/aqrit-prefixlen
Yann Collet [Mon, 24 Aug 2020 22:13:18 +0000 (15:13 -0700)]
Merge pull request #898 from aqrit/aqrit-prefixlen

rejigger bit counting intrinsics

4 years agoMerge pull request #902 from lz4/realhw
Yann Collet [Mon, 24 Aug 2020 22:07:18 +0000 (15:07 -0700)]
Merge pull request #902 from lz4/realhw

added target aarch64, ppc64lw and s390x to TravisCI

4 years agoMerge pull request #907 from terrelln/travis
Yann Collet [Mon, 24 Aug 2020 19:20:10 +0000 (12:20 -0700)]
Merge pull request #907 from terrelln/travis

[CI][ossfuzz] Fix string equal

4 years agoMerge pull request #903 from lz4/nofastforce
Yann Collet [Mon, 24 Aug 2020 18:28:47 +0000 (11:28 -0700)]
Merge pull request #903 from lz4/nofastforce

removed LZ4_compress_fast_force()

4 years ago[CI][ossfuzz] Fix string equal
Nick Terrell [Mon, 24 Aug 2020 18:22:57 +0000 (11:22 -0700)]
[CI][ossfuzz] Fix string equal

4 years agoUse fstat() to determine file size.
Niklas Hambüchen [Mon, 24 Aug 2020 04:23:51 +0000 (06:23 +0200)]
Use fstat() to determine file size.

This allows us to get the file size even when the input file is passed
via stdin. This fixes `--content-size` not working in situations like

    $ lz4 -v --content-size < /tmp/test > /tmp/test.lz4
    Warning : cannot determine input content size

With this change, it works.

Also helps with #904.

4 years agoremoved LZ4_compress_fast_force()
Yann Collet [Sat, 22 Aug 2020 02:23:49 +0000 (19:23 -0700)]
removed LZ4_compress_fast_force()

which serves no more purpose.

The comment implies that the simple presence of this unused function was affecting performance,
and that's the reason why it was not removed earlier.
This is likely another side effect of instruction alignment.

It's obviously unreliable to rely on it in this way,
meaning that the impact will be different, positive of negative,
with any minor code change, and any minor compiler version change, even parameter change.

4 years agoadded target aarch64, ppc64lw and s390x to TravisCI
Yann Collet [Sat, 22 Aug 2020 01:12:13 +0000 (18:12 -0700)]
added target aarch64, ppc64lw and s390x to TravisCI

more portability tests

4 years agoMerge pull request #886 from servusDei2018/dev
Yann Collet [Fri, 21 Aug 2020 21:17:43 +0000 (14:17 -0700)]
Merge pull request #886 from servusDei2018/dev

Optimized by replacing `endl` with `'\n'`

4 years agoMerge pull request #900 from terrelln/cmake-build
Yann Collet [Thu, 20 Aug 2020 03:08:01 +0000 (20:08 -0700)]
Merge pull request #900 from terrelln/cmake-build

[build] Move CMake and Visual build systems to build/

4 years agoMerge pull request #901 from terrelln/travis
Yann Collet [Thu, 20 Aug 2020 03:07:50 +0000 (20:07 -0700)]
Merge pull request #901 from terrelln/travis

[CI][Fuzz] Fix Travis-CI fuzzer tests

4 years ago[CI][Fuzz] Fix Travis-CI fuzzer tests
Nick Terrell [Thu, 20 Aug 2020 01:19:11 +0000 (18:19 -0700)]
[CI][Fuzz] Fix Travis-CI fuzzer tests

Fixes #781

4 years ago[build] Move CMake and Visual build systems to build/
Nick Terrell [Thu, 20 Aug 2020 01:06:49 +0000 (18:06 -0700)]
[build] Move CMake and Visual build systems to build/

Fixes #852.

4 years agoMerge branch 'dev' into extraInput
Yann Collet [Tue, 18 Aug 2020 22:34:51 +0000 (15:34 -0700)]
Merge branch 'dev' into extraInput

4 years agoMerge pull request #897 from lz4/lz4wlib
Yann Collet [Tue, 18 Aug 2020 17:24:36 +0000 (10:24 -0700)]
Merge pull request #897 from lz4/lz4wlib

added target lz4-wlib