Yann Collet [Tue, 29 Sep 2020 05:37:20 +0000 (22:37 -0700)]
fix minor static analyzer warnings
detected by scan-build and cppcheck
fix #786
Yann Collet [Mon, 28 Sep 2020 21:04:56 +0000 (14:04 -0700)]
Merge pull request #923 from lz4/fix784
fix efficiency of LZ4_compress_HC_destSize()
Yann Collet [Mon, 28 Sep 2020 18:39:00 +0000 (11:39 -0700)]
improved last literals run on LZ4_compress_destSize
applying new more accurate formula from LZ4_compress_HC_destSize()
also : fix some minor display issue in tests/frametest
Yann Collet [Mon, 28 Sep 2020 06:59:56 +0000 (23:59 -0700)]
ensure last match not too close to end
must respect MFLIMIT distance from oend
Yann Collet [Mon, 28 Sep 2020 05:47:05 +0000 (22:47 -0700)]
fix incorrect counting
after truncation of last sequence
Yann Collet [Mon, 28 Sep 2020 04:09:06 +0000 (21:09 -0700)]
Merge pull request #921 from lz4/doubleNull
fix compressing into NULL
Yann Collet [Mon, 28 Sep 2020 04:08:26 +0000 (21:08 -0700)]
Merge pull request #922 from XVilka/tinycc-support
Fix compilation with TinyCC
Yann Collet [Mon, 28 Sep 2020 04:04:40 +0000 (21:04 -0700)]
fix efficiency of LZ4_compress_HC_destSize()
LZ4_compress_HC_destSize() had a tendency
to discard its last match when this match overflowed specified dstBuffer limit.
The impact is generally moderate,
but occasionally huge,
typically when this last match is very large
(such as compressing a bunch of zeroes).
Issue #784 fixed for both Chain and Opt implementations.
Added a unit test suggested by @remittor checking this topic.
Anton Kochkov [Sun, 27 Sep 2020 09:07:51 +0000 (17:07 +0800)]
Fix compilation with TinyCC
Yann Collet [Sat, 26 Sep 2020 18:31:57 +0000 (11:31 -0700)]
fix compressing into NULL
fails properly
bug discovered by oss-fuzz
Yann Collet [Wed, 23 Sep 2020 15:23:59 +0000 (08:23 -0700)]
Merge pull request #919 from attilaolah/cli
Add CMake option LZ4_BUILD_CLI
Attila Oláh [Wed, 23 Sep 2020 12:49:25 +0000 (12:49 +0000)]
Add CMake option LZ4_BUILD_CLI.
Defaults to ON so nothing changes, but it can be set to OFF so that the
CLI is not built at all, only the library.
Yann Collet [Fri, 18 Sep 2020 03:59:01 +0000 (20:59 -0700)]
comment bug on older versions of ZSTD_compress_destSize()
following investigation in #859
Yann Collet [Fri, 18 Sep 2020 03:23:06 +0000 (20:23 -0700)]
Merge pull request #918 from lz4/usan_null
Properly fix #847
Yann Collet [Thu, 17 Sep 2020 23:49:21 +0000 (16:49 -0700)]
fixed ubsan tests with dynamic library
Yann Collet [Thu, 17 Sep 2020 22:34:03 +0000 (15:34 -0700)]
ubsan: make pointer overflow recoverable
the way `base` is used must allow 2-complement address overflow.
`base` is effectively a virtual pointer, never dereferenced directly.
What matters is `base + index`.
This is the address that will be dereference and must be valid (it is properly validated).
Yann Collet [Thu, 17 Sep 2020 21:43:02 +0000 (14:43 -0700)]
fixed lz4frame with blocks of size 1
properly track history
Yann Collet [Thu, 17 Sep 2020 17:38:59 +0000 (10:38 -0700)]
Merge branch 'usan_null' of github.com:Cyan4973/lz4 into usan_null
Yann Collet [Thu, 17 Sep 2020 17:32:05 +0000 (10:32 -0700)]
Merge branch 'usan_null' of github.com:lz4/lz4 into usan_null
Yann Collet [Thu, 17 Sep 2020 17:29:31 +0000 (10:29 -0700)]
fix ubsan test
which now fails with a clear error as intended
(not just print a status and move on).
should be reproduced on travisCI
Yann Collet [Thu, 17 Sep 2020 06:46:39 +0000 (23:46 -0700)]
added the actual code change
Yann Collet [Thu, 17 Sep 2020 06:24:41 +0000 (23:24 -0700)]
Merge pull request #913 from lz4/usan_null
fix #847
Yann Collet [Tue, 15 Sep 2020 21:54:53 +0000 (14:54 -0700)]
fix #847
support NULL input
without triggering undefined sanitizer
Yann Collet [Tue, 15 Sep 2020 20:45:06 +0000 (13:45 -0700)]
added test triggering NULL arithmetic with usan
described in #847
Yann Collet [Thu, 27 Aug 2020 18:00:28 +0000 (11:00 -0700)]
Merge pull request #910 from lz4/extraInput
Fix issue #783
Yann Collet [Thu, 27 Aug 2020 16:20:01 +0000 (09:20 -0700)]
fixed minor pre-existing printf formatting
different signedness
Yann Collet [Thu, 27 Aug 2020 07:38:58 +0000 (00:38 -0700)]
fixed strict c90 support
Yann Collet [Thu, 27 Aug 2020 07:17:57 +0000 (00:17 -0700)]
fix #783
LZ4_decompress_safe_partial()
now also supports a scenario where
nb_bytes_to_generate is <= block_decompressed_size
And
nb_bytes_to_read is >= block_compressed_size.
Previously, the only supported scenario was
nb_bytes_to_read == block_compress_size.
Pay attention that,
if nb_bytes_to_read is > block_compressed_size,
then, necessarily, it requires that
nb_bytes_to_generate is <= block_decompress_size.
If both are larger, it will generate corrupted data.
Yann Collet [Thu, 27 Aug 2020 06:20:28 +0000 (23:20 -0700)]
Merge branch 'dev' into extraInput
Yann Collet [Wed, 26 Aug 2020 15:13:52 +0000 (08:13 -0700)]
Merge pull request #909 from lz4/fasterCount
Improved software byte count
Yann Collet [Wed, 26 Aug 2020 05:18:54 +0000 (22:18 -0700)]
Merge branch 'dev' into fasterCount
Yann Collet [Wed, 26 Aug 2020 05:17:29 +0000 (22:17 -0700)]
added documentation about LZ4_FORCE_SW_BITCOUNT
Also : added memory-frugal software byte count for big endian 64-bit cpus.
Disabled by default.
Yann Collet [Wed, 26 Aug 2020 04:15:31 +0000 (21:15 -0700)]
Merge pull request #908 from lz4/stdin_size
Added ability to determine content-size
Yann Collet [Tue, 25 Aug 2020 21:53:08 +0000 (14:53 -0700)]
fix test under non-interactive environment
Yann Collet [Tue, 25 Aug 2020 21:42:15 +0000 (14:42 -0700)]
added more tests around --content-size
notably in association with `stdin`
Yann Collet [Tue, 25 Aug 2020 21:25:51 +0000 (14:25 -0700)]
use variables for program invocation
Yann Collet [Tue, 25 Aug 2020 21:19:40 +0000 (14:19 -0700)]
Merge pull request #905 from nh2/use-fstat
Use fstat() to determine file size
Niklas Hambüchen [Tue, 25 Aug 2020 01:37:41 +0000 (03:37 +0200)]
fixup: Portable fileno()
Niklas Hambüchen [Tue, 25 Aug 2020 01:05:01 +0000 (03:05 +0200)]
fixup: C90 compliance, clean up stat() types/calls
Yann Collet [Mon, 24 Aug 2020 22:13:18 +0000 (15:13 -0700)]
Merge pull request #898 from aqrit/aqrit-prefixlen
rejigger bit counting intrinsics
Yann Collet [Mon, 24 Aug 2020 22:07:18 +0000 (15:07 -0700)]
Merge pull request #902 from lz4/realhw
added target aarch64, ppc64lw and s390x to TravisCI
Yann Collet [Mon, 24 Aug 2020 19:20:10 +0000 (12:20 -0700)]
Merge pull request #907 from terrelln/travis
[CI][ossfuzz] Fix string equal
Yann Collet [Mon, 24 Aug 2020 18:28:47 +0000 (11:28 -0700)]
Merge pull request #903 from lz4/nofastforce
removed LZ4_compress_fast_force()
Nick Terrell [Mon, 24 Aug 2020 18:22:57 +0000 (11:22 -0700)]
[CI][ossfuzz] Fix string equal
Niklas Hambüchen [Mon, 24 Aug 2020 04:23:51 +0000 (06:23 +0200)]
Use fstat() to determine file size.
This allows us to get the file size even when the input file is passed
via stdin. This fixes `--content-size` not working in situations like
$ lz4 -v --content-size < /tmp/test > /tmp/test.lz4
Warning : cannot determine input content size
With this change, it works.
Also helps with #904.
Yann Collet [Sat, 22 Aug 2020 02:23:49 +0000 (19:23 -0700)]
removed LZ4_compress_fast_force()
which serves no more purpose.
The comment implies that the simple presence of this unused function was affecting performance,
and that's the reason why it was not removed earlier.
This is likely another side effect of instruction alignment.
It's obviously unreliable to rely on it in this way,
meaning that the impact will be different, positive of negative,
with any minor code change, and any minor compiler version change, even parameter change.
Yann Collet [Sat, 22 Aug 2020 01:12:13 +0000 (18:12 -0700)]
added target aarch64, ppc64lw and s390x to TravisCI
more portability tests
Yann Collet [Fri, 21 Aug 2020 21:17:43 +0000 (14:17 -0700)]
Merge pull request #886 from servusDei2018/dev
Optimized by replacing `endl` with `'\n'`
Yann Collet [Thu, 20 Aug 2020 03:08:01 +0000 (20:08 -0700)]
Merge pull request #900 from terrelln/cmake-build
[build] Move CMake and Visual build systems to build/
Yann Collet [Thu, 20 Aug 2020 03:07:50 +0000 (20:07 -0700)]
Merge pull request #901 from terrelln/travis
[CI][Fuzz] Fix Travis-CI fuzzer tests
Nick Terrell [Thu, 20 Aug 2020 01:19:11 +0000 (18:19 -0700)]
[CI][Fuzz] Fix Travis-CI fuzzer tests
Fixes #781
Nick Terrell [Thu, 20 Aug 2020 01:06:49 +0000 (18:06 -0700)]
[build] Move CMake and Visual build systems to build/
Fixes #852.
Yann Collet [Tue, 18 Aug 2020 22:34:51 +0000 (15:34 -0700)]
Merge branch 'dev' into extraInput
Yann Collet [Tue, 18 Aug 2020 17:24:36 +0000 (10:24 -0700)]
Merge pull request #897 from lz4/lz4wlib
added target lz4-wlib
aqrit [Mon, 17 Aug 2020 21:53:07 +0000 (17:53 -0400)]
silence warning
MSVC debug mode complains
Yann Collet [Fri, 14 Aug 2020 22:48:21 +0000 (15:48 -0700)]
Merge pull request #899 from lz4/endMark
Clarifies and fix EndMark
Yann Collet [Fri, 14 Aug 2020 22:03:03 +0000 (15:03 -0700)]
clarify endMark definition
Yann Collet [Thu, 13 Aug 2020 00:27:33 +0000 (17:27 -0700)]
Clarifies and fix EndMark
EndMark, the 4-bytes value indicating the end of frame,
must be `0x00000000`.
Previously, it was just mentioned as a `0-size` block.
But such definition could encompass uncompressed blocks of size 0,
with a header of value `0x80000000`.
But the intention was to also support uncompressed empty blocks.
They could be used as a keep-alive signal.
Note that compressed empty blocks are already supported,
it's just that they have a size 1 instead of 0 (for the `0` token).
Unfortunately, the decoder implementation was also wrong,
and would also interpret a `0x80000000` block header as an endMark.
This issue evaded detection so far simply because
this situation never happens, as LZ4Frame always issues
a clean 0x00000000 value as a endMark.
It also does not flush empty blocks.
This is fixed in this PR.
The decoder can now deal with empty uncompressed blocks,
and do not confuse them with EndMark.
The specification is also clarified.
Finally, FrameTest is updated to randomly insert empty blocks during fuzzing.
BellaXlp [Wed, 12 Aug 2020 21:42:10 +0000 (05:42 +0800)]
fix issue #783 (#862)
* fix issue #783
Yann Collet [Wed, 12 Aug 2020 21:39:01 +0000 (14:39 -0700)]
added test decompress-partial with extraneous input bytes
fails currently,
for investigation of #783
aqrit [Wed, 12 Aug 2020 01:14:09 +0000 (21:14 -0400)]
rejigger bit counting intrinsics
Fix lz4/lz4#867
Optimize software fallback routines.
Delete some faulty (and dead?) MSVC big endian code.
Yann Collet [Tue, 11 Aug 2020 21:03:20 +0000 (14:03 -0700)]
added target lz4-wlib
variant of lz4 linking to liblz4 dynamic library
requires the dynamic library to expose static-only symbols (experimental API)
Example for #888
Yann Collet [Tue, 11 Aug 2020 01:17:09 +0000 (18:17 -0700)]
Merge branch 'fix832' into dev
Yann Collet [Tue, 11 Aug 2020 00:11:49 +0000 (17:11 -0700)]
fixed test of gnu c version
Yann Collet [Mon, 10 Aug 2020 21:31:48 +0000 (14:31 -0700)]
Merge pull request #896 from lz4/fix832
fix #832
Yann Collet [Mon, 10 Aug 2020 19:52:32 +0000 (12:52 -0700)]
Merge pull request #895 from lz4/hugefast
Fix #876
Yann Collet [Mon, 10 Aug 2020 18:51:57 +0000 (11:51 -0700)]
fix #832
does no longer rely on default 0-interpretation when __GNUC__ is not defined
Yann Collet [Mon, 10 Aug 2020 18:28:46 +0000 (11:28 -0700)]
Merge pull request #892 from felixhandte/lz4-clean-up-fast-reset
Remove dirty Flag in LZ4_stream_t
Yann Collet [Mon, 10 Aug 2020 18:03:27 +0000 (11:03 -0700)]
fix #876
by introducing a max limit acceleration value
Yann Collet [Mon, 10 Aug 2020 17:54:35 +0000 (10:54 -0700)]
add test that breaks --fast
with huge values, as expected from #876
Also : added target `check`
W. Felix Handte [Mon, 10 Aug 2020 17:46:31 +0000 (13:46 -0400)]
Fix Enum Casts
Fixes `-Wsign-compare` issues.
Yann Collet [Sat, 8 Aug 2020 20:21:07 +0000 (13:21 -0700)]
Merge pull request #856 from indragiek/indragiek/lz4f-visibility
Add LZ4FLIB_VISIBILITY for controlling library symbol visibility
W. Felix Handte [Thu, 6 Aug 2020 20:06:40 +0000 (16:06 -0400)]
Remove dirty Field From LZ4_stream_t
Indragie Karunaratne [Wed, 5 Aug 2020 21:34:07 +0000 (15:34 -0600)]
LZ4LIB_VISIBILITY -> LZ4FLIB_VISIBILITY in comment
W. Felix Handte [Wed, 5 Aug 2020 16:46:32 +0000 (12:46 -0400)]
Remove Extraneous Reset in LZ4_attach_dictionary()
Nothing internally sets dirty anymore. The only way to get that is if you use
an uninitialized context, in which case your warranty is void anyways.
Yann Collet [Tue, 4 Aug 2020 01:34:16 +0000 (18:34 -0700)]
Merge pull request #890 from terrelln/lz4-memcpy
Call LZ4_memcpy() instead of memcpy()
Nick Terrell [Mon, 3 Aug 2020 18:18:35 +0000 (11:18 -0700)]
Call LZ4_memcpy() instead of memcpy()
`LZ4_memcpy()` uses `__builtin_memcpy()` to ensure that clang/gcc
can inline the `memcpy()` calls in freestanding mode.
This is necessary for decompressing the Linux Kernel with LZ4.
Without an analogous patch decompression ran at 77 MB/s, and with
the patch it ran at 884 MB/s.
Yann Collet [Mon, 3 Aug 2020 17:47:24 +0000 (10:47 -0700)]
Merge pull request #885 from neheb/mes
meson: remove build requirement for distutils
Yann Collet [Wed, 29 Jul 2020 00:44:25 +0000 (17:44 -0700)]
Merge pull request #860 from adeason/old-style-definitions
Avoid old-style function definitions
Nate [Fri, 24 Jul 2020 00:13:14 +0000 (20:13 -0400)]
Optimized by replacing `endl` with `'\n'`
Rosen Penev [Tue, 21 Jul 2020 21:04:17 +0000 (14:04 -0700)]
meson: remove build requirement for distutils
Tested on a default installation of Debian 10.
Same as https://github.com/facebook/zstd/pull/2197
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Yann Collet [Thu, 16 Jul 2020 16:44:40 +0000 (09:44 -0700)]
Merge pull request #863 from Devernua/reducing_stack_usage_in_t_alignment
Reducing stack usage in _t_alignment checks
Yann Collet [Tue, 14 Jul 2020 01:09:02 +0000 (18:09 -0700)]
Merge pull request #866 from sandyharvie/dev
Fix issue #865
Yann Collet [Tue, 14 Jul 2020 01:07:05 +0000 (18:07 -0700)]
Merge pull request #871 from bimbashrestha/bsd
[build] FreeBSD update version 12.0 -> 12.1
Yann Collet [Wed, 8 Jul 2020 16:12:25 +0000 (09:12 -0700)]
Merge pull request #884 from vectorizedio/clang-ubsan
avoid computing 0 offsets from null pointers
Alexander Gallego [Wed, 8 Jul 2020 15:21:45 +0000 (08:21 -0700)]
avoid computing 0 offsets from null pointers
Similar work in the kernel:
https://patchwork.kernel.org/patch/
11351499/
UBsan (+clang-10) complains about doing pointer arithmetic (adding 0)
to a nullpointer.
This patch is tested with clang-10+ubsan
Yann Collet [Sat, 13 Jun 2020 21:31:15 +0000 (14:31 -0700)]
Update issue templates
Bimba Shrestha [Fri, 5 Jun 2020 16:57:44 +0000 (11:57 -0500)]
bsd upsate to 12-1
Yann Collet [Sat, 30 May 2020 07:07:48 +0000 (00:07 -0700)]
Merge pull request #869 from bket/fix_tests_openbsd
Fix tests on OpenBSD
Björn Ketelaars [Sat, 30 May 2020 05:36:22 +0000 (07:36 +0200)]
Fix tests on OpenBSD
Add condition for OpenBSD, which uses gmake
Christopher Harvie [Thu, 14 May 2020 03:01:43 +0000 (23:01 -0400)]
add tests
Christopher Harvie [Thu, 14 May 2020 01:48:12 +0000 (21:48 -0400)]
fix LZ4F_compressBound when automatic flushing is enabled
Aleksandr Kukuev [Mon, 11 May 2020 20:32:02 +0000 (23:32 +0300)]
Reducing stack usage in _t_alignment checks
Andrew Deason [Wed, 6 May 2020 19:36:56 +0000 (14:36 -0500)]
Avoid old-style function definitions
Define 0-argument functions like foo(void) instead of foo(), in order
to avoid a warning with -Wold-style-definition. This makes it easier
to embed lz4.c in projects that compile with -Werror
-Wold-style-definition.
Indragie Karunaratne [Sun, 26 Apr 2020 21:49:34 +0000 (14:49 -0700)]
Add LZ4FLIB_VISIBILITY for controlling library symbol visibility
Yann Collet [Tue, 3 Mar 2020 22:17:52 +0000 (14:17 -0800)]
Merge pull request #846 from bimbashrestha/issues-tmp
Adding issue templates
Bimba Shrestha [Tue, 3 Mar 2020 21:29:57 +0000 (13:29 -0800)]
Adding issue templates
Yann Collet [Wed, 26 Feb 2020 18:06:54 +0000 (10:06 -0800)]
Merge pull request #838 from Yanpas/stack_frame_2
fix: allocate LZ4HC_optimal_t opt on heap each time (#837)
Yann Collet [Sat, 8 Feb 2020 02:49:24 +0000 (18:49 -0800)]
Merge pull request #843 from filipecalasans/multiples-files-with-legacy
Implement -m option with legacy format on cli
Filipe Calasans [Sat, 8 Feb 2020 01:06:40 +0000 (17:06 -0800)]
Add test scenario test-lz4-multiple-legacy