Bimba Shrestha [Mon, 7 Oct 2019 15:12:42 +0000 (08:12 -0700)]
Adding unamestr var
Bimba Shrestha [Fri, 4 Oct 2019 17:12:22 +0000 (10:12 -0700)]
Adding condition for FreeBSD and using gmake
Bimba Shrestha [Fri, 4 Oct 2019 16:29:58 +0000 (09:29 -0700)]
Using instead of gmake (to address the travis failure)
Bimba Shrestha [Fri, 4 Oct 2019 15:40:46 +0000 (08:40 -0700)]
Using gmake instead of make
Bimba Shrestha [Fri, 4 Oct 2019 15:36:08 +0000 (08:36 -0700)]
Adding cirrus config file for freebsd-12-0
bimbashrestha [Sat, 17 Aug 2019 00:14:47 +0000 (17:14 -0700)]
Adding comments, fixing nit, and hiding the struct in data producer api
bimbashrestha [Fri, 16 Aug 2019 23:43:28 +0000 (16:43 -0700)]
Seperating fuzz data producer api impl and header, using data producer on the easy fuzzers
bimbashrestha [Fri, 16 Aug 2019 21:19:06 +0000 (14:19 -0700)]
Created a data producer API and used in decompress_fuzzer
bimbashrestha [Fri, 16 Aug 2019 17:50:46 +0000 (10:50 -0700)]
Adding fuzz data producer for uint32 and using in decompress_fuzzer
Summary: Consuming bytes from the end of data instead of from the front to prevent "all-in-one" decisions.
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Yann Collet [Thu, 15 Aug 2019 11:59:59 +0000 (13:59 +0200)]
fixed test error
could trigger %0 on exceptional circumstances
due to wrong buffer size parameter.
Yann Collet [Sat, 10 Aug 2019 00:08:03 +0000 (02:08 +0200)]
Merge pull request #777 from terrelln/off-by-one
[LZ4_compress_destSize] Fix off-by-one error
Nick Terrell [Fri, 9 Aug 2019 17:32:26 +0000 (10:32 -0700)]
[LZ4_compress_destSize] Fix off-by-one error
PR#756 fixed the data corruption bug, but didn't clear `ip`. PR#760
fixed that off-by-one error, but missed the case where `ip == filledIp`,
which is harder for the fuzzers to find (it took 20 days not 1 day).
Verified this fixed the issue reported by OSS-Fuzz.
Credit to OSS-Fuzz.
Yann Collet [Wed, 7 Aug 2019 23:48:53 +0000 (01:48 +0200)]
Merge pull request #773 from felixhandte/attach-empty-dict-behavior-conformance
Make Attaching an Empty Dict Behave the Same as Using it Directly
W. Felix Handte [Tue, 6 Aug 2019 23:08:41 +0000 (19:08 -0400)]
Only Bump Offset When Attaching Non-Null Dictionary
We do want to bump, even if the dictionary is empty, but we **don't** want to
bump if the dictionary is null.
W. Felix Handte [Tue, 6 Aug 2019 22:54:03 +0000 (18:54 -0400)]
Add Attach Dict Debug Log
W. Felix Handte [Tue, 6 Aug 2019 19:24:51 +0000 (15:24 -0400)]
Make Attaching an Empty Dict Behave the Same as Using it Directly
When using an empty dictionary, we bail out of loading or attaching it in
ways that leave the working context in potentially slightly different states.
In particular, in some paths, we will cause the currentOffset to be non-zero,
while in others we would allow it to remain 0.
This difference in behavior is perfectly harmless, but in some situations, it
can produce slight differences in the compressed output. For sanity's sake,
we currently try to maintain a strict correspondence between the behavior of
the dict attachment and the dict loading paths. This patch restores them to
behaving identically.
This shouldn't have any negative side-effects, as far as I can tell. When
writing the dict attachment code, I tried to preserve zeroed currentOffsets
when possible, since they benchmarked as very slightly faster. However, the
case of attaching an empty dictionary is probably rare enought that it's
acceptable to minisculely degrade performance in that corner case.
Yann Collet [Tue, 6 Aug 2019 17:17:16 +0000 (19:17 +0200)]
Merge pull request #772 from lz4/offset0
silence msan warning when offset==0
Yann Collet [Tue, 6 Aug 2019 12:46:31 +0000 (14:46 +0200)]
silence msan warning when offset==0
Yann Collet [Thu, 1 Aug 2019 23:27:53 +0000 (01:27 +0200)]
Merge pull request #771 from terrelln/rep-ext-fix
[lz4hc] Further improve pattern detection and chain swapping
Nick Terrell [Wed, 31 Jul 2019 07:57:16 +0000 (00:57 -0700)]
[lz4hc] Chain swap with acceleration
Nick Terrell [Wed, 31 Jul 2019 06:40:58 +0000 (23:40 -0700)]
[lz4hc] Only allow chain swapping forwards
When the match is very long and found quickly, we can do
matchLength * nbCompares iterations through the chain
swapping, which can really slow down compression.
Nick Terrell [Wed, 31 Jul 2019 06:39:39 +0000 (23:39 -0700)]
[lz4hc] Fix pattern detection end of dictionary
The pattern detection in extDict mode could put `matchIndex`
within the last 3 bytes of the dictionary. This would cause
a read out of bounds.
Nick Terrell [Tue, 30 Jul 2019 22:21:52 +0000 (15:21 -0700)]
[lz4hc] Fix minor pessimization in extDict pattern matching
We should be comparing `matchPtr` not `ip`. This bug just means
that this branch was not taken, so we might miss some of the
forward length.
Nick Terrell [Tue, 30 Jul 2019 22:16:35 +0000 (15:16 -0700)]
[lz4hc] Improve pattern detection in ext dict
It is important to continue to look backwards if the current pattern
reaches `lowPrefixPtr`. If the pattern detection doesn't go all the
way to the beginning of the pattern, or the end of the pattern it
slows down the search instead of speeding it up.
The slow unit in `round_trip_stream_fuzzer` used to take 12 seconds
to run with -O3, now it takes 0.2 seconds.
Credit to OSS-Fuzz
Yann Collet [Wed, 24 Jul 2019 20:47:19 +0000 (13:47 -0700)]
Merge pull request #768 from terrelln/rep-ext
[LZ4HC] Speed up pattern compression with external dictionary
Nick Terrell [Wed, 24 Jul 2019 00:54:09 +0000 (17:54 -0700)]
[LZ4HC] Speed up pattern compression with external dictionary
Fixes #761.
Yann Collet [Tue, 23 Jul 2019 08:00:53 +0000 (01:00 -0700)]
Merge pull request #766 from Low-power/cli-option---best
Add option '--best' to lz4(1)
WHR [Tue, 23 Jul 2019 05:37:11 +0000 (13:37 +0800)]
lz4cli: add option '--best' as an alias of '-12'
Yann Collet [Fri, 19 Jul 2019 23:54:01 +0000 (16:54 -0700)]
Merge pull request #763 from terrelln/unused
[lz4frame] Fix unused variable warnings in fuzzing mode
Yann Collet [Fri, 19 Jul 2019 22:22:51 +0000 (15:22 -0700)]
Merge pull request #760 from terrelln/destSize
[LZ4_compress_destSize] Fix off-by-one error in fix
Nick Terrell [Fri, 19 Jul 2019 21:44:06 +0000 (14:44 -0700)]
[lz4frame] Fix unused variable warnings in fuzzing mode
Yann Collet [Fri, 19 Jul 2019 16:11:12 +0000 (09:11 -0700)]
Merge pull request #758 from dooxe/develop
Added `BUNDLE DESTINATION`
Yann Collet [Fri, 19 Jul 2019 04:53:33 +0000 (21:53 -0700)]
Merge pull request #762 from terrelln/frame-fuzz
[fuzz] Add LZ4 frame fuzzers
Nick Terrell [Fri, 19 Jul 2019 01:49:40 +0000 (18:49 -0700)]
[fuzz] Add LZ4 frame fuzzers
* Round trip fuzzer
* Compress fuzzer
* Decompress fuzzer
Nick Terrell [Fri, 19 Jul 2019 01:45:32 +0000 (18:45 -0700)]
[lz4frame] Skip magic and checksums in fuzzing mode
When `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined we skip
magic and checksum checks. This makes it easier to fuzz decompression.
Nick Terrell [Thu, 18 Jul 2019 02:40:04 +0000 (19:40 -0700)]
Fix LZ4_attach_dictionary with empty dictionary
Nick Terrell [Thu, 18 Jul 2019 01:31:44 +0000 (18:31 -0700)]
[fuzz] Add HC fuzzers for round trip, compress, and streaming
Nick Terrell [Thu, 18 Jul 2019 00:41:41 +0000 (17:41 -0700)]
[fuzzer] Update scripts for new fuzzers
Nick Terrell [Thu, 18 Jul 2019 00:33:31 +0000 (17:33 -0700)]
[fuzz] Add a streaming round trip fuzzer
Nick Terrell [Thu, 18 Jul 2019 19:20:29 +0000 (12:20 -0700)]
[LZ4_compress_destSize] Fix off-by-one error in fix
The next match is looking at the current ip, not the next ip,
so it needs to be cleared as well.
Credit to OSS-Fuzz
W. Felix Handte [Thu, 18 Jul 2019 17:35:12 +0000 (13:35 -0400)]
Unconditionally Clear `dictCtx`
W. Felix Handte [Thu, 18 Jul 2019 16:41:12 +0000 (12:41 -0400)]
Fix Data Corruption Bug when Streaming with an Attached Dict in HC Mode
This diff fixes an issue in which we failed to clear the `dictCtx` in HC
compression. The `dictCtx` is not supposed to be used when an `extDict` is
present: matches found in the `dictCtx` do not account for the presence of an
`extDict` segment, and their offsets are therefore miscalculated when one is
present. This can lead to data corruption.
This diff clears the `dictCtx` whenever setting an `extDict`.
This issue was uncovered by @terrelln's fuzzing work.
dooxe [Thu, 18 Jul 2019 09:25:43 +0000 (11:25 +0200)]
Added BUNDLE DESTINATION in CMakeLists.txt so that it works with newer versions of cmake
Yann Collet [Wed, 17 Jul 2019 20:25:41 +0000 (13:25 -0700)]
Merge pull request #756 from terrelln/destSize
[LZ4_compress_destSize + multi-blocks streaming] Fix rare data corruption bug
Nick Terrell [Wed, 17 Jul 2019 18:50:47 +0000 (11:50 -0700)]
[LZ4_compress_destSize] Fix overflow condition
Nick Terrell [Wed, 17 Jul 2019 18:28:38 +0000 (11:28 -0700)]
[LZ4_compress_destSize] Fix rare data corruption bug
Nick Terrell [Wed, 17 Jul 2019 18:07:24 +0000 (11:07 -0700)]
[LZ4_compress_destSize] Allow 2 more bytes of match length
Yann Collet [Tue, 16 Jul 2019 18:18:09 +0000 (11:18 -0700)]
Merge pull request #752 from terrelln/fuzzers
[ossfuzz] Improve the fuzzers
Yann Collet [Mon, 15 Jul 2019 23:38:28 +0000 (16:38 -0700)]
Merge pull request #755 from lz4/custom_distance
ensure conformance with custom LZ4_DISTANCE_MAX
Nick Terrell [Sat, 13 Jul 2019 02:27:00 +0000 (19:27 -0700)]
[ossfuzz] Improve the fuzzers
* Run more decompression variants
* Round trip the compression fuzzer and do partial decompression as well
* Add a compression fuzzer that compresses into a smaller output buffer
and test the destSize variant
These fuzzers caught 2 bugs that were fixed in the previous commit.
* Input buffer over-read in partial decompress
* Partial decompress fails if output size is 0
Nick Terrell [Sat, 13 Jul 2019 01:36:28 +0000 (18:36 -0700)]
[lz4] Fix bugs in partial decoding
* Partial decoding could read a few bytes beyond the end of the input
* Partial decoding returned an error with an empty output buffer
Yann Collet [Mon, 15 Jul 2019 19:11:34 +0000 (12:11 -0700)]
ensure conformance with custom LZ4_DISTANCE_MAX
It's now possible to select a custom LZ4_DISTANCE_MAX at compile time,
provided it's <= 65535.
However, in some cases (when compressing in byU16 mode),
the new distance wasn't respected,
as it used to implied that it was necessarily within range.
Added a distance check for this case.
Also : added a new TravisCI test which ensures that
custom LZ4_DISTANCE_MAX compiles correctly
and compresses correctly (relying on `assert()` to find outsized offsets).
Yann Collet [Mon, 15 Jul 2019 16:08:11 +0000 (09:08 -0700)]
Merge pull request #753 from Hitatm/fix_LZ4_DISTANCE_MAX
bugfix: correctly control the offset < LZ4_DISTANCE_MAX,when change t…
Hitatm [Mon, 15 Jul 2019 14:53:46 +0000 (22:53 +0800)]
bugfix: correctly control the offset < LZ4_DISTANCE_MAX,when change the value of LZ4_DISTANCE_MAX,
Yann Collet [Fri, 12 Jul 2019 00:29:16 +0000 (17:29 -0700)]
keep the "lorem ipsum" topic of the example string
but make it compressible
Yann Collet [Fri, 12 Jul 2019 00:26:15 +0000 (17:26 -0700)]
Merge pull request #751 from hamidzr/simple-buffer-example-input
simple buffer example minor input update. fixes #750
Hamid Zare [Thu, 11 Jul 2019 21:39:29 +0000 (14:39 -0700)]
print the compression ratio
Hamid Zare [Thu, 11 Jul 2019 21:34:52 +0000 (14:34 -0700)]
changed the input text to something more compression friendly
Yann Collet [Thu, 4 Jul 2019 20:03:08 +0000 (13:03 -0700)]
Merge pull request #749 from sylvestre/patch-1
Remove an useless declaration
Sylvestre Ledru [Thu, 4 Jul 2019 16:13:36 +0000 (18:13 +0200)]
Remove an useless declaration
Yann Collet [Wed, 3 Jul 2019 23:16:52 +0000 (16:16 -0700)]
Merge pull request #746 from lz4/circleci
CircleCI : reduced test duration
Yann Collet [Wed, 3 Jul 2019 23:16:19 +0000 (16:16 -0700)]
Merge pull request #743 from lz4/fuzzasan_fixed
updated frametest
Yann Collet [Wed, 3 Jul 2019 23:13:44 +0000 (16:13 -0700)]
Merge pull request #748 from amchoukir/amchoukir-doublebuffer-doc
Update blockStreaming_doubleBuffer.md
Amine Choukir [Wed, 3 Jul 2019 09:50:38 +0000 (11:50 +0200)]
Update blockStreaming_doubleBuffer.md
Yann Collet [Mon, 1 Jul 2019 18:52:08 +0000 (11:52 -0700)]
Merge pull request #745 from lz4/bump_192
bumped version number to v1.9.2
Yann Collet [Mon, 1 Jul 2019 17:55:54 +0000 (10:55 -0700)]
CircleCI : reduced test duration
Ideally, we want to make CircleCI a "fast" test environment, with short feedback loop (~5mn).
We are still far from this goal.
This patch starts this process by removing "long" tasks which are non-essential and redundant with travisCI.
It also acknowledges that parallelism is broken.
The script should be more heavily updated to support parallelism,
which might be important to support its goal.
Yann Collet [Mon, 1 Jul 2019 16:01:43 +0000 (09:01 -0700)]
bumped version number to v1.9.2
to reduce risks that future bug reports in `dev` branch report `v1.9.1` as the failing version.
Yann Collet [Sun, 30 Jun 2019 22:36:32 +0000 (15:36 -0700)]
fix minor cppcheck warnings
Yann Collet [Sun, 30 Jun 2019 21:57:12 +0000 (14:57 -0700)]
updated frametest
so that noisy src decompression
doesn't generate output
nor fails when decompression fails (which is expected).
Yann Collet [Sun, 30 Jun 2019 21:01:08 +0000 (14:01 -0700)]
Merge branch 'fuzzasan' into fuzzasan_fixed
Yann Collet [Sun, 30 Jun 2019 20:59:49 +0000 (13:59 -0700)]
frametest: added LZ4F decoder noise test
Yann Collet [Sun, 30 Jun 2019 20:05:15 +0000 (13:05 -0700)]
Merge pull request #739 from cmeister2/cmeister2/ossfuzz
oss-fuzz: Add a fuzzing target that compiles in the oss-fuzz environment
Max Dymond [Sun, 30 Jun 2019 19:16:03 +0000 (20:16 +0100)]
Remove unnecessary call to Makefile.inc
Yann Collet [Sun, 30 Jun 2019 18:33:38 +0000 (11:33 -0700)]
Merge pull request #741 from lzutao/meson-deprecated-warn
meson: Fix deprecated warnings on build options
Yann Collet [Sun, 30 Jun 2019 16:41:56 +0000 (09:41 -0700)]
Merge pull request #742 from lz4/fuzzasan_fixed
New fuzzer test for #738
Lzu Tao [Sat, 29 Jun 2019 13:54:39 +0000 (20:54 +0700)]
meson: Rename options in travis config
Yann Collet [Sat, 29 Jun 2019 04:03:07 +0000 (21:03 -0700)]
Merge branch 'dev' into fuzzasan_fixed
Yann Collet [Sat, 29 Jun 2019 03:55:47 +0000 (20:55 -0700)]
travisCI: added ASAN fuzzer tests
and fixed minor formatting warnings
Yann Collet [Sat, 29 Jun 2019 03:23:12 +0000 (20:23 -0700)]
moved noisy-src decoder test into cBuffer_exact
so that it can also catch any potential read out-of-bound in the input buffer
(none reported so far, just a precaution for the future).
Yann Collet [Sat, 29 Jun 2019 03:15:43 +0000 (20:15 -0700)]
fuzzer: added test to catch #738
Lzu Tao [Sat, 29 Jun 2019 03:11:32 +0000 (10:11 +0700)]
meson: Always build gen_manual on build machine
As gen_manual is using as a generator, not a binary target
installed on host machine.
Lzu Tao [Sat, 29 Jun 2019 03:10:05 +0000 (10:10 +0700)]
meson: Fix deprecated warnings on build options
Meson now reserves `build_` prefix options.
Yann Collet [Sat, 29 Jun 2019 02:22:54 +0000 (19:22 -0700)]
Merge pull request #740 from terrelln/fix2
Fix out-of-bounds read
Max Dymond [Fri, 28 Jun 2019 23:23:06 +0000 (00:23 +0100)]
More markups for style changes
Max Dymond [Fri, 28 Jun 2019 22:48:33 +0000 (23:48 +0100)]
Move to using C rather than C++ for compilation
Nick Terrell [Fri, 28 Jun 2019 21:40:14 +0000 (14:40 -0700)]
Fix out-of-bounds read of up to 64 KB in the past
Max Dymond [Fri, 28 Jun 2019 21:19:27 +0000 (22:19 +0100)]
Write a simple decompress target as well
Max Dymond [Fri, 28 Jun 2019 19:54:46 +0000 (20:54 +0100)]
Code review markups:
- Correct use of CPPFLAGS
- Detect allocation failure
- Add a CHECK macro for failure
Max Dymond [Tue, 25 Jun 2019 16:22:02 +0000 (17:22 +0100)]
Add a fuzzing target that compiles in the oss-fuzz environment
Yann Collet [Tue, 25 Jun 2019 17:32:19 +0000 (10:32 -0700)]
Merge pull request #735 from lz4/versionOut
fix #734 : --version should output to stdout
Yann Collet [Mon, 24 Jun 2019 23:08:30 +0000 (16:08 -0700)]
fix #734 : --version should output to stdout
instead of stderr
Yann Collet [Thu, 6 Jun 2019 21:17:44 +0000 (14:17 -0700)]
update simple_buffer example
there were a few tiny inaccuracies, especially in error conditions.
Yann Collet [Thu, 6 Jun 2019 20:20:30 +0000 (13:20 -0700)]
precise again that LZ4 decoder needs metadata
and that such metadata must be provided / sent / saved by the application.
Yann Collet [Wed, 5 Jun 2019 21:34:28 +0000 (14:34 -0700)]
Merge pull request #727 from ephiepark/dev
CircleCI : use custom Docker Image with pre-installed dependencies
Ephraim Park [Tue, 4 Jun 2019 19:45:31 +0000 (12:45 -0700)]
circleci : use custom docker image with pre-installed dependencies
Yann Collet [Tue, 4 Jun 2019 21:04:49 +0000 (14:04 -0700)]
restored FORCE_INLINE
Yann Collet [Fri, 31 May 2019 22:18:12 +0000 (15:18 -0700)]
Merge pull request #718 from lz4/fullbench
fullbench: added test scenario LZ4F_decompress_followHint
Yann Collet [Fri, 31 May 2019 20:25:12 +0000 (13:25 -0700)]
fullbench: ensure decompressionFunction and dName are initialized
Visual Studio seems to miss that they are necessarily initialized in the switch() { case: }
Yann Collet [Fri, 31 May 2019 19:59:38 +0000 (12:59 -0700)]
Merge pull request #717 from lz4/inplace
Added documentation and macro to support in-place compression and decompression
Yann Collet [Fri, 31 May 2019 19:23:52 +0000 (12:23 -0700)]
Merge pull request #719 from parheliamm/dev
FAST_DEC_LOOP: move offset check in specific condition.