Releases: brantburnett/Snappier
1.2.0-beta0001
Major Changes
- Support for block compression and decompression using
ReadOnlySequence<byte>
as the source andIBufferWriter<byte>
as the destination Snappy.TryCompress
andSnappy.TryDecompress
overloads that return false instead of throwing an exception if the output buffer is too smallSnappyStream
compression now recognizes blocks that compress poorly and encodes them as uncompressed data- Temporary buffers returned to the
ArrayPool
are now cleared with zeroes before they are returned to protect any sensitive data being compressed or decompressed. This adds some overhead but benchmarks showed it wasn't very significant and it's worthwhile for better security. - General performance improvements
- Improved documentation
What's Changed
- Redesign LeftShiftOverflows test by @brantburnett in #95
- Optimize Read to use Math.Min by @brantburnett in #97
- Implement a faster encoding algoritm for varint by @brantburnett in #98
- Add .NET 9 targets for testing by @brantburnett in #100
- Remove an unnecessary branch from Log2Floor by @brantburnett in #101
- Optimize varint reading for Intel using intrinsics by @brantburnett in #102
- Refactor VarInt reading to support TryRead by @brantburnett in #103
- Use VarIntEncoding.TryRead in SnappyDecompressor by @brantburnett in #104
- Add overloads to decompress from ReadOnlySequence by @brantburnett in #105
- Compress from ReadOnlySequence to an IBufferWriter by @brantburnett in #107
- Add code coverage report to CI by @brantburnett in #108
- Improve the efficiency of decompressing to IBufferWriter by @brantburnett in #109
- Drop .NET 7 target by @brantburnett in #110
- Allow benchmarks to run on Linux/Mac by @brantburnett in #111
- Add coverage summary to build results by @brantburnett in #112
- Add some tweaks to SnappyStream by @brantburnett in #113
- Add uncompressed block support to stream compression by @brantburnett in #114
- Parallelize Windows tests in GHA by @brantburnett in #115
- Use artifacts dir and add NuGet package source mapping by @brantburnett in #116
- Validate proper implementation of bufferWriter to prevent overflows by @brantburnett in #117
- Add TryCompress and TryDecompress by @brantburnett in #118
- Update readme, add a package icon, and add package validation by @brantburnett in #119
- Add a documentation site by @brantburnett in #120
- Ensure compression buffers do not overlap by @brantburnett in #121
- Misc documentation and code coverage improvements by @brantburnett in #122
- Create SECURITY.md by @brantburnett in #123
- Clear buffers before returning them to the ArrayPool by @brantburnett in #124
Full Changelog: release/1.1.6...release/1.2.0
release/1.1.6
What's Changed
- Fix nullable attributes leaking as public by @brantburnett in #94
Full Changelog: release/1.1.5...release/1.1.6
release/1.1.5
What's Changed
- Update README.md - Correcting wrong descriptions by @mjebrahimi in #91
- Switch NuGet to use a license expression to assist with SBOM analysis by @brantburnett in #93
New Contributors
- @mjebrahimi made their first contribution in #91
Full Changelog: release/1.1.4...release/1.1.5
release/1.1.4
This release primarily includes performance improvements around stream decompression in .NET 8, but does show some minor decompression improvements on other runtimes (except .NET 4.8 which is not a priority). Also includes a bug fix for a stream decompression corner case.
What's Changed
- Eliminate most CharTable range checks by @brantburnett in #81
- Apply various code analysis improvements by @brantburnett in #82
- Handle null chunk type using enum value by @brantburnett in #83
- Improve .NET 8 stream decompression perf by @brantburnett in #84
- Refactor DecompressAllTags to reduce duplicate code by @brantburnett in #85
- Use an inline array for decompression scratch by @brantburnett in #86
- Fix stream decompression corner case failure by @brantburnett in #89
- Don't pass around references to the scrach buffer by @brantburnett in #90
Decompression Benchmarks
BenchmarkDotNet v0.13.10, Windows 11 (10.0.22621.3007/22H2/2022Update/SunValley2)
12th Gen Intel Core i7-1270P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.101
Method | Runtime | Build | PGO | Mean | Error | StdDev | Ratio | RatioSD | Rank |
---|---|---|---|---|---|---|---|---|---|
BlockDecompress | .NET Framework 4.8 | Previous | N | 14.55 us | 0.089 us | 0.079 us | 1.00 | 0.00 | 1 |
BlockDecompress | .NET Framework 4.8 | Default | N | 15.78 us | 0.189 us | 0.158 us | 1.08 | 0.01 | 2 |
BlockDecompress | .NET 6.0 | Previous | N | 12.99 us | 0.151 us | 0.134 us | 1.00 | 0.00 | 2 |
BlockDecompress | .NET 6.0 | Default | N | 12.75 us | 0.181 us | 0.170 us | 0.98 | 0.02 | 1 |
BlockDecompress | .NET 8.0 | Previous | N | 10.57 us | 0.208 us | 0.214 us | 1.00 | 0.00 | 2 |
BlockDecompress | .NET 8.0 | Default | N | 10.30 us | 0.163 us | 0.152 us | 0.97 | 0.02 | 1 |
BlockDecompress | .NET 8.0 | Previous | Y | 10.51 us | 0.198 us | 0.194 us | 1.00 | 0.00 | 2 |
BlockDecompress | .NET 8.0 | Default | Y | 10.15 us | 0.173 us | 0.162 us | 0.97 | 0.02 | 1 |
Method | Runtime | Build | PGO | Mean | Error | StdDev | Ratio | RatioSD | Rank |
---|---|---|---|---|---|---|---|---|---|
StreamDecompress | .NET Framework 4.8 | Previous | N | 337.3 us | 6.35 us | 5.94 us | 1.00 | 0.00 | 1 |
StreamDecompress | .NET Framework 4.8 | Default | N | 332.4 us | 5.09 us | 4.51 us | 0.99 | 0.02 | 1 |
StreamDecompress | .NET 6.0 | Previous | N | 164.9 us | 2.57 us | 2.40 us | 1.00 | 0.00 | 2 |
StreamDecompress | .NET 6.0 | Default | N | 161.2 us | 3.21 us | 3.29 us | 0.98 | 0.02 | 1 |
StreamDecompress | .NET 8.0 | Previous | N | 194.4 us | 2.16 us | 2.02 us | 1.00 | 0.00 | 2 |
StreamDecompress | .NET 8.0 | Default | N | 189.6 us | 2.59 us | 2.42 us | 0.98 | 0.02 | 1 |
StreamDecompress | .NET 8.0 | Previous | Y | 201.7 us | 4.01 us | 4.45 us | 1.00 | 0.00 | 2 |
StreamDecompress | .NET 8.0 | Default | Y | 176.6 us | 3.49 us | 3.43 us | 0.87 | 0.03 | 1 |
Full Changelog: release/1.1.3...release/1.1.4
release/1.1.3
What's Changed
- Update to target .NET 8 by @brantburnett in #80
Full Changelog: release/1.1.2...release/1.1.3
1.1.2
What's Changed
- Update COPYING.txt by @brantburnett in #74
- Rework test reporter for public PRs by @brantburnett in #78
- Fix test reporter typo by @brantburnett in #79
- check table size greater than by @ak88 in #77
New Contributors
Full Changelog: release/1.1.1...release/1.1.2
1.1.1
This release includes some fixes to logic that could have caused unsafe memory access in cases where GC rearranges memory if they occurred at the precise moment when an out-of-range reference was on the stack. These moments are very short-lived and should be impossible to create intentionally, but they still represent a security risk.
What's Changed
- Allow ref byte to point just past the end of spans by @brantburnett in #73
Full Changelog: release/1.1.0...release/1.1.1
1.1.0
Breaking Changes
This release no longer includes explicit support for the unsupported runtimes .NET Core 3.0/3.1 or .NET 5.0. These runtimes are now supported via the .NET Standard 2.0 target. While they should still function, it is untested and the lack of hardware intrinsic support will most likely result in degraded performance compared to 1.0.
What's Changed
- All use of GC pinning and fixed buffers has been replaced with byte references and System.Runtime.CompilerServices.Unsafe. This should have less impact on garbage collection performance, especially on systems with continuous Snappy load.
- Reduced heap allocations
- Ported some performance improvements from the reference implementation that were added since the original port
- Lots more performance tweaks
Performance
This release sees a consistent improvement in both compression and decompression performance across all supported frameworks on my Core i7-10850H CPU, especially on decompression. On .NET 6, the time was reduced by 18% when compressing 64KB of HTML and 52% when decompressing. In some cases, the decompression time is reduced by as much as 66%.
Changelog
- Add .NET 6.0 target by @brantburnett in #40
- Update CodeQL to use .NET 6 by @brantburnett in #41
- Remove support for unsupported runtimes by @brantburnett in #42
- Rename GitHub actions and run on release branches by @brantburnett in #43
- Preload Pshufb fill pattern Vector128 by @brantburnett in #44
- Make unit tests .NET 4.8 compatible by @brantburnett in #45
- Switch CopyHelpers to use ref byte instead of byte* by @brantburnett in #46
- Improve perf of UnalignedCopy64/128 by @brantburnett in #47
- Rewrite decompressor AppendXXX methods to use refs by @brantburnett in #48
- Rewrite decompressor RefillTagFromScatch method to use refs by @brantburnett in #49
- Switch decompression input and scratch to use ref byte by @brantburnett in #50
- Switch decompression buffer and op to use ref byte by @brantburnett in #51
- Move steps to decompress from scratch to a separate method by @brantburnett in #52
- Switch compression hash table from pinned ptr to ref by @brantburnett in #53
- Reduce compression heap and buffer overhead by @brantburnett in #54
- Use ref byte for compression output pointer by @brantburnett in #55
- Use ref byte for compression input pointers by @brantburnett in #56
- Switch stream decompression from pointers to spans by @brantburnett in #57
- Use ThrowHelper for all exceptions by @brantburnett in #58
- Cleanup benchmarks by @brantburnett in #59
- Allow shorter compression output buffer lengths by @brantburnett in #60
- Improve benchmark summaries by @brantburnett in #61
- Fix compression stream benchmarks by @brantburnett in #62
- Improve block benchmarks by @brantburnett in #63
- Improve speed of hash table clear by @brantburnett in #64
- Simplify PSHUFB copy logic for .NET 6/7 by @brantburnett in #65
- Use CRC32 hash when available during compression by @brantburnett in #66
- Remove some unnecessary unsafe blocks by @brantburnett in #67
- Use ArrayPool instead of MemoryPool for the lookback buffer by @brantburnett in #68
- Improve Log2Floor for legacy runtimes by @brantburnett in #69
- Optimize FindMatchLength for x64 by @brantburnett in #70
- Simplify stream decompress logic by @brantburnett in #71
- Switch several arrays to static data references by @brantburnett in #72
Full Changelog: release/1.0.0...release/1.1.0
Initial release
Note that this release is focused on performance on Intel/AMD x64 processes on .NET Core 3.1 or .NET 5. It uses Intel intrinsics extensively, so performance will be slower on other CPU architectures or runtimes.
1.0.0 Beta 2
Various bug fixes found during beta